Issues installing a fully working openminted platform #6
Comments
Do the Galaxy instances have omtdImporter tool installed? Please look in the left ... under "Tools"? I do not know which applications are you running/trying but I think that the respective workflows |
Both galaxy instances have the omtdImporter installed. This should enable the application creation from within galaxy by itself or do I need to load components by hand into galaxy first, so they are available in the portal when we try to create an application?
The data available in the registry is that uploaded by the upload.yml I assumed the data is valid and used for testing purposes. |
OK great. This means that at least a part of the whole thing has been installed correctly.
If Registry is correctly installed and integrated to Galaxies you can register TDM components
Yes I think that the data are valid.
I assume that if these two things are working as they should then it is pretty easy to have a replica of |
I can create components and they show up on the galaxy editor, but do not on the executor. I'm trying to find what might be the issue here. Then if I upload an application (one from the dump.zip) it shows up in the executor, but not in the editor.
All my config changes are in the forked repo, like the registry data, and also my nginx config.
|
There are 2 things that should be synced
|
|
It seems that in your case the editor and executor are installed in the same machine. I think that in this https://github.com/openminted/omtd-standalone-setup/blob/master/roles/editor/tasks/main.yaml#L70 |
|
NFS is still needed to share data between Galaxy and workers.
So, basically, the same NFS server is used for two different
functionalities:
1. to share the tools between Galaxy executor and Galaxy editor and
2. to share the data between Galaxy and container executors (Mesos cluster)
If both Galaxies run on the same machine, you can disable NFS for the tools.
If you don't have a Mesos cluster setup and you don't plan to set up one,
you can disable NFS for the data.
…On Wed, Jul 10, 2019 at 4:59 PM Dimitrios Galanis ***@***.***> wrote:
It seems that in your case the editor and executor are installed in the
same machine. I think that in this
specific case NFS is not used but something much simpler. A symbolic link
between the 2 folders.
@saxtouri <https://github.com/saxtouri> is this correct?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6?email_source=notifications&email_token=AAXUSWZJBBK3SN5U3YT3I23P6XTNJA5CNFSM4H7EWG4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZTRT5I#issuecomment-510073333>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXUSW54VRSOHOY2Q65Q26LP6XTNJANCNFSM4H7EWG4A>
.
--
Stavros 0 Sahtouris
|
Correct but in the case of BSC (standalone setup) it seems that editor <-> executor syncing is done with the symbolic link https://github.com/openminted/omtd-standalone-setup/blob/master/roles/editor/tasks/main.yaml#L70. Not sure about that..... but I think that you have to check if the link was created because it might cause the problem you have mentioned. |
|
The link was created and is shared between both galaxy instances, the Regarding errors in the registry, if try to create an application the following messages show up in the registry container: Then after some time this (which repeats every few minutes) It makes reference to content-connector-service which is handled by nginx to point to the omtd-content-service container The omtd-content-service container |
|
Another detail, when I try to create an application with the registry galaxy editor, the page stays blank, but the registry container is able to create the workflow on the galaxy editor: And in the postgres instance (and also the galaxy editor) |
This is very strange. I have tested the standalone installation and I remember that everything worked as it should. The standalone installation was also tested and by another engineer. A few questions that will help in understanding what happens...
|
|
|
I suggest the following plan Step 1: then ... it seems that the tools are synced. If you want to verify that you have to register a component/tool by using the respective form (https:///resourceRegistration/component e.g. https://services.openminted.eu/resourceRegistration/component). Step 2: and save it as application in the Registry. Based on the screenshots it seems that the only component/tool that is available right now in the |
I tried adding a component as suggested, and registry reports everything going ok, but I see nothing new on the galaxy tools.
|
|
I've been trying to figure out why what I import/create from the registry does not show up on the galaxy instances. After checking the registry logs, this two lines: and: I checked the the registry repo in search of a path or variable to define the target location for the tool to be copied and could not find any (all references use relative path |
Yes via NFS. However (as far as I understand) for some reason copying does not work. You have to check that NFS mount is OK.
Great. So syncing works
I remember that some tool ids were causing some issues in specific Galaxy pages. E.g we were getting errors like the following ... "Could not find tool with id 'mvn:de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.clearnlp-asl:1.9.2'." The problem is 1. I also see exceptions that have to do with the content service Not sure how this affects NFS copying. It shouldn't .... I think |
Registry resides in 10.43.0.11 and galaxy instances in 10.43.0.12. 10.43.0.12 exports /srv/executor/tools via NFS and is mounted by 10.43.0.11, the node that runs the registry and other containers at /media/galaxy. The tool that I imported, I copied via NFS so the mountpoint and sync work as expected. I took the mountpoint from the ansible playbook however I could not find/media/galaxy neither in the environment configuration files nor any of the repository code.
I'm not sure how or where should I point registry to that folder. |
|
Can you send us the contents of in 10.43.0.12:/etc/export and
10.43.0.11/etc/fstab ?
…On Mon, Jul 15, 2019 at 8:49 AM bscopenminted ***@***.***> wrote:
The problem is *1*.
It is probably NFS. @saxtouri <https://github.com/saxtouri> any ideas?
The Registry and Galaxy machines are different? If yes, is it possible to
ping the one from the other?
I also see exceptions that have to do with the content service
10:10:21.786 INFO [pool-4-thread-1] StatsServiceImpl - Checking for totals
10:10:23.324 ERROR [pool-4-thread-1] StatsServiceImpl - Request on http://10.43.0.11:8888/content-connector-service/ failed
Not sure how this affects NFS copying. It shouldn't .... I think
@antleb <https://github.com/antleb> @Jodee90 <https://github.com/Jodee90>
?
Registry resides in 10.43.0.11 and galaxy instances in 10.43.0.12.
10.43.0.12 exports /srv/executor/tools via NFS and is mounted by
10.43.0.11, the node that runs the registry and other containers at
/media/galaxy. The tool that I imported, I copied via NFS so the mountpoint
and sync work as expected. I took the mountpoint from the ansible playbook
https://github.com/openminted/install-tutorial/blob/f0d0d04aed53376e1e23b7e75e58c47b84176755/registry/scripts/ansible-scripts/nfs-init.yml#L15
however I could not find /media/galaxy neither in the environment
configuration files nor any of the repository code.
I'm not sure how or where should I point registry to that folder.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6?email_source=notifications&email_token=AAXUSW3TGBWQ3NZKRUOE4ITP7QFWFA5CNFSM4H7EWG4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ4YNLQ#issuecomment-511280814>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXUSW32WGDBM527DR6QS43P7QFWFANCNFSM4H7EWG4A>
.
--
Stavros 0 Sahtouris
|
|
The exports: The client mount options in |
|
This is the OMTD Registry code that writes the Galaxy XML wrappers. The Registry runs in a Docker container and the code uses a specific path in the container.
Please check that mount 1 is also OK. |
"docker inspect ..." shows mounts I think. |
|
OK, it looks good, but is it mounted? If you run a "df" on the registry
host, do you see this line?
10.43.0.12:/srv/executor/tools <some numbers> <some numbers> <some
numbers> <some percentage>% /media/galaxy
You have probably already checked the following, but I will ask just to
make sure:
on the galaxies host, have you linked the editor's tools directory to the
executor's link directory?
…On Mon, Jul 15, 2019 at 11:25 AM bscopenminted ***@***.***> wrote:
The exports:
***@***.***:/root/omtd-installations/omtd-store/scripts# cat /etc/exports
/srv/executor/database 10.43.0.10(rw,sync,no_subtree_check,no_root_squash)
/srv/executor/database 10.43.0.11(rw,sync,no_subtree_check,no_root_squash)
/srv/executor/tools 10.43.0.10(rw,sync,no_subtree_check,no_root_squash)
/srv/executor/tools 10.43.0.11(rw,sync,no_subtree_check,no_root_squash)
The client mount options in /etc/fstab
10.43.0.12:/srv/executor/tools /media/galaxy/ nfs rsize=8192,wsize=8192,timeo=14,intr
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6?email_source=notifications&email_token=AAXUSWZBDCNJPPUHTPTYURDP7QX65A5CNFSM4H7EWG4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5APRY#issuecomment-511313863>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAXUSW5UITAHYJT4STA2HRLP7QX65ANCNFSM4H7EWG4A>
.
--
Stavros 0 Sahtouris
|
|
I managed to get it working by editing the extra mount at I changed it so that it mounts the local nfs client mount:This from the recommendation of checking the java code that @galanisd pointed out, where I found:https://github.com/openminted/omtd-registry/blob/0f3a0bec7c76411bd58c09bd32e94425941205fa/omtd-registry-service/src/main/java/eu/openminted/registry/service/tool/WorkflowEngineComponentRegistryGalaxyImpl.java#L32 applying the changes and after the registry restart I can now add a maven tool and it shows up in the galaxy instances :) Now what I'm missing is the galaxy workflow editor when I try to add an app, it shows a blank frame where the galaxy editor canvas should show up. Any ideas? could this be the nginx conf or some other thing? Thanks! |
|
Nginx acts as proxy for Galaxy editor. So, please check this URL (https://nginx IP/galaxy) and tell me what happens. Nginx should be configured with with secret key of Galaxy editor etc. |
|
Before checking https://<nginx IP>/galaxy I would also restart the Galaxy editor. Sometimes it gets stuck. |
|
Frontend nginx galaxy node 10.43.0.12 has apache as reverse proxy for both galaxies This is what the omtd-standalone-setup apache role sets (abeit ProxyPass twice instead of ProxyPass and ProxyPassReverse) https://github.com/openminted/omtd-standalone-setup/blob/master/roles/apache2/templates/vhosts.conf.j2 The nginx conf also contains (from the config you sent me by mail ) The If I try to reach http://nginx-frontend/galaxy I reach the apache server in 10.43.0.12 . Do you see anything that stands out as wrong here? and should the 5555 port be configured for any of the containers? |
<REMOTE_USER> should be substituted with something else. I think that it does not matter with what
<executoromtdsecret> should be substituted with the value that is set to There are some instructions on how to set this secret when the editor is installed here.
Check also in the .../config/galaxy.ini file that |
|
Any updates? Did it work? |
|
Hi, sorry I've been caught up with other issues :/ I did check the config and updated the nginx config to remove the <> as the values where correct. I had galaxy configured without the I checked the headers nginx is passing to the apache proxy that sits in front of galaxy and the values are correct: Then in the galaxy log the lines in galaxy.ini for this values: From what I understand, the user field is not relevant, but required. Could this be the issue? |
|
If I understand well you got the same message as in #4 At the nginx side
At the Galaxy side (Galaxy.ini) remote_user_maildomain = ???? |
Correct, the same message
Yes, not secure but it's just for testing
The headers that nginx sends are: It correctly sends |
|
Also just confirmed that REMOTE_USER value does not matter. I changed it in nginx config and restarted it. |
|
I was wondering if the apache that proxies the galaxy instances was not forwarding all headers, so I changed the nginx conf to point directly to the galaxy port of the editor, but the result is the same. Here's the data received by galaxy editor and sent from nginx: *The redacted IP is the one of the client connecting to nginx(my computer), not the one of the nginx server. |
|
Not sure whether the problem is in the nginx side or in the Galaxy side. A way to solve this is the following
If you get the same message then probably the problem is in Galaxy configuration. |
|
I get the same error, so it seems a galaxy issue then. The galaxy version I'm using is the one pulled from the ansible playbook that is used in |
|
As far as I know this is the correct one. My guess is that you are installing the correct version, however, the problem is in the configuration. I just had a look into the config.ini file of the editor I noticed one difference. Please try it. Set allow_user_creation = False |
However I do not understand why allow_user_creation should affect "remote_user" logins. |
|
I tried allow_user_creation = True but it didn't make any difference.
Good idea. Strange that it didn't work. |
|
I just discussed the issue with @saxtouri from GRNET and he remembered that enabling Please try the following steps: Step 1: Step 2: Step 3: It seems that Galaxy requires one user in order to enable remote_user access. |
|
Yes, that's required as is the only way to generate the api token. Not sure if this might be the cause, but its worth checking. |
|
I managed to make the auth work, it seems that I shouldn't use quotes for the secret string in the ini file. After that I got the following error trying to add a app (no issue with reaching /galaxy) I can comment the https://github.com/openminted/galaxy/blob/23bb846ef20188a941bcf7721c6ee2083f42c29d/lib/galaxy/managers/base.py#L61-L62 lines and then the galaxy frame shows up in the registry. Tried getting both values but both are empty when I print them in the same error line. I'm not sure if just removing the check is the best idea, but also I don't know why it happens. Any ideas? Using: |
Great!
I remember that we had some ownership issues and these were solved by (I think, not sure) by either changing the source code in Galaxy or by generating the appropriate calls to Galaxy api. I suspect that our installation (services.openminted.eu) uses a version of Galaxy code that contains the required changes. I just checked the commit of the editor that is deployed for services.openminted.eu: --> 15372f7369137b964534e9ddb8e81fb150dc2677. You reported a different commit in a previous comment. I hope that are not any additional manual changes. I can't provide any help on this .....I didn't work on this part at all. |
|
This new issue about the workflow not being owned by the current user rings a vague bell. I have a feeling that this happened if the workflow had been added to Galaxy by a different user than the proxy was logging in as. In our deployment version (assuming it hasn't changed since I worked on the project) we ensured that the registry pushed workflows into Galaxy using the same user as the proxy was configured to use. Is this the case in your configuration (apologies that I've not followed the thread as closely as I no longer work on the project so you might have covered this already). There were also two bugs we found related to permissions in Galaxy galaxyproject/galaxy#4401 and galaxyproject/galaxy#5955 but both of those issues were fixed and should have been merged into the version you are using so I doubt they are relevant, but if the issue persists after checking that the usernames match then they might be a useful starting point for further debugging. |
|
@greenwoodma I checked the database and I had two users, yesterday during the worker reinstall I did some tests with the http headers and created a second user by mistake, which I was using to authenticate now, and was not the original. Reverting to the first user fixes the @galanisd the commit I'm using for the editor is the latest of the editor branch, but fixing the user issue it seems to work. |
We are having issues to succesfully deploy a working setup of the openminted platform following the current installation tutorial.
I'll describe here our setup a bit, and I have forked and uploaded the changes to the two repos, onto the bsc_changes branches.
https://github.com/bscopenminted/install-tutorial
https://github.com/bscopenminted/omtd-standalone-setup
The following is a brief description on how we setup the test nodes:
Current setup
General components
Web Frontend
Manager node:
OMTD worker node:
Current status
We can't:
In the case of create an application, I see a connection to the apache proxy as seen in the logs from the OMTD-worker node
However we can't see anything on the browser. It does seem that the galaxy instances are in fact empty, but there might be still something missing as nothing shows up in the browser, just: 'powered by Galaxy' banner.
Is there something that we might have missed during the setup, or some other tests/checks we can make?
Let me know if you need me to clarify or provide configs or other data.
Regards,
The text was updated successfully, but these errors were encountered: