Virtual Neuro Machine (VNM)

[civier]

hi @ohbm/project-monitors: My project is ready

---

Project info

Title:
Virtual Neuro Machine (VNM)

Project lead:
Oren Civier, https://github.com/civier
[email protected]

Timezone:
UTC/GMT +10 hours

Hub:
Asia and Pacific

Description:
With the advent of container technology, cumbersome virtual machines have been reduced to a single container file that can be easily run on multiple-environments. Containers can be run both interactively (e.g., within a Virtual Machine on the cloud) and in batch processing (e.g., on HPC), thus contributing to reproducibility across stages of the analysis. The goal of this project is to generate a "virtual neuro machine" container, i.e. a large container that includes most popular neuroimaging tools, and can be used as a stand-alone VM. Although tools to generate neuroimaging software containers are already in place (e.g., neurodocker), continued maintenance of such a large container with complex library dependencies requires in-built testing capabilities, and especially so, of included GUI applications. This project will bring together people working on different environments and using different neuroimaging tools, in order to: 1) build a working container with the newest software versions available in June 2020, 2) develop in-built testing capabilities, to facilitate future container upgrades, 3) develop a standard "home screen" for the container, to provide common interface across host environments.

Link to project:
https://github.com/NeuroDesk

Mattermost handle:
orenciv

Goals for the OHBM Brainhack

1. build a working container with the newest software versions available in June 2020
2. have a routine for testing the container on an OpenStack virtual machine accessed via VNC

Good first issues:

1. choose a tool to test graphical applications running within a container
2. identify built-in testing routines provided by each of the main neuroimaging tools

Skills:

1. neurodocker (https://github.com/ReproNim/neurodocker), or equivalent tools (recommendations would be appreciated)
2. docker
3. singularity
4. expertise in neuroimaging packages included in neurodocker
5. expertise in graphical design (to design a "home screen" for the container, as graphical desktops cannot be easily containarised)
6. expertise in GUI development for Linux (any language)
7. continous integration (CI) for containers

Chat channel:
https://mattermost.brainhack.org/brainhack/channels/virtual-neuro-machine-vnm

Video channel:
Please have a look at the mattermost channel (pinned posts) for the URL of the video channel, or alternatively, email [email protected]

Project snippet for the OHBM Brainhack website:
title: "Virtual Neuro Machine (VNM)"
image: https://drive.google.com/file/d/18NiPnrzK3ylPKjiXOJyuP8eBTSZ4ic1D/view
details: "With the advent of container technology, cumbersome virtual machines have been reduced to a single container file that can be easily run on multiple-environments. Containers can be run both interactively (e.g., within a Virtual Machine on the cloud) and in batch processing (e.g., on HPC), thus contributing to reproducibility across stages of the analysis. The goal of this project is to generate a "virtual neuro machine" container, i.e. a large container that includes most popular neuroimaging tools, and can be used as a stand-alone VM. Although tools to generate neuroimaging software containers are already in place (e.g., neurodocker), continued maintenance of such a large container with complex library dependencies requires in-built testing capabilities, and especially so, of included GUI applications. This project will bring together people working on different environments and using different neuroimaging tools, in order to: 1) build a working container with the newest software versions available in June 2020, 2) develop in-built testing capabilities, to facilitate future container upgrades, 3) develop a standard "home screen" for the container, to provide common interface across host environments."

More information in this github issue

Project submission

Submission checklist

Once the issue is submitted, please check items in this list as you add under 'Additional project info'

Please include the following above (all required):

• Link to your project: could be a code repository, a shared document, etc. See here
• Include your Mattermost handle (i.e. your username). If you do not have an account, please sign up here.
• Goals for the OHBM Brainhack: describe what you want to achieve during this brainhack. See here.
• Flesh out at least 2 "good first issues": those are tasks that do not require any prior knowledge about your project, could be defined as issues in a GitHub repository, or in a shared document, cf here.
• Skills: list skills that would be particularly suitable for your project. We ask you to include at least one non-coding skill, cf. here.
• Chat channel: A link to a chat channel that will be used during the OHBM Brainhack. This can be an existing channel or a new one. We recommend using the Brainhack space on mattermost, cf. here.
• Video channel: Please create a video channel that will be used during the OHBM Brainhack and share it in your chat channel above. This can be an existing channel or a new one. For instance a jitsi meet room, cf. here.

You can also include information about (all optional):

• Number of participants, cf. here
• Twitter-size summary of your project pitch, cf. here
• Provide an image of your project for the OHBM brainhack website
• Set up a kanban board on your repository to better divide the work and keep track of things, cf here
• Project snippet for the OHBM Brainhack website, cf. here

We would like to think about how you will credit and onboard new members to your project. We recommend reading references from this section. If you'd like to share your thoughts with future project participants, you can include information about (recommended):

• Specify how will you acknowledge contributions (e.g. listing members on a contributing page).
• Provide links to onboarding documents if you have some.

[welcome]

(Image: CC-BY license, The Turing Way Community, & Scriberia. Zenodo. http://doi.org/10.5281/zenodo.3332808)
🎉 Welcome to the 2020 OHBM Hackathon! 🎉 We're really excited to have your input on this repository and welcome your idea! 💖 If you haven't done so already, please make sure you check out our Code of Conduct. Did you issue a new hackathon project? Please check if you have provided all relevant information and an image for the website.

[Remi-Gau]

Hey 👋

Looking good. 👍

Added some labels. Check if they are OK with you.

git-3 is the level we use to denote skills in continuous integration: is that OK with you?

[civier]

Yes. All good. Thank you. Oren

…

-- Oren Civier Neuroimaging Informatics Fellow Swinburne Neuroimaging | National Imaging Facility https://www.swinburne.edu.au/neuroimaging https://anif.org.au/ Swinburne University of Technology Hawthorn Campus, Advanced Technologies Centre, Room 914 +61 3 9214 4628 | +61 4 3101 6603 Teams: [email protected] https://twitter.com/orencivier

On Sat, Jun 13, 2020 at 5:08 PM Remi Gau ***@***.***> wrote: Hey 👋 Looking good. 👍 Added some labels. Check if they are OK with you. git-3 is the level we use to denote skills in continuous integration: is that OK with you? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7FUI7ZJDAA7E6GC5NSHELRWMQW3ANCNFSM4N4EJU4A> .

[civier]

I have some questions to the neurodocker people (i.e. gods). I wonder if they can give us a hand? ... Let's see :-)

1. NeuroDebian
   Neurodocker still offers a limited range of tools. For our specific goal, it might be possible to use a Debian base, and then install the rest of the tools (not available on neurodocker) through NeuroDebian. Would you recommend that?
   One issue that I noticed is that NeuroDebien stopped supporting the new versions of some tools (like FSL), and users are referred instead to neurodocker (see Yaroslav's comments at the bottom of http://neuro.debian.net/pkgs/fsl-complete.html). Is there a reason for that? Are there any plans for the NeuroDebian and neurodocker projects to collaborate more closely in the future?

2. Testing
   I saw that there are testing procedures on the neurodocker github, but that they are very elementary. Also, for my understanding they are only used for CI, correct?
   Did you have plans to provide a testing suite that will be run by users, on their machines? (it's obvious that not all software combinations could be tested at development). Any inputs you have would be appreciated.

3. OpenGL software implementation
   Running my resulting containers on VM on the cloud, I found out that many applications require GPUs (e.g., mrview). That's a pity, because often acceleration is only required for very specific functions of the GUI, and I think that for most usage cases it'd be fine to work with an OpenGL software implementation. I wonder -- did you try neurodocker with any software implementation of OpenGL, such as OpenSWR?

4. Home screen / Application menu
   Did you look into adding a "home screen" to neurodocker (or "application menu" of sorts). This will enable graphical navigation to the software installed, helpful for novice users. That is another thing I want to work on during the hackathon, and which could potentially be incorporated directly into neurodocker.

5. Matlab
   Unfortunately, many people still need full-fledged MATLAB for neuroimaging analysis. Do you have plans to include it, with instructions on how users can provide their own license? Is there a way to install an unlicensed MATLAB via the command line, so it will be included in the container?

[satra]

NeuroDebian
Neurodocker still offers a limited range of tools. For our specific goal, it might be possible to use a Debian base, and then install the rest of the tools (not available on neurodocker) through NeuroDebian. Would you recommend that?
One issue that I noticed is that NeuroDebien stopped supporting the new versions of some tools (like FSL), and users are referred instead to neurodocker (see Yaroslav's comments at the bottom of http://neuro.debian.net/pkgs/fsl-complete.html). Is there a reason for that? Are there any plans for the NeuroDebian and neurodocker projects to collaborate more closely in the future?

the neurodebian + fsl issue is something @mih can probably talk about, but it will likely require somebody from the community to create the necessary updates. in general i think neurodebian updates for some software, like freesurfer and fsl will require some community contribution to package and test the packaging. these are also complex in terms of licensing. but i know afni was updated recently, so there is hope yet for others. i do like neurodebian as a source because it minimizes the install footprint through shared libraries.

Testing
I saw that there are testing procedures on the neurodocker github, but that they are very elementary. Also, for my understanding they are only used for CI, correct?
Did you have plans to provide a testing suite that will be run by users, on their machines? (it's obvious that not all software combinations could be tested at development). Any inputs you have would be appreciated.

btw default neurodocker does not test the container, but we have been developing a separate project TestKraken to do exactly that. TestKraken uses neurodocker and pydra under the hood and you can create various tests that you can test in the environments. pinging @djarecka - who develops it.

OpenGL software implementation
Running my resulting containers on VM on the cloud, I found out that many applications require GPUs (e.g., mrview). That's a pity, because often acceleration is only required for very specific functions of the GUI, and I think that for most usage cases it'd be fine to work with an OpenGL software implementation. I wonder -- did you try neurodocker with any software implementation of OpenGL, such as OpenSWR?

the last time i tried something inside neurodocker was to get VTK working and that was done here: https://github.com/neurohackweek/jupyterhub-docker (note the addition of mesa and dri in the early installs) @pwighton was looking into this more recently (and i completely forgot to mention the jupyter container to him). neurodocker can install vnc, but would likely require mesa and others to actually do the opengl rendering. in our compute cluster we have installed virtualgl to help support opengl rendering on compute nodes. perhaps adding that to neurodocker will help with that issue.

Home screen / Application menu
Did you look into adding a "home screen" to neurodocker (or "application menu" of sorts). This will enable graphical navigation to the software installed, helpful for novice users. That is another thing I want to work on during the hackathon, and which could potentially be incorporated directly into neurodocker.

that would be great. we have a json file inside the container that is the json representation of the commandline used to create the container, and should therefore tell people what is installed. but i can easily see some specific commands to be automatically added to the neurodocker container. it would also be nice to create a graphical interface to create the neurodocker container, and to extend neurodocker to do some form of dependency checking (a much harder task)

Matlab
Unfortunately, many people still need full-fledged MATLAB for neuroimaging analysis. Do you have plans to include it, with instructions on how users can provide their own license? Is there a way to install an unlicensed MATLAB via the command line, so it will be included in the container?

we only include the matlab common runtime as an install process. regarding how to connect to matlab perhaps this singularity container, which was based on something we did on our cluster, can provide some inspiration. https://github.com/researchapps/matlab-sherlock

pinging @stebo85 who had also reached out regarding neurodocker development.

[civier]

Thanks @satra for all the great input. We have only 3 days, but we'll see what we can do.
BTW, what do you mean by:

" ... and to extend neurodocker to do some form of dependency checking (a much harder task)"

It can mean different things, so I want to make sure I understand you.
Oren

[satra]

all of this doesn't have to be done in 3 days :)

" ... and to extend neurodocker to do some form of dependency checking (a much harder task)"

It can mean different things, so I want to make sure I understand you.

currently there is no check in neurodocker that when you install two different packages that their underlying requirements won't clash. presently this is ok because of a small set of packages. but this is exactly where every package manager eventually runs into trouble. this is just a note that neurodocker was intended originally to be a helpful tool, not become a package manager. keeping that line somewhat clear to consumers would be good.

[jhlegarreta]

@civier I removed the video link to avoid unexpected events in the video channel. Would you add and pin that information to the mattermost channel, please? Thanks.

[civier]

Sure. But can you explain me how to do it? I'm not familiar with mattermost. Also -- what if a participant doesn't use mattermost? or is it obligatory? Oren

…

On Mon, 15 Jun 2020, 06:37 Jon Haitz Legarreta Gorroño, < ***@***.***> wrote: @civier <https://github.com/civier> I removed the video link to avoid unexpected events in the video channel. Would you add and pin that information to the mattermost channel, please? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7FUIY6I273MCNDYAJPHDLRWUYGZANCNFSM4N4EJU4A> .

[jhlegarreta]

Sure. But can you explain me how to do it? I'm not familiar with mattermost.
Also -- what if a participant doesn't use mattermost? or is it obligatory?

Although you could choose discord, gitter or slack, we strongly encourage using mattermost for a few reasons:

• A workspace called brainhack already exists and many, many people in the computational neuroscience/neuroimaging field are already using it. In fact, the community transitioned from slack to mattermost about a year ago.
• Note that the volunteers from the OHBM brainhack organization are already on mattermost, and in order to get some help from them using a mattermost channel seems a reasonable choice. This allows them to centralize channels and be more efficient when monitoring them.
• Although this is the case of the other platforms as well, if the project continues to thrive beyond the OHBM hacktrack, you will be able to easily keep using the same communication channel, and the community will be able to have a look a it. Taking into account that the ability to monitor different platforms and channels is limited for all of us, you will increase the chances of communicating with other folks.

As far as I can see, you already signed up for the brainhack mattermost. and your project has already a channel: https://mattermost.brainhack.org/brainhack/channels/virtual-neuro-machine-vnm

You can post a message in the channel with the video link and pin it so that it stays visible 👀 .

Thanks !

[civier]

I have a question @jhlegarreta about your comment above:

@civier I removed the video link to avoid unexpected events in the video channel. Would you add and pin that information to the mattermost channel, please? Thanks.

Isn't it enough to write something like "channelname on Jitsi" or include the Jitsi URL as an image? How sophisticated are videobombers anyway?

[jhlegarreta]

Isn't it enough to write something like "channelname on Jitsi" or include the Jitsi URL as an image?

Yes, a jitsi url in the mattermost channel is considered to be enough and having some degree of safety (requires already a mattermost, acount, etc.) for the OHBM hacktrack purposes.

[complexbrains]

Dear @civier Thank you very much for your project submission. Your project looks ready. Welcome aboard! 🤗 🎊

[stebo85]

Hi @civier,
This is a great project and very useful :) We are currently working on these problems for CVL where we use singularity containers build using neurodocker (https://github.com/CAIsr/caid) and I built transparent singularity that integrates these containers into the module system (https://github.com/CAIsr/transparent-singularity). Looking forward to the hackathon :)

Cheers
Steffen

[civier]

Proposed outputs of the VNM project (asterisks are for extensions that can serve @sbollmann_mri CVL project extension):

1. Adding more recipes to neurodocker
   1.1 A list of things that will be useful
   1.2 Adding X new packages
   1.3 * Add options to generate stand-alone versions of applications to https://github.com/CAIsr/caid?

2. Graphical UI
   2.1 Choose tool (consult with @cnolan on Mattermost)
   2.2 Prepare a list of GUI commands of each application (can consult https://github.com/CAIsr/transparent-singularity)
   2.3 * Possibility to run commands through transparent Singularity

3. Testing
   3.1 Check if we can use NeuroLibre Juoyter notebook pipelines for testing
   3.2 Check if we something similar exists if we have tests written in Matlab
   3.3 Write tests for popular MATLAB software like SPM, CONN, etc.

[yarikoptic]

FWIW, another point of "integration" between neurodocker and NeuroDebian is the --ndfreeze option in neurodocker to take advantage of https://github.com/neurodebian/neurodebian/blob/master/tools/nd_freeze which is shipped within any neurodebian based container or could be just apt-get install neurodebian-freeze 'ed. This allows to make container images reproducible (as far as apt package manager is concerned).

[pwighton]

Re: OpenGL software implementation, @rudolphpienaar has made some progress getting FreeSurfer's FreeView running in a docker container: https://github.com/FNNDSC/pl-fshack/wiki/1.0-Running-freeview (but not sure if software rendering is supported)

[civier]

FWIW, another point of "integration" between neurodocker and NeuroDebian is the --ndfreeze option in neurodocker to take advantage of https://github.com/neurodebian/neurodebian/blob/master/tools/nd_freeze which is shipped within any neurodebian based container or could be just apt-get install neurodebian-freeze 'ed. This allows to make container images reproducible (as far as apt package manager is concerned).

Thank you @yarikoptic I will definitely look into it.
Oren

[civier]

Re: OpenGL software implementation, @rudolphpienaar has made some progress getting FreeSurfer's FreeView running in a docker container: https://github.com/FNNDSC/pl-fshack/wiki/1.0-Running-freeview (but not sure if software rendering is supported)

Thank you Paul. We will definitely try to get freeview working in VNM. Are you interested to contribute to our project? Check our main repository at github.com/NeuroDesk/vnm

[stebo85]

@civier Freeview is already working in VNM :)

[civier]

@civier Freeview is already working in VNM :)

I've just seen that you have an updated freesurfer.yaml in the master branch, but I'm still using 0.7.0
@kaczmarj, when do usually publish a new release of neurodocker? Are you doing any testing on each release, or simply release when enough updates have accomulated?

[pwighton]

Very cool @civier and @stebo85! Freeview isn't working for me, but I'll follow up in the main repo.