Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document hotfix process for IDAaaS and other platforms #1296

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Document hotfix process for IDAaaS and other platforms #1296

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4482ff2
begin first draft
cmarooney-stfc Sep 27, 2023
7990e3c
first draft continue 1
cmarooney-stfc Sep 27, 2023
d0ee909
add to draft 2
cmarooney-stfc Sep 27, 2023
6bf0f1c
draft 3
cmarooney-stfc Sep 27, 2023
d967bd9
draft 4
cmarooney-stfc Sep 27, 2023
69adf8d
draft 5
cmarooney-stfc Sep 27, 2023
41b4c3a
revision requests
cmarooney-stfc Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 123 additions & 50 deletions documentation/smg/11_ISIS_release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,79 +1,152 @@
# ISIS Release Process
# SMG-11: ISIS Release Process

Horace is deployed separately to a number of systems across ISIS
Horace has been deployed separately to a number of systems across ISIS. The current ones are:

## ISIS Compute
- IDAaaS: This is currently the normal access point for users of Horace in ISIS.
- SCARF: This is an alternative HPC environment. In contrast to the deployment on IDAaaS, the current SCARF deployment is experimental, and its parallel capabilities are more limited.
Details of this are given below, including contacts for improving the installation.

#### Contact

FBU IT support team: [email protected]


#### Deploy Location
The physical location of the code on ISISCOMPUTE file system is:

`/home/isis_direct_soft/Horace_git`
`/home/isis_direct_soft/Horace_bugfix`
`/home/isis_direct_soft/Herbert_git`
`/home/isis_direct_soft/Herbert_bugfix`

By agreement, this location is accessed by code users through symbolic links, created in `/usr/local/mprogs` folder.

The release link are pointing to the user's code base, namely:
`/usr/local/mprogs/Herbert` -> `/home/isis_direct_soft/Herbert_git/herbert_core`
`/usr/local/mprogs/Horace` -> `/home/isis_direct_soft/Horace_git/horace_core`

hiding technical areas from inexperienced users.

To ensure compatibility and smooth user's experience, all code operations should be performed with the simulink path-es, which will remain constant regardless of physical location of the code.
This document is primarily concerned with the process of upgrading Horace and releasing new Horace versions on IDAaaS. However, details for how we might update SCARF are also given for completeness.

## IDAaaS

Code base for bugfix purposes is linked directly to the git repository for support team convenience, namely:
`/usr/local/mprogs/Herbert_bugfix` -> `/home/isis_direct_soft/Herbert_bugfix`
`/usr/local/mprogs/Horace_bugfix` -> `/home/isis_direct_soft/Horace_bugfix`
IDAaaS is the cloud computing service "ISIS Data Analysis as a Service" provided by Scientific Computing for ISIS users.
It is accessed from https://isis.analysis.stfc.ac.uk/, from where you can login to the service. This also gives instructions for new users; similar instructions are also available in the
registration part of the dialog on the login page. Detailed use of IDAaaS and Matlab on IDAaaS is outside the scope of this document.

#### Deploy Process
#### Contact

As soon as release branch is tagged and created in the repository, manual build/release process is preformed in the private area of an ISISCOMPUTE or iDaaaS machine to obtain release artefacts for the computational service. It is also possible to pull the release artefacts from Jenkins builds, though this process needs thorough testing.
The extraction of the release artefacts will be automated in a future, as [iDaaaS testing pipeline](https://github.com/pace-neutrons/Horace/issues/271) is implemented. The releaser then manually checks out the release branch into `Herbert_git/Horace_git` areas above and copies the release artefacts into appropriate places of the code tree.
IDAaaS support team : [email protected]

#### Main Deploy Location

Various versions and components of Horace are deployed in a number of local repositories under the path `/usr/local/mprogs`.

The current physical disk location of these Horace repositories are given at the end of this document.
As this may change subject to development of the underlying IDAaaS system structure, the path `/usr/local/mprogs` provides symbolic links to the repositories' current locations.
Note that the repository symbolic link names may not always match exactly the directory name of the underlying repository location.
To ensure compatibility and smooth user's experience, all code operations should be performed with these symbolically linked paths,
which will remain constant regardless of the actuall physical location of the code.

The principal repositories are
- Horace (underlying name: Horace_git): this is the directory corresponding to `horace_core` in a clone of the full Horace github repository.
- Herbert (underlying name: Herbert_git): this is the directory corresponding to `herbert_core` in a clone of the full Horace github repository.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Herbert repository. Still used but outdated for Horace 4

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abuts Not clear on this. looking again at the directories in /usr/local/mprogs, the Horace slink point to Horace_git/horace_core, while the Herbert slink points to Herbert_git/herbert_core. The Horace_git path is a clone of the current github repo for Horace, including horace_core and herbert_core, but that herbert_core is not accessed from mprogs. The herbert_core accessed from mprogs is in Herbert_git, which appears to be a cut-down version of the release 3.6.3 branched repo. So it's not clear to me what Herbert content IDAaaS users are getting.

Copy link
Member

@abuts abuts Sep 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Herbert (underlying name: Herbert_git): this is the directory corresponding to herbert_core in a clone of the full Horace github repository.

actually not -- Herbert_git is the clone of Herbert repository, used together with Horace_git repository if Horace_git repositoy contains old version of the Horace, which uses separate Herbert.

Now you probably see why I do not want to write this. I would not be able to understand this sentence.


This path defines the folder containing the Horace initialisation script Horace_on.m, which on execution adds all the folders Horace needs to the
MATLAB search path, and initialises all other Horace variables needed (e.g. Horace configuations). The path also contains whe worker script which
invokes the horace_on script and initialises Horace for all parallel MATLAB sessions.

An administrative script (not write-accessible by Horace developers) `/var/daaas/dynamic/matlabsetup.m` is executed on start-up of MATLAB user sessions
on IDAaaS machines, and invokes `horace_on` for user sessions dedicated for using Horace.


#### Simple Deploy Process

The version deployed is the one on the master branch of the github repository https://github.com/pace-neutrons/Horace, and the process of deployment for a simple fix is the same as for
fixing or updating the master branch in any development process:
- clone the repository, either to IDAaaS or your own local machine
- create and checkout a new branch referencing a ticket for the changes
- make the changes, and add them and commit them to your repository clone
- push the changes, which should pass all tests and be approved by reviewers before merging into the master branch
- if all tests pass and all reviewers agree, merge the pull request.

The final additional step is to make the changes in github available on IDAaaS; so far they have only been made on github.
- On IDAaaS, open a terminal and change directory to the Horace (Horace_git) repository named above.
- Check the repository's status with the `git status` command, You should find that it is a git repository checked out on the `master` branch.
For other outputs, please consult with the Horace developers.
- Bring the changes into the IDAaaS repository with the command `git pull`.
- If you have also made changes in the `herbert_core` directory, repeat these steps in the Herbert (Herbert_git) repository.
Although Horace and Herbert have been merged into a single repository on github, they are stilll represented by two separate repositories on IDAaaS.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

old versions are still present on `IDAAAS until expected to be used with old versions of Horace

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abuts As noted above on my comment on line 36, the mprogs/Horace slinks to repo Horace_git (its folder horace_core) which is the latest master or branch from it, while mprogs/Herbert slinks to a different repo Herbert_git (its folder herbert_core) which is branched to the v3.6.3 release. In general I am confused about how Horace and Herbert are paired up.


#### Safe rapid deploy process

The above deployment method runs some risk of having changes made to the code base which could break on IDAaaS. In particular
there is no test of the changes on IDAaaS itself. If the change has been made due to an error found by a user on an IDAaaS instrument, it is also
desirable to have the user test the fix before merging. Consequently alternative repositories are available to isolate changes until they
are sufficiently tested and approved, and to enable rapid immediate testing on IDAaaS itself. These correspond to the Horace and Herbert repositories
in the main deployment process above, and are:
- Horace_bugfix
- Herbert_bugfix
(these have the same underlying and public names.)

The developer should decide whether to make changes on IDAaaS or on their own machine and pass the changes into the IDAaaS repositories.
The latter case is similar to the main deployment process described above; here we will assume that the changes are being done on IDAaaS
for immediate consumption.
- Open a terminal and go to the relevant repositories (Horace_bugfix and/or Herbert_bugfix).
- check that the repository/ies are checked out to master. If it is otherwise checked out, someone else is doing this process;
ensure you do not overwrite their changes.
- Checkout a branch for your ticket and issue e.g. branchname 6666_myname_hotfix_user_issue. You should only need to create the branch in the
first repository you modify; the other will just need a checkout.
- make your changes.
- Open Matlab. It should be initialised with the latest Horace-3 version (3.6.3). To change to the latest Horace-4 with your changes, input
```
horace_4on("/usr/local/mprogs/Horace_bugfix")
```
- Run such tests as you need to ensure that the user's problem is fixed. The user can carry on working with your bugfix area until github is updated and propagated.
- As with the main deployment process, now add, commit and push your changes to github, making a pull request.
- Github via Jenkins should run the tests, and reviewers should approve your changes.
- when tests pass and reviewers approve, merge your changes into master on github.
- Now do a git pull on ALL the repositories you have touched; Horace/Herbert to make sure your changes have updated IDAaaS; and Horace_bugfix/Herbert_bugfix
to ensure that these bugfix areas are ready for the next fix.

Note that if you are not changing versions at this point, the conventional `horace_on("/usr/local/mprogs/horace_bugfix")` may be used to activate Horace
with your fixes on the modified branch.

#### Stable and historical versions of Horace

The following additional repositories in /usr/local/mprogs hold earlier or current release versions
- Horace_3_6 (underlying name Horace_3.6): the current stable release version which is being upgraded to Horace-4
- Horace 3_2 (underlying name Horace; note this is not the same as the symbolically-linked name Horace): preceding historically stable version; not expected to be used
Copy link
Member

@abuts abuts Sep 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just asked to remove this so, may be removed

I mean Horace_3_2, and Herbert_3_2 -- this simulinks will be removed, so should not be described

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 3.2, guessing it's fine if you remove it. I can remove references. For 3.6, i thought that was the default choice if not switched to v4 so checking you're not going to remove that

- Herbert equivalents of the above with the same naming conventions


#### General deploy comments

The process is currently absolutely the same as on ISISCOMPUTE. The real ISISCOMPUTE and virtual IDAaaS machines are currently share the same OS version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no ISISCOMPUITE

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

and the same file system, so the deployment performed on one system currently means that the same changes are then deployed on the other;
only one system needs to be changed.

A disadvantage of IDAaaS is currently the absence of the system-wide cmake installation, so a person, who wants to make a release on iDaaaS machine needs to do local cmake installation.

#### Notes

The script which automates these operations and performs internal releases as one-click operation will be developed in the nearest future.
Write access to code repository on iDaaaS is granted by the IDAaaS team on request. Currently the list of people who have write access coincides with
the members of **mslice** group on ISISCOMPUTE. This may change in a future.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no isiscompute

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes


The members of **mslice** group on ISISCOMPUTE have write access to release code areas. Alex Buts or FBU IT support team can currently add users to **mslice** group on ISISCOMPUTE. Normal users have read-only access to the code area, sufficient to use the code in their computations.

## IDAaaS
## SCARF

#### Contact

iDaaaS support team : [email protected]

#### Deploy Location
TBD

physical location is the same as on ISISCOMPUTE, except ISISCOMPUTE file system is currently mounted on iDaaaS at `/mnt/nomachine`. This can change in a future, but by agreement with iDaaaS team, the symbolic links in `/usr/local/mprogs` will always point to a physical location of appropriate Horace/Herbert repository clones.
#### State of deployment

To ensure compatibility and smooth user's experience, all code operations should be performed with the simulink path-es, which will remain constant regardless of physical location of the code.
The deployment on scarf is currently experimental. Generally speaking, user can make local installation of Horace/Herbert as described on [Horace installation pages](http://horace.isis.rl.ac.uk/Download_and_setup) and [SCARF user pages](https://www.scarf.rl.ac.uk/)

#### Deploy Process
Horace parallel capabilities for SCARF are currently implemented on the basis of Matlab MPI framework only. The installation of appropriate cluster component on SCARF and SCARF cluster configuration should be agreed with Jon Roddom<[email protected]> or other members of SCARF support team.

The process is currently absolutely the same as on ISISCOMPUTE. The real ISISCOMPUTE and virtual IDAaaS machines are currently share the same OS version and the same file system, so the deployment, performed on one system currently means the deployed on another and have to be done only on one system.
As soon as appropriate SCARF cluster configuration is available to user's Matlab, it must be made default in Matlab parallel computing toolbox settings. Then Maltab MPI framework (parpool framework) will submit and run parallel jobs on SCARF.

The disadvantage of iDaaaS is currently the absence of the system-wide cmake installation, so a person, who wants to make a release on iDaaaS machine needs to do local cmake installation.
## Current disk locations for the repositories

#### Notes
#### Deploy Locations

The write access to code repository on iDaaaS is granted by iDaaaS team on request. Currently the list of people, who has the write access coincides with the members of **mslice** group on ISISCOMPUTE. This may change in a future.
The repositories with the Horace code on iDAAaS file system are currently located (28.9.2023) under:
`/mnt/ceph/auxiliary/excitations/isis_direct_soft/`

the symbolic links in `/usr/local/mprogs` will always point to a physical location of appropriate Horace/Herbert repository clones.

The physical locations of the code are:

## SCARF
- `.../isis_direct_soft/Horace_git`
- `.../isis_direct_soft/Horace_bugfix`
- `.../isis_direct_soft/Herbert_git`
- `.../isis_direct_soft/Herbert_bugfix`
relative to the ceph location given above.

#### Contact

The deployment on scarf is currently experimental. Generally speaking, user can make local installation of Horace/Herbert as described on [Horace installation pages](http://horace.isis.rl.ac.uk/Download_and_setup) and [SCARF user pages](https://www.scarf.rl.ac.uk/)
By agreement, this location is accessed by code users through symbolic links, created in `/usr/local/mprogs` folder.

Horace parallel capabilities for SCARF are currently implemented on the basis of Matlab MPI framework only. The installation of appropriate cluster component on SCARF and SCARF cluster configuration should be agreed with Jon Roddom<[email protected]> or other members of SCARF support team.
To ensure compatibility and smooth user's experience, all code operations should be performed with the symbolically linked paths, which will remain constant regardless of physical location of the code.

As soon as appropriate SCARF cluster configuration is available to user's Matlab, it must be made default in Matlab parallel computing toolbox settings. Then Maltab MPI framework (parpool framework) will submit and run parallel jobs on SCARF.