Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new GCN alerts and SCIMMA broker #84

Merged
merged 33 commits into from
Jul 30, 2024
Merged

Add new GCN alerts and SCIMMA broker #84

merged 33 commits into from
Jul 30, 2024

Conversation

martinjohndyer
Copy link
Member

@martinjohndyer martinjohndyer commented Jul 23, 2024

This PR contains some major upgrades for the sentinel alert listener and backend code:

  • First, the alert code is no longer built around the old VOEvent format and now allows the newer JSON and AVRO formatted alerts (see Process new-style GCN and IGWN alerts #82 for details). This was quite a pain, and there are still a few compromises - we still rely on IVORNs in the database, but that could use a bit of a reorganisation anyway (Rework notices database table, split out skymaps #85). Internally this did include splitting out the notice classes to project-specific versions, which make things a bit easier to deal with, but that shouldn't make any visible difference.
  • There was also a big reorganisation of the included test notices, which is why there are so many modified files. They're now also grouped by source, and include a few "bad notices" to test different cases.
  • Along with the move away from VOEvents, we can now process the new-style Einstein Probe alerts (Add Einstein Probe alerts #80). They don't include a trigger ID right now, which is awkward, but they apparently plan to add one so I'll wait for the next alert to see how it works before deploying this.
  • The actual purpose of this branch was to switch over to the SCIMMA hop-client backend (Move from GCN Kafka client to SCIMMA's Hop Client #81). Ironically, after implementing that it tuns out that the SCIMMA broker currently doesn't process the EP alerts which meant having to switch back to the NASA GCN broker (but still using hop). Aside from a few alerts on the first connection it seems to be working well, but it will need longer-term testing.
  • One of the Kafka features that is now implemented is the ability to "backdate" notices, ensuring we don't miss any if there is an outage (should help with e.g. More robust checking to prevent missing alerts #78). This does make things a little tricky on start up so we don't try to duplicate already-processed alerts, for now it just checks the IVORNs in the database but as above this will probably need improving alongside any database updates.
  • Incidentally, there are a few updates needed to GOTO-tile which I'll need to also make a PR for, to do with processing the IGWN embedded skymaps (https://github.com/GOTO-OBS/goto-tile/pull/96). For now I've added a skymap module into this, but I think that code for decoding binary FITS should be moved to the SkyMap class. Also we failed to save skymaps with COMMENT headers correctly, so that needed a fix.

Things to do before this can be merged:

@martinjohndyer martinjohndyer added -events_GRB Issues/PRs related to GRB events -event_handler Issues/PRs related to the event handler -sentinel Issues/PRs related to the sentinel alert listener labels Jul 23, 2024
@martinjohndyer martinjohndyer marked this pull request as ready for review July 26, 2024 14:00
@martinjohndyer
Copy link
Member Author

This seems to be working live, with the NASA brokers going through the hop client. Once SCIMMA adds the new GCNs then we can look at swapping over.

There are some annoying errors that appear in the logs but they seem to be harmless, I'll look at making another issue for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-event_handler Issues/PRs related to the event handler -events_GRB Issues/PRs related to GRB events -sentinel Issues/PRs related to the sentinel alert listener
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move from GCN Kafka client to SCIMMA's Hop Client Add Einstein Probe alerts
1 participant