New Feature: Data Product Saving and Ground Reconstruction

[nabcouwer]

This is a big lift in terms of implementation, and I think its something that has been on the minds of some F' devs, so if there is discussion elsewhere, happy to merge with that. But it's not in the call for comments yet.

It would be very nice to support some kind of saving of data products. By that I mean saving structured data or serializable types in such a way that they end up in a file that can be downlinked and, on the gds side, expanded into something like xml, where you can see each instance of the type that was written to over time.

As an example application, suppose you are not always able to receive the stream of events generated by components, to active logger, to framer/downlink, to byeStreamDriver/comm, because your system has limited communication windows. So you split the log events somewhere (either before getting to ActiveLogger, or within ActiveLogger), also send them to a Generic serial port on the Data Product saving component (DPSC), and the DPSC eventually writes that data to a file. THat file can be downlinked using existing infrastructure. Once on the ground, the file can be read and used reconstruct events that may not have been able to be communicated,

This would be in contrast to https://github.com/nasa/fprime/tree/devel/Svc/BufferLogger , which seems to save raw bytes, necessitating custom reading of the data. (Please correct my if I'm wrong or missing a Svc component that is getting what I'm after) Maybe the DPSC feeds into a BufferLogger.

Other possible applications:

• Saving Channelized telemetry for later downlink
• Saving high-rate data that would not be appropriate for low-rate chanelized tlm or events (such as sensors, IMU, FTS, motor feedback)
• Saving data in a defined way (sure you could save acquired images as some file, but you could also feed them into this scheme, put system metadata with each file, etc)

Challenges:

• Is opening files, writing to files as you go, and eventually closing files the way you would do this? Or write to very large memory spaces and writing to files in big chunks?
• How do you know when to open a file / start a buffer for writing?
  • Upon client component request?
• How do you know when to close a file / stop a buffer?
  • Upon filling up?
• In the case of long-lived writing (saving events or telemetry), do you stop writing to one file and start a new file? When?

More detailed design ideas:

• DPSC provides ports for
  1. Input Generic Serial ports (several) to receive data product elements
  2. Input port for requesting to open a data product that will receive Generic Serial port data on port number N
  3. Output response to previous response, indicating yay/nay
  4. Input port for requesting to close data product
• When DPSC receives the port data, the port data is saved to file in such a way as to allow ground reconstruction
  • There is a concept of type ID in serialized types. According to FpConfig, that isn't working yet, but once it is, that could be used and stored, maybe with delimiters and length indicators as well. This would work for ports that send serialized types, and saving those serialized types, but ports don't have unique ids. Maybe they could have them.

Beyond creating this component, we'd also want to create a ground application that can covert the raw data into human readable xml or similar.

[nabcouwer]

I'm now realizing that when @LeStarch mentioned https://github.com/nasa/fprime/discussions/categories/call-for-comments , he wasn't soliciting ideas, but feedback on what F' has in the near-term plans. So I may have been barking up the wrong tree when writing this, but it is still a feature I'd love to see added / would be interested in contributing to (but don't have all the context and bandwidth to do by myself).

[LeStarch]

@nabcouwer call-for-comments is specifically for near term implementation for our F´  team. However, discussions like this help us form that near term plan....so I think you are on-pont to post here.

[LeStarch]

@timcanham you were thinking about something like this, yes? Could you comment/post here with your thoughts?

[timcanham]

Yes - I have a design concept that I'll present in the next few weeks. I could post the essentials of it here before then if you like.

[LeStarch]

@timcanham in some way we'll need to capture that design here. You can post quick notes here, or we can capture your presentation and post that.