Define API for reader and image_mapper #143
Conversation
For looping over a given dl1 table and a single dl1 event (charges, peak times and mask), we can now retrieve the 2D images (input of the CNNs) without init and running the dl1dh reader.
also create separate function for the trigger patches on R0 data
remove also apply IM functions because it can be now replace by the internal .map_image() function
If prefix camera_frame is in the file, the user should add this prefix to the config file.
added batch generator removed dl1dh transform dl1dh provide now a static batch and we get the relevant information about the labels in the data loader of ctlearn
now stored in dl0 monitoring tree
last dimension of sample was missing
replaced by new design
Mainly astropy table operations are now used to retrieve the exmaple identifiers for the stereo reading mode. Code base is therefore heavily reduced and operations are more efficient. Moved parameter settings outside the dl1dh. User can request to also the read dl1b parameters by passing a list of column names in the batch_generation() Removed init skip when pandas hdf5 with example identifiers is provided. It is not needed anymore since we are now fast and efficient with astropy tables and their operations. split transformation into sub-functions for better readability.
this is removing redundant code astopy table operations should be used a retrieved sum(), min or max etc.
Everything related to the selection of the subarray is done with ctapipe now Whenever a new file is processed, it checks the consistency of the SubarrayDescription to the reference which is the first provided file; this ensures that all files have the subarray.
It defines a reading API with two childs for reading in mono and stereo mode. Then, the Image and Waveform childs inherits from both child classes (mono and stereo). Finally, the trigger child only inherits from the mono child
this is somehow needed because we modify the batch Table for the Trigger subclass fix trigger subclass
TjarkMiener
added
enhancement
ctapipe
if obs id and event id is the same for different types, the code failed.
…ch should be a table use iterrows for iteration over the batch store also subarray trigger table as class attribute make get_tel_pointing also external
while processing real stereo data we do not have the true shower id in the batch
SST1M pipe writes real data in data format v5.0.0. We can support this version for real data only.
SST1M write the event-wise pointing directly in dl1 monitoring
polish docstrings
obs and event ids can be equal for protons and gammas; if they end up in the same batch the code break; this is fixing the issue by grouping also by particle type for simulation
|
I'm done with my last fixes which I wanted to include here @Pablitinho @mexanick. In order not to populate even more this PR (Sorry!), I'm suggesting to merge after a second look. We can fix with smaller dedicated PRs if needed. @nietootein would that be fine with you? |
polish docstrings
|
setup.cfg has a deprecated line. [tool:pytest] Action (remove pep8ignore line): If you dont remove this line you will have a warning in pytest like this: `============================================================================================================================ warnings summary ============================================================================================================================= -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html` After remove the pep8ignore line , the warning is out. |
|
Add an empty `from dl1_data_handler.reader import DLDataReader def test_inputs(): ` It is working for me. |
This PR suggests APIs for the reader and image_mapper. The configuration system and component scheme of
ctapipeis adopted. Besides we adoptastropytables for the batch generation. Reading is working in monoscopic and stereoscopic mode for DL1 images and R1 waveforms. The code properly process data with different array and telescope (divergent) pointings.A subclass designed for the adv. trigger system processing R0 waveforms will be added in a separated PR.
–––
Closes #31 #104