-
-
Notifications
You must be signed in to change notification settings - Fork 283
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Codec unit tests #2035
base: main
Are you sure you want to change the base?
Add Codec unit tests #2035
Conversation
One area of feedback on the Codec API: it makes very little sense to me that the Codec API is async. Almost by definition Codecs are blocking, CPU-intensive code. They are not doing I/O. Why should their core methods be async? It should be the Pipeline's job to dispatch blocking Codecs calls to threads. Not the Codec itself. |
The reason why all codecs need to be async is because sharding is a codec, and the encode / decode operation of the sharding codec requires doing IO. I would love to see some formal separation between "codecs that read from storage" (i.e., just sharding) and "codecs that transform bytes in memory" (all the other codecs), but I'm not sure what this would look like. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love tests! thanks for this @rabernat
oops, I approved without noting that this is a draft. sorry for the noise. consider it a draft approval. |
While working on #2031 I became familiar with the new V3 Codec API and its peculiarities. And I saw that we don't yet have actual unit tests for the codecs. We have some tests in
tests/v3/test_codecs/
, but I'd call these more end-to-end tests, since they are creating Arrays.I think it's important for us to unit-test al of the important internal interfaces separately from end-to-end tests. This is particularly important for codecs, so we can guard against data corruption issues.
This PR is a step in that direction.
TODO: