Implement missing JSON export for OpenSearch version #871
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Requires redirecting log data to stderr.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
That adds the as-of-yet unimplemented JSON export to OpenSearch.
There is a new option to hand in '-' as the filename to dump the data to standard out. This is needed to pack these files on the fly. To make that correctly work in turn, log information is now directed to standard error.
Note that these export dump the JSON that is used to import documents in the ES database. That means that the dumps are not compatible when done from different versions of Photon. To account for that, I have now added a "header" document which contains a version field and may get other global information in later versions.
The dumped documents now also include the document ID. This should make it possible to recreate a database exactly from the dump. A bzipped dump of the planet has only 11GB. So this gives us an interesting opening for importing custom filtered databases or databases with extra indexes for structured search, for example. Time to dust off PR #438.