Snapshot management

[f-galland]

Description

The Content Manager must be able to manage content snapshots in zip format, either for the initialization or offline management of the content.

The zip contains a huge JSON file in NDJSON format. This file can reach several gigabytes in size, so it needs to be handled efficiently, using JSON streaming.

Functional requirements

The Snapshot management module will take care of:

• Download the snapshot.
• Generate a hash for the snapshot.
• Unzip the snapshot.
• Index the content.

Implementation restrictions

• The snapshot is downloaded using the URL present in the internal index.
• The decompression of the content is done in Java code (no process, exec or similar, that runs a command on the operating system layer). Consider using an external library.
• The indexing of the content uses JSON streaming for efficiency. Use gson library (see Selection of Java libraries for JSON patch and JSON streaming #237).