Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[wicketd] Store TUF artifacts on disk instead of in memory (#3953)
Prior to this PR, `wicketd` could end up with quite a large heap because we were keeping several things in memory: 1. When a TUF repo was uploaded from `wicket`, we streamed it into memory 2. After extracting the TUF repo into a temporary directory, we neglected to free it until _after_ we'd ingested all its contents (pushing our high-water mark `$TUF_REPO_SIZE` higher than it needed to be) 3. We then read every extracted artifact from the TUF repo back into memory 4. We then further extract the contents of all RoT (tiny) and host OS (not at all tiny) artifacts to get at their inner artifacts (A/B images for the RoT; phase1 / phase2 blobs for OS images) We then kept all the in-memory artifacts from 3 and 4 around indefinitely. If a new TUF repo is uploaded, we would repeat the above 1-4, and only free the item 3 and 4 artifacts from the first repo _after_ successfully finishing all four steps on the second repo. This leads to some large heaps: 1. After uploading one TUF repo: ``` root@oxz_switch:/opt/oxide# pgrep wicketd | xargs pmap -S | grep heap 0000000002979000 3390440 3390440 rw--- [ heap ] 00000000D1873000 2392080 2392080 rw--- [ heap ] ``` 2. After uploading a second TUF repo: ``` root@oxz_switch:/opt/oxide# pgrep wicketd | xargs pmap -S | grep heap 0000000002979000 3390440 3390440 rw--- [ heap ] 00000000D1873000 3424296 3424296 rw--- [ heap ] 00000001A287D000 2392080 2392080 rw--- [ heap ] ``` Actually performing updates does not grow these sizes. --- This change makes some significant cuts on the above by shifting several things out of memory and into files on disk (currently all stored in tempdirs under `/tmp`, so still RAM, but we could move them to a physical disk if desired): 1. When receiving a TUF repo from wicket, we now stream it directly to a temporary file instead of into memory, and we extract from that temp file to a temp directory. 2. Instead of reading all the TUF contents into memory, we now copy them into a temporary directory (where "temporary" here is perhaps misleading: we keep this directory around until a new TUF repo is uploaded); we then read them (in a streaming fashion: not entirely into memory) from this directory on-demand while updates are performed Heap usage as of this PR after uploading one TUF repo: ``` BRM44220001 # pgrep wicketd | xargs pmap -S | grep heap 00000000029D7000 36216 36216 rw--- [ heap ] ``` After uploading a second TUF repo: ``` BRM44220001 # pgrep wicketd | xargs pmap -S | grep heap 00000000029D7000 48708 48708 rw--- [ heap ] ``` Neither uploading additional repos nor running updates grows this size. This change has some tradeoffs: 1. Ingesting an uploaded TUF repo is slower, but tolerably so (~10 seconds on main, ~13 seconds on this PR) 2. We now have more potential I/O errors, including possibly failing to open files that we expect to be in our temporary directory when they're needed during an update 3. We're using a nontrivial amount of space under `/tmp` (high water mark is during the ingest of a second TUF repo, where we would have all the extracted artifacts of both repos in separate tempdirs just before we remove one of them) but these are probably worth it in light of RFD 413 / #3943. Expanding on item 3 on `/tmp` usage slightly, in case it ends up being valuable: 1. When `wicketd` first starts, it is not using `/tmp`. 2. As a repo is uploaded, it is streamed into a file under `/tmp` (space used: roughly 1.2 GiB currently) 3. Once the upload is complete, the repo is unzipped into a temporary directory under `/tmp` (space used: the size of the repo .zip file plus the size of the unpacked repo; roughly 1.2 GiB + 1.2 GiB = 2.4 GiB currently) 4. The temp file from step 2 is deleted (space used: back down to the size of the unpacked repo; roughly 1.2 GiB) 5. wicketd creates a new temporary directory with a somewhat meaningful name (`wicketd-update-artifacts.ZZZZZZ`), then iterates through the artifacts in the repo, copying some and further extracting others; e.g., host and trampoline artifacts are expanded into their phase1 and phase2 components (space used: the size of the unpacked repo plus the size of the unpacked artifacts; roughly 1.2 GiB + 1.4 GiB = 2.6 GiB currently) 6. The unpacked repo temp directory from step 3 is deleted (space used: back down to the size of the unpacked artifacts: roughly 1.4 GiB currently) If a _second_ repo is then uploaded, we go through all of the above steps again, but we still have a `wicketd-update-artifacts.ZZZZZZ` from the first repo that is not removed until we get to the end of step 6, making our high water step 5 of a second (or later) repo upload, where we're using a total of (current values in parens): ``` size of extracted artifacts from first repo (1.4 GiB) + size of expanded second repo (1.2 GiB) + size of extracted artifacts from second repo (1.4 GiB) = maximal space used under /tmp (4.0 GiB) ``` --- Fixes #3943.
- Loading branch information
1 parent
77fb3be
commit 951651a