Out of memory

[reynir]

I triggered an out of memory exception while trying to deploy a ~17 MB unikernel with no arguments. The mollymawk unikernel itself runs with "only" 128 MB.

console 2024-10-18T09:54:08-00:00: 2024-10-18T09:54:08-00:00: [ERROR] [application] error exception Out of memory while processing request ((method "POST") (target "/unikernel/create") (version "HTTP/1.1") (headers 
console 2024-10-18T09:54:08-00:00:                                                  (("Priority" "u=0")
console 2024-10-18T09:54:08-00:00:                                                  ("Sec-Fetch-Site" "same-origin")
console 2024-10-18T09:54:08-00:00:                                                  ("Sec-Fetch-Mode" "cors")
console 2024-10-18T09:54:08-00:00:                                                  ("Sec-Fetch-Dest" "empty")
console 2024-10-18T09:54:08-00:00:                                                  ("Cookie" "molly_session=REDACTED")
console 2024-10-18T09:54:08-00:00:                                                  ("Connection" "keep-alive")
console 2024-10-18T09:54:08-00:00:                                                  ("Sec-GPC" "1")("DNT" "1")
console 2024-10-18T09:54:08-00:00:                                                  ("Origin" "https://mollymawk.robur.coop")
console 2024-10-18T09:54:08-00:00:                                                  ("Content-Length" "17617146")
console 2024-10-18T09:54:08-00:00:                                                  ("Content-Type" "multipart/form-data; boundary=---------------------------120320325341882077743984541787")
console 2024-10-18T09:54:08-00:00:                                                  ("Referer" "https://mollymawk.robur.coop/unikernel/deploy")
console 2024-10-18T09:54:08-00:00:                                                  ("Accept-Encoding" "gzip, deflate, br, zstd")
console 2024-10-18T09:54:08-00:00:                                                  ("Accept-Language" "en-US,en;q=0.5")
console 2024-10-18T09:54:08-00:00:                                                  ("Accept" "*/*")
console 2024-10-18T09:54:08-00:00:                                                  ("User-Agent" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0")
console 2024-10-18T09:54:08-00:00:                                                  ("Host" "mollymawk.robur.coop"))))

[reynir]

Trying again with a stripped unikernel image (9.1 MB) it OOM'd in a different place:

console 2024-10-18T10:13:06-00:00: Fatal error: exception Out of memory
console 2024-10-18T10:13:06-00:00: Raised by primitive operation at Stdlib__Buffer.resize in file "buffer.ml", line 87, characters 19-40
console 2024-10-18T10:13:06-00:00: Called from Stdlib__Buffer.add_string in file "buffer.ml", line 178, characters 34-46
console 2024-10-18T10:13:06-00:00: Called from Multipart_form.RAW.parser.(fun).choose in file "duniverse/multipart_form/lib/multipart_form.ml", line 139, characters 10-51
console 2024-10-18T10:13:06-00:00: Called from Angstrom__Parser.Monad.(>>=).(fun).succ' in file "duniverse/angstrom/lib/parser.ml", line 58, characters 38-43
console 2024-10-18T10:13:06-00:00: Called from Angstrom__Parser.to_exported_state.(fun) in file "duniverse/angstrom/lib/parser.ml", line 32, characters 29-57
console 2024-10-18T10:13:06-00:00: Called from Multipart_form.parse.(fun) in file "duniverse/multipart_form/lib/multipart_form.ml", line 540, characters 14-78
console 2024-10-18T10:13:06-00:00: Called from Multipart_form.of_stream_tbl.go in file "duniverse/multipart_form/lib/multipart_form.ml", line 574, characters 10-20
console 2024-10-18T10:13:06-00:00: Called from Multipart_form.of_stream_to_list in file "duniverse/multipart_form/lib/multipart_form.ml", line 581, characters 8-41
console 2024-10-18T10:13:06-00:00: Called from Multipart_form.of_string_to_list in file "duniverse/multipart_form/lib/multipart_form.ml" (inlined), line 605, characters 2-55
console 2024-10-18T10:13:06-00:00: Called from Dune__exe__Unikernel.Main.unikernel_create.(fun) in file "unikernel.ml", line 739, characters 14-54
console 2024-10-18T10:13:06-00:00: Called from Lwt.Sequential_composition.bind.create_result_promise_and_callback_if_deferred.callback in file "duniverse/lwt/src/core/lwt.ml", line 1844, characters 16-19
console 2024-10-18T10:13:06-00:00: Solo5: solo5_exit(2) called

[hannesm]

Thanks for your report.

My first observation: something catches out of memory. I guess for a web server that by discarding one (large) request can free up a good chunk of memory, this is fine. Although I'm not entirely convinced.

From a second observation: when we receive the unikernel image, we have it in the request in memory, then do the multipart decoding (keeping it in memory a second time)? Can we revise/dig more into the details to not need it multiple times in memory -- eventually even streaming the binary from the request to albatross (I guess this will be pretty hard, since we'd need a streaming x509 certificate signing request / signing API). Instead of taking the request in full, then multipart decoding in full, then creating the certificate signing request & certificate -- why not stream the request and stream the decoding (or do an in-place decoding without the need of more memory)?

[hannesm]

Further discussion lead to:

• multipart_lwt supports streaming \o/
• now, the json content of the form, we need it in a streaming manner (esp. the unikernel image content) -- not entirely clear how to achieve this (note: we also would like to support a REST API, thus doing some chunking on the client side in javascript is not a good solution)
• once we have the json in a streaming manner, we'll need the out path (communication to albatross) to be streaming-ready as well
• the communication currently uses TLS + X509 client certificate with the unikernel image
  --> we need TLS in a way that we can stream a X509 certificate
  --> we also need the X509.Signing_request.create/sign in a streaming way

Best achieved likely with seq. Requires changes to asn1-combinators. But there'll be a benefit in other packages (such as albatross). But this will take some time to push through.

[hannesm]

As alternative for the out path, we could modify the interface albatross has (and e.g. allow the unikernel binary on the TLS channel, instead of in the client certificate).