High disk usage during docker:dind scenarios with Testcontainers

[mawl]

Environment:
Rancher: v2.9.3
sysbox: registry.nestybox.com/nestybox/sysbox-deploy-k8s:v0.6.5

Container Runtime Version: cri-o://1.30.4
Image: Ubuntu 22.04.5 LTS
Kernel Version: 5.15.0-124-generic
Kube Proxy Version: v1.30.5+rke2r1
Kubelet Version: v1.30.5+rke2r1
vCPUs: 16
RAM: 36GB

Hey,

We're seeing high disk usage and defective CI jobs on rancher worker nodes once or twice a week, mainly in docker-dind scenarios using Testcontainers. An update to Sysbox v0.6.5 doesn't fix this.

sudo journalctl -eu crio

Nov 25 12:47:28 ranchernode crio[1642137]: time="2024-11-25 12:47:28.516519173Z" level=warning msg="Stopping container ad738e20887bad39204cd6106c7d16308a1f8a53a5f21d54cf26ca3714938edb with stop signal timed out. Killing..."
Nov 25 12:47:28 ranchernode crio[1642137]: time="2024-11-25 12:47:28.517333181Z" level=warning msg="Stopping container 68095211d7cb09101f5ad415e7f160f8298c9d5c232068553ce10ebbd64a8ce1 with stop signal timed out. Killing..."

Thanks for any help in advance.

[ctalledo]

Hi @mawl, can you provide more info on the setup and how to reproduce the problem?

I see you are installing Sysbox on a Rancher K8s cluster, but what Testcontainers workload are you running on it? And when does the problem manifest itself? Is it somehow related to the number (or size) or images created by the Docker-in-Docker engine? In other words, does the problem occur when the Docker engine running inside the Sysbox pod has too many images? And does it occur when the pod gets started or stopped, or while the pod is running?

The more info you can provide, the better. Thanks.

[mawl]

@ctalledo: Here are more insights - we're using a 1TB HD to catch high disk usage peaks.

full list of volumes during this peak:

/
/var/lib/containers/storage/overlay
/boot
/run/credentials/systemd-sysusers.service
/run/containers/storage/overlay-containers/00fbaa929c9db19aa88a4c77729e222752aa617c5d3770e28fb5656fcf34e0ee/userdata/shm
/run/containers/storage/overlay-containers/0d67927b7ade1ba3fbf11b0578c30516acb05dd38927f175344b61f6903918bf/userdata/shm
/run/containers/storage/overlay-containers/1245c91e54e0f320863af984041a109b82bd22fa294b1b5f08d630df422cde0e/userdata/shm
/run/containers/storage/overlay-containers/1309f5f2660c18aa0ba234d5f107207b67eb8f07ffba299298456a201ca122fc/userdata/shm
/run/containers/storage/overlay-containers/1a4e29bf6b395f4e36f5cd476e68231566b9753a93f1aab4e5daa2d189c0e1ec/userdata/shm
/run/containers/storage/overlay-containers/2d970890c0a97229f299f45c5bb3d656d425ee74f914a7458465def1ea792505/userdata/shm
/run/containers/storage/overlay-containers/30c3e452d3bffc69c15c471f40f3498a831f47d60c049defd13de4fa4ad078ae/userdata/shm
/run/containers/storage/overlay-containers/409058192a4455dd1a2ab7efb5107dcca40bf0bf4d639d977d6ad9f33da9bdf7/userdata/shm
/run/containers/storage/overlay-containers/45fd3c174cd1c10f21767a9ccfce02e4594f587fe3591a898b476a6c87173d32/userdata/shm
/run/containers/storage/overlay-containers/466f8a75b985ed4c3acaffc8f58892a6cdfe491d4aca851f9772231d6c956c27/userdata/shm
/run/containers/storage/overlay-containers/4c8467f91699d88f7cef18e2755598dcd6f533a4d084591ef4f0e5b782700451/userdata/shm
/run/containers/storage/overlay-containers/4d8cd573c9bcf96f23b91754d12f4ba146d1b255780319bd74c9d9f7462d71ed/userdata/shm
/run/containers/storage/overlay-containers/56bb97fb77d59f6623659e180625c4efdd927764634afbfeedfc1c5ba60b848d/userdata/shm
/run/containers/storage/overlay-containers/5ef5cbade00877f79702725e3fc10592253c86fb6b3ad6d2d254f5cd289f0d13/userdata/shm
/run/containers/storage/overlay-containers/5ef9ef88e0851fe791e6ad50a8a8759942fd8bbac982d9973c682f1a1b5c814f/userdata/shm
/run/containers/storage/overlay-containers/615daadd3e30d2ee7f7a4122f795b148ec34f5da076589284c105cf714551bcb/userdata/shm
/run/containers/storage/overlay-containers/6c0326e844c00598ba217d0e293fc556936805beeaba86ef6b4d9894983095b3/userdata/shm
/run/containers/storage/overlay-containers/6d816371c7f1052b53f4f9cfe8af0ff4dd82949a0573b9b1ff7846b1cd7ff67a/userdata/shm
/run/containers/storage/overlay-containers/6d9d24a7675d658aae0e66dc5ae0e6ecdd34f8e61bd26d5ca550dd1315666ccd/userdata/shm
/run/containers/storage/overlay-containers/6fa263298788c7917f2b570b6f618d3da0098c810e9e28bd3bdac487be6f1f62/userdata/shm
/run/containers/storage/overlay-containers/70c2bf87f8c816f00189052c09a25ff6c2290a47f23138c6d3d1ea13c035f17b/userdata/shm
/run/containers/storage/overlay-containers/777f62593e777d9bd6f7339fa63f39b62fe07b0dedc1b23cdd520aa429c379fd/userdata/shm
/run/containers/storage/overlay-containers/778de73dc594d3f5fea9bfde39afa1764b39ff85338bb020cd70e212ccc1d68f/userdata/shm
/run/containers/storage/overlay-containers/8d8d68f4e807168c4983155f780e874d5c1177a8abf81541cd62403dae9361ef/userdata/shm
/run/containers/storage/overlay-containers/8f2cdefb90bb6eb7d4b896da9a7cf54f6c33ee897b61b9d033d84d1e466120f8/userdata/shm
/run/containers/storage/overlay-containers/9826f4c6e4fadc1a4dca815666fe796e385b4c3926295e3364f47a27d4c8cfd0/userdata/shm
/run/containers/storage/overlay-containers/a419a319202775107425229d659ef535ead41671de65b9ab66996f5504846856/userdata/shm
/run/containers/storage/overlay-containers/ab63c598168f1e38d00eb2eb372a1fe3c96c6acec3fd564598bc01fab8012065/userdata/shm
/run/containers/storage/overlay-containers/ae96d35da1498fe79baa6b928c27bacd469f5026ef1d4873cc9874e4a8aedab0/userdata/shm
/run/containers/storage/overlay-containers/b06d2bc39c5b6d73c6c0ef11be4ccb448fec325ad8d01b3dea21ecfef0fdda15/userdata/shm
/run/containers/storage/overlay-containers/b884e71b7ad789f1884e11e5167e64c0a80c351c6f81feb95e3532f4364030f8/userdata/shm
/run/containers/storage/overlay-containers/b8fef3f611aa95f44f524f3bbcf76bc08d093bd57d30a57f8b4672a7a1c709fd/userdata/shm
/run/containers/storage/overlay-containers/bbebfd438807329d4c493bedc3d665b11ec608eb981932395bf7bbd1a85976d6/userdata/shm
/run/containers/storage/overlay-containers/c1e99a3a93e36c9fcbc11f5e2d090f894ce7ff1ed3dbb009ee8e208a151f40cd/userdata/shm
/run/containers/storage/overlay-containers/c275fa0202ad6ce807044a3ac4d70f209cc807567f52354d8c27d69927839beb/userdata/shm
/run/containers/storage/overlay-containers/cbb23d9a8b743f2c2e1f84cf55adb961f537e8974bf57ad54ab259fce1b9ceed/userdata/shm
/run/containers/storage/overlay-containers/cf350e28106ad4d93cf52bbcd22f4eff6c71b23f14757e8fabc2d2965ebe95be/userdata/shm
/run/containers/storage/overlay-containers/d1b93fb8d66f60c07fe22ffd28ef69da98b3e5784809473e33de64962b6ea6e0/userdata/shm
/run/containers/storage/overlay-containers/d35b683ff60a553b9aabdbca76779be698471639f0028c11df3a6a31d3a63ef0/userdata/shm
/run/containers/storage/overlay-containers/d5519fa18d4b4aff29600778ca0a30a7112681f48e299f30ae3bf22f2c333468/userdata/shm
/run/containers/storage/overlay-containers/d5d9ed97dda5bafb607b310a95f9559c5292cb1ad6f9b2eabb0b59c18b123ee0/userdata/shm
/run/containers/storage/overlay-containers/d759b20837c4b66a29d5e955352a521c3377391277f203f4034739163b58d7a5/userdata/shm
/run/containers/storage/overlay-containers/e4825877aa97a1fbfeb7ea014c079af887610e42a3bd55bd28a2ac16696c927e/userdata/shm
/run/containers/storage/overlay-containers/e77b9868afed36c8bf264dd9299234183c0d982c9ab6e14ec2864b2c68642072/userdata/shm
/run/containers/storage/overlay-containers/e8198cfeb09d97d2552efb3975f4834a0e3a2f7583204c0069fa4c65826d15de/userdata/shm
/run/containers/storage/overlay-containers/e940968e6f53c38a9415f86665e00a94d746dd1cc7289c09127faa48912efdab/userdata/shm
/run/containers/storage/overlay-containers/ebbc8b9cf560792ba7e6176ceee59598b9dd93348d22a257d999d10144a8efb0/userdata/shm
/run/containers/storage/overlay-containers/ec3531ae0aeb16cbb9502c4af34d559da7c31931a603f01d0306b85fd45726c1/userdata/shm
/run/containers/storage/overlay-containers/f951fcf63a7cf388b4d69f3db9d56b86d68704f78877ade08a414c95c203ae6e/userdata/shm
/run/containers/storage/overlay-containers/ff0ff3f08a3e2b633a40b9eda50fc56ac3bcd496567ce0f25c275e47672db0e1/userdata/shm
/var/lib/sysboxfs/038317587e15589eac95d34633f974dd497656d41946b2bb1f606af3c74cb505
/var/lib/sysboxfs/048d58e36909dd428e513ff4a33a252d283833b04e2036eae53f9f13c316b850
/var/lib/sysboxfs/05b64d8e25522de6a191e5217f47406fb0cddef2d67ce64a9699070bf6910bb1
/var/lib/sysboxfs/05e18ea2dc408f4e9c7dd57960f66ad339ff547588b0f79711d7639196d9ad63
/var/lib/sysboxfs/06dbea97aa2d70ca8e564bcb1c4cfcb274aced81de0b6e7faabb271a1acb2ee8
/var/lib/sysboxfs/0768a2b594d21ec1f1589182d6b5b6d3b37a0bb907e550a984c3905834e331b4
/var/lib/sysboxfs/08d0bade3696b3900d173505bbed7492c2c9a80aa2be4eb42d9fc11ce5f796fc
/var/lib/sysboxfs/0ad3dfe4349ed64e516b9e39a5366b7281ce5a7b5ad290d9636e1fa2bb9b5d49
/var/lib/sysboxfs/0afe409c24ea69c2067b01ff2e008474f83373c3eadad933a50ec383891bdcb1
/var/lib/sysboxfs/0c25f1489a4685f3282ab1e40b14fb23d1985260953f05911449af64b913bafb
/var/lib/sysboxfs/0cc5c7a62448ea45c0d1b455af2591fd535ebc3b3f7a6ddd8017a20ece50493e
/var/lib/sysboxfs/0cd18b0c384f362d01100f14ce857214bacccded910defdfadf9abb4e8029b93
/var/lib/sysboxfs/10b6452e6d72c55933a730056314bdab13a746e00e3a21d6606c68b042462600
/var/lib/sysboxfs/10cfb027c388fbe035c931c13930ef2f2319d35f665af167cfb8947c0b3aa67a
/var/lib/sysboxfs/114aabaf92408b630a3e2d3f1fa8d4c10b32223d17dbdb158f23c59da92d8474
/var/lib/sysboxfs/11a3d485a6d94eb9d5fff39d7de2ebc4683b1b43b5b0f3324fd7df8da850f0af
/var/lib/sysboxfs/1449f13c9b2755ebce1c97ecfead8af6697e6652d493de07c590ea73eef3ef4a
/var/lib/sysboxfs/14690caa8b20ae160ae6b83f1e422cc43063f623ac5828793aa588e88d140886
/var/lib/sysboxfs/16752aabe791ce7e056ff2f799fa72922f551171dc0cb8b7d85b92515296b3fb
/var/lib/sysboxfs/17db9c70df38f210154087635f1b8d0d0a5d0bd9a4699356515933486fbe5fdc
/var/lib/sysboxfs/186ca520ed6be0fbfe57bd6616f87cff0496c32e546bbd32c9785e4b4803134d
/var/lib/sysboxfs/18a2f83bc5a7a32e4c0914ba5613f75ba4cf6e0425140c722b2974307a3d99b2
/var/lib/sysboxfs/18fc8628a0c6dd62db84f8d84fe92a2cc5580eed9a3b161ae6171ad55a414d13
/var/lib/sysboxfs/19589ca392a080508801c6afbb43c979cd4eaa124af3b6b433013378e32e8008
/var/lib/sysboxfs/213cd778f0a15b527a3db51ec57d14884ebcc266c1a225cf4a9aa52c3b43630a
/var/lib/sysboxfs/24969815fcb28b3977598ca44b8dad71a31c0c021ac7ebb3ff5e89c8d6c73347
/var/lib/sysboxfs/24983e3785f7b9d9feebee332e540a86d07baef3b00901fc1edf937e8bdbb254
/var/lib/sysboxfs/2571706c767a56e3d8d44b823c1544db08b9fa469cef004a35a211916ae28d24
/var/lib/sysboxfs/2b32a1624f80378d8072dc278818b80e094d277583c945bd00d35bcdc57242e7
/var/lib/sysboxfs/2be17b30e794df591bc024f5728d3d6c93dced1e69db358f58f7ed3daa9ab070
/var/lib/sysboxfs/2d34173a6f444c43453bc6e0208253ac9bdd8ef6e54b93c7298e2184285d4b80
/var/lib/sysboxfs/2eeb88fdc691ee69acea3f77ee598427e258837742a4f0b62ee271e12984a727
/var/lib/sysboxfs/315ffccad63d3efd7d19965bfa28b85bbd745b6c88933cf4397769b4803a0c9d
/var/lib/sysboxfs/3189e71c57084af9cbe7ffc643b8f71bbb8cfbae532fc3f43f6b57df14967178
/var/lib/sysboxfs/31e2901f4a8621ad9522e2cc3621c92e1f9325d9d1ab7c79f4e5aef93ce84f94
/var/lib/sysboxfs/33ab080757fe6f003f9f1bbe90a56f18d130ed8345c24c41637e751d844cebe6
/var/lib/sysboxfs/347c37639fd7a2c7a29917fc70e6c6be7fd36f7b186ebec6f6a436984150f29e
/var/lib/sysboxfs/34a99189d3255b459c4308c022a092a6ac4d5869b49df3388ad6f63be9cc10aa
/var/lib/sysboxfs/3574e0fd9cb8c00722219eef84af0d3f506db5c2ee6143e244f2470753678c40
/var/lib/sysboxfs/3754a2c2ce140ef93c28a16f36999e01cf5646eaaf0f21c523defb9c191154d7
/var/lib/sysboxfs/377a8cfa20cbb22904ccd78e7871939738b656e76294bbb30317b13ffd6188a2
/var/lib/sysboxfs/37cbe6735273c67f237b7d4f8702fb0766632142bf913612bea50b8c17fa10e9
/var/lib/sysboxfs/38ea8ef622a5340629324c57628c8da78be2931a737425e005430433f7b6e95b
/var/lib/sysboxfs/39619a4b06edf4206d8af80c9e73b59296fd368af957474d25c513b1852dc58d
/var/lib/sysboxfs/3a1105053c6f4a1034f7948806e8e405c7629fbfd65dfb9f82c46e125665dee8
/var/lib/sysboxfs/3d236b5c0e19425ac8cc8920b90ac1e95382d97f775c179267bd9da0826d66f2
/var/lib/sysboxfs/3e931782c66423c7a99a6f29fe80bcbca018a1e3a35915ab5823af8a3b77c476
/var/lib/sysboxfs/3ec4a32b42f2bd8abc5929f0352c7c699a91c8e535a6f10ff15caf56972e0357
/var/lib/sysboxfs/3f6d3f8939956c34782bef4d3246594aa0897a2361cb9adc7be169ddee54f5b6
/var/lib/sysboxfs/3fea433434782033490349a8c20d0cdb886c8df590053842f9b9d79ed51b9b08
/var/lib/sysboxfs/405041b9bbd962337d9d7742a41a5484bef2888be8f4389bb2fb0aab1bc11ed3
/var/lib/sysboxfs/42ddf1bbf9ee8b852fdbba95f8f835b11e0997f6fea2bbde2978e67596353d85
/var/lib/sysboxfs/43551b3f3f558a971a0de061ebf8b7641da4b614ff6db9b34ae6b718f848dcac
/var/lib/sysboxfs/43aac4c04ea1d02e51619edab92174d246b8b289aff1cecabcfa84b8ae0a8152
/var/lib/sysboxfs/45f08dc735db3b23b8dc2e114328f5ac3bd40593d0b9d1dc7f82f49ff99b4295
/var/lib/sysboxfs/467f77f72d20c5a826abf340a039e35eff5ee898e7b8fd75d02b7095b5ae45cb
/var/lib/sysboxfs/480eccbf5fc249f0ced6c556fbd33df8d181d78abafedbadb6d426721f5fb1f6
/var/lib/sysboxfs/4a0cf43d67b731e42d1716d4b26a29cfd9b2905a3b9072a05abdf19b09c85fe0
/var/lib/sysboxfs/4c922048324b8a69c4a8c7f9d6ce9d4701ad503d084c2c986cbfc38294106e6b
/var/lib/sysboxfs/4df950ab5ccbe799b6351c07ac305ff99831453ff1b498e0ba759688eb2e928d
/var/lib/sysboxfs/4ee4afa66ad0716fe3b881bb696320524488ecaded644c2b88f90e0e34604e3b
/var/lib/sysboxfs/4f5e5f0c6482d82ffd330a069982ae28bb479bb325c7a5c5317b0631a123c593
/var/lib/sysboxfs/5025987e7d6364d60454432b09691ddf24d7835e13b313859029b045bd59969b
/var/lib/sysboxfs/506fa038d22d3105bedbefddb5e52e98531ce37b3017e23c43ef5b2bee44117a
/var/lib/sysboxfs/5150b1c3d8eeca1c53f594d73c7bd82cce71913f4686237e66bffa59686df3af
/var/lib/sysboxfs/51a2e0eebeba7a10e1e1c829a47deec4c0195aa44aa4fd109a7b713394067b8c
/var/lib/sysboxfs/51abd31a0f0e4bee20347543a5f147477b14c0b558d4b005a491e009242d872c
/var/lib/sysboxfs/51b5039e8d6a641f5a1a3fceb18824d78e2088a9540b94ee08c1fb2dbb0406d0
/var/lib/sysboxfs/533a48bc4017c0901b1c20f63519283fbee240577c44f573bf08ea06c1482248
/var/lib/sysboxfs/540859d60023bcd94af1f5a4f9a00619afdfcaa4857db24d3891d076fa56e6e7
/var/lib/sysboxfs/55eb44581dd32772f3a92adddc915267181bc3ea4bd18c69ce84dce15c0a0004
/var/lib/sysboxfs/571036ab2c0736de6ba60d9a9d78fbab58d9296753f8163325afa39f8df8bf79
/var/lib/sysboxfs/589f18df7c89b397863b3b544aa3798b0adbce3c27e9e5d0488951076f440e00
/var/lib/sysboxfs/58dd285f59dfad664fa033229ab5835b1e29f9e0a1b9be75a9d1636ca594af76
/var/lib/sysboxfs/5a1665f6ce9662b2bd79ba46fd15eb2cf618dc1434b9a1a6ff0ee688764b12d9
/var/lib/sysboxfs/5bb21b4ed8c45bf949cf48e2a9f36141515c18361da6f6989b337c201b1137ac
/var/lib/sysboxfs/5d65761db869b1e987773cc961ff30a9eb1a4a90aae441644013848e9c06c253
/var/lib/sysboxfs/5dd6d94121eb37ae0128a636d28169be776eed9aa5a87b125163ee5ffa890e3c
/var/lib/sysboxfs/62b918618aeb23c5822d926e28cad8fc8ef54e6cb0bace6d570f7edf7d94cc98
/var/lib/sysboxfs/636dc3332b0fb78f8d4750f1faa8b465193ccf2cad695126cec036beb777efdf
/var/lib/sysboxfs/63de0bdd1b06601c301f9236b570d04114462d12c8d9608cfbd5d5019b9be501
/var/lib/sysboxfs/66d77f26900dfb6143a9df1f0911d7ed58ef6ebd9bdf902e7773e748cee87fd1
/var/lib/sysboxfs/67970241b3c7d1d5f165f868fc5e79aeea9e8134c11a819201c587f66b9aa944
/var/lib/sysboxfs/6926f065bddc96ee27e89f2b9551aa58b874924bdd56dbe5822c8011ec7b4de6
/var/lib/sysboxfs/69aa439eed89e39d2d472c0f4fd33432f389ecbcaffcaa1c5434820f4bda003b
/var/lib/sysboxfs/6b33aee3bcb81d7abba8ac90f2c8e2eb74f3ba35006f9b80f43a1401af8f9e84
/var/lib/sysboxfs/6c1c0e17051d1622165294ad2e8275d9b20856cc1c8cb65c859a5a79fe93e9d9
/var/lib/sysboxfs/6ef4f16c7337247250499cf357c163e07dd08fb96a06c634b9a3ac3e9284627b
/var/lib/sysboxfs/6f5f94c0b1da40113639f6bccf7ae023ef1ac11ebe343d07e4d40d58878e8bf0
/var/lib/sysboxfs/703eca76777fc0059bcd183c67c68d85d27544a26a16f0535828eefd6f518cc3
/var/lib/sysboxfs/70f594174a28f4a05ac2bd32bbf22429ea786da04aee4da8bf4dc3fec5733162
/var/lib/sysboxfs/71c0013fa517db7326777dbe000b6e1cd5fbed48fa3acfb91238e5abb592e2a6
/var/lib/sysboxfs/721bc7a3d648b47865f8a5704d8b7f4d14be4b3b90c633c1504157e9acb3faab
/var/lib/sysboxfs/7370496b4ef1693dd70b46bfc1248169ff8c54122834fb69bcfb20e6c9531816
/var/lib/sysboxfs/77971b22f55c2f48b230be2805a8d759057db317350df4b24205f5b0a9ac2a6f
/var/lib/sysboxfs/7982bc5edd16791f34cf16f44c36ebdf3dbb1ef2b193ba4ace3f60b4e819059b
/var/lib/sysboxfs/7f36f5663705b33a7faec781efddf63b89e69a83115e3cecef85953d7f99af67
/var/lib/sysboxfs/7f52eebc0388cc313293c571fda29254b3572f4b7f7a13ca73f3f608f45654eb
/var/lib/sysboxfs/7f5f0f53e5c552c303ce009e66ff147ae3682d3ac7ae6bda02f03de13bc8051c
/var/lib/sysboxfs/7f94650032dd3e5139e5a1dad17e716ebfde6f8ef0cf0cdfa7ce89480f4b0834
/var/lib/sysboxfs/8048d1108275e7bd09fe95ec8700e50dd116e2cfa09cc9ecc0a7117b7eb09814
/var/lib/sysboxfs/80af6e2730cea77f11caebd6a20486545b0833862118423f0f797bf0522904e2
/var/lib/sysboxfs/818f982937940da12f1d38ab86630ae7789ed0e3e5aa4153c4066cf5557fb19d
/var/lib/sysboxfs/8313fb99ecda3fc95780a849af7b16a8271abdd9947469d598efddb8337c3488
/var/lib/sysboxfs/844dca042e8f7769ab3f12ce6bc9324055fe73c61f50815e0dd017b05525106f
/var/lib/sysboxfs/85f16f50464ff3cdd9a920babf6c5cc14c06fc76153a1fd1d15b56b48dd3a94a
/var/lib/sysboxfs/86085dedf95271f18ab21ea825ffbf62bad996b9a71385fc7bfeee2ad007699a
/var/lib/sysboxfs/86913a1d858d7f3c7fb0ee8325c1e726428ff17284ea18514910e81147753f3c
/var/lib/sysboxfs/88fea01b140396edc245493b6d068a7adda55d3d32e7530543ee509996df9a75
/var/lib/sysboxfs/8ac1236b059a13819954530e5fbd61623b0968a6525f662a246c321c6f18ed35
/var/lib/sysboxfs/8b3aeb9c867e181f0d389956fb9694b85d63a805c971f64fe15251047582f3d3
/var/lib/sysboxfs/8c671787810035c94e27f760938ebb3ad25c2ae5f493f545ff7aac7ada75de83
/var/lib/sysboxfs/8d20dd0c5c4152d945541f8b8c20d959695e1e058c5f2f04b48fbd86ba9af149
/var/lib/sysboxfs/8d469cb051d568066a3085a297bc444983c3164f5397c3de9acdf7edba69a4f3
/var/lib/sysboxfs/8e338ec22f78e08e9e604af5c82c3a0ce5fc434730c8fba3dbc9f80f66944083
/var/lib/sysboxfs/9202e6e106abdb2b522c154c3e38ccd32700a536ba160994af40d6c30cf9c77c
/var/lib/sysboxfs/9216e1c03da3db88f18d14ddee473ded5749746907ff7e81ff35b079bc290cd6
/var/lib/sysboxfs/92c1b29ba4d1bed8629630ae44e9bc4152c7857ce12eb03d7bd0caec9796ce6d
/var/lib/sysboxfs/9524306edc4444beeead5819bfb2ab408dc140c2490c90064abb75b25b3aab21
/var/lib/sysboxfs/954dd050c00032ba259882527b643cf410004f4d23a0e5de3b560ca93b275ee5
/var/lib/sysboxfs/958eec78ec174666a32d36549eaf51aa43c0eb8f7cef9314e5296f96f97beb26
/var/lib/sysboxfs/974aa796ad1cf0bda02e6400437638bcdfc60c0f756d95b5ced92b9837609208
/var/lib/sysboxfs/979d0da5278a5b0118010d497e815f8b442dd39bfd7350368374cd5155e3c30e
/var/lib/sysboxfs/99298f6c72163857040320cc4a86bef26a4906a958697889d1e948c093664651
/var/lib/sysboxfs/999787ab68824fdc0c5d2f41b4206f1cee696e34dbe52ba144638f45fb781538
/var/lib/sysboxfs/9a25fd60e9dff2a0d3dd41eb257f66f9f5ce56f583998d1bc680bf762508d113
/var/lib/sysboxfs/9c6d7371d588c501d1e909774bb22e3e59f234bf0864a59ae8a9ca45f5e91a39
/var/lib/sysboxfs/9cacab316ed4a8a377333cf7088569b55263751a78436f71b2cf36ecd1fc6bb5
/var/lib/sysboxfs/9d851fbe4223d8494d5be8b6d5bf3e76b3aea9f2442be2119788e3b15ce9d2fe
/var/lib/sysboxfs/9e21000fecfb30666642d484da23d8d632b50b877a0ad7a2d73db92e597fbaf8
/var/lib/sysboxfs/9e69526057422ebe694cb6b577946a15ee1d74e02111be7e9c25b887ec0d80bf
/var/lib/sysboxfs/9ea5c570d5cb0aca7dbcaf06b8f403a9205a19433cda14fadf529e19a1831df8
/var/lib/sysboxfs/a038c3e1ef3a6779790fac7983eb017a0dfc95103d573465f00f8fb6a7be0dbd
/var/lib/sysboxfs/a0baaf0963dab91e2488c624853cb2e7e912bf1355af222beb3596cdcfdb5f6d
/var/lib/sysboxfs/a289032d05cfca5e1ece5fc899f91bcf2018f711ea76f6653ac5efd77b639078
/var/lib/sysboxfs/a2ae9b9e952f6275587cc2b9a8a8933fa06ac6bc799ff536d791926d8f8db2c2
/var/lib/sysboxfs/a4774d74d7bd1fd543e26ddd57606fcba862c7b9462fa933612349176caf1f91
/var/lib/sysboxfs/a5b0adb1200572a63b68db0302922e11d11e273cca1cfdfb9add78b51a0fab5f
/var/lib/sysboxfs/a6f34b78c5be2e547748179ab7df1fce05e2cea36c3afcc3ea5faa8bf0030281
/var/lib/sysboxfs/a94494a21d805d1c3ac1aab4990a6d7ea85ec5e2b64bdd0513526f129f47a319
/var/lib/sysboxfs/a99cd64b3c05a54ba37e1ad6d7d26cf34af1c0930b5a00b16ba109515ce7ce60
/var/lib/sysboxfs/ac76ae4e022adebb0422d4b2cc514d5837377563fbe6a9f2c4053d986a403bcf
/var/lib/sysboxfs/ad1d938c8440fc08dff39ba8fe1707412b365251c36b902b6b568042163226fa
/var/lib/sysboxfs/aec3190063dddff101e1e4adcef7ba3199e3872e6abb71c07d50175a89f48447
/var/lib/sysboxfs/aef3b1a27a19288a75ce406a784a223fb46bee8b908842aca4aede674bb9176d
/var/lib/sysboxfs/af10c0c68bdb3f470b1e3e6932bdeda37a21d76bcd3b02314e3b89f2b45663c5
/var/lib/sysboxfs/b1a9148d4d1862a92bac4302a4a406e457a2f7ea49cf3e6c1c4bb85e86c3f8ca
/var/lib/sysboxfs/b3c1653bdec28c6f09939393596c91c69bc757beaffc53c15f135ec8cdc5ddfe
/var/lib/sysboxfs/b3db831ad694ee478310e7900c03e1fa44ccc2edf88efbc7e03bd16afd8fa20c
/var/lib/sysboxfs/b44043d19a59331e7c8eec6a8c0afc45281447bd7061bfb8b5f020a8b969f63a
/var/lib/sysboxfs/b80e5ab0137e492c152ba4e0cb102f82fe62010791bfc700316ee07e69092ea4
/var/lib/sysboxfs/b8f6a0efba5eb5f8516f150568ef6886d2ba6b1993269fc1b95d020e27a972a8
/var/lib/sysboxfs/b968db6ae738485b1e1d04de18ef42ed4cb83ca145ae31cc6d77a5f57bae38b4
/var/lib/sysboxfs/b9cdd45810fd2bc534ca711a4a05ffcafb5592260b5bfdc388dd5300ecef8779
/var/lib/sysboxfs/bae88d9343224a78202fabbe96abd80125d62efba0ce05cbc4a9461c32c29a47
/var/lib/sysboxfs/bc24bcec04d80986db8964ec48168bac61a43ff89688ad250ece0f0791ce6bc1
/var/lib/sysboxfs/bcd4b4e8a41ad77dccd08cbf36c4dce0cd4a5ff28b51bbfca746ae28430f20ef
/var/lib/sysboxfs/bd073f0082ac77506e160a436a87d6fab14f474d664af8a9add134940445190d
/var/lib/sysboxfs/be727d7c116d67694b64e26f4c3a2cae7fa89b16cb89c3a6e194075e336057de
/var/lib/sysboxfs/c223d15d31db5775c3c70c16c4ce1075c3287f72084a35c5e243e786cf92665b
/var/lib/sysboxfs/c2c0308d46284bdc2febae11be9ebdb5c27e07992e30e9000bb64ba1273f0e26
/var/lib/sysboxfs/c2eae01526f0d12443fc52f0d1eeea7bfeb4f938e4423f7b946729e4cb30f4cd
/var/lib/sysboxfs/c405f33e117d1bf35e5508880fc4dfc2e978100e436e7763b2032f99b420d499
/var/lib/sysboxfs/c55bf42768d86019433dd8792bc154f5ad74dbffaa4e7f9f8c260268c015e8bc
/var/lib/sysboxfs/c6606926880c4be8a97a3372d2e2eb3a9cdae2622ae34f8174d7b61f99875bed
/var/lib/sysboxfs/c669f564bbcd5dba4299edc599ae46f87a2f6e87d8250efa0044608160467c75
/var/lib/sysboxfs/c90988c3f2fbf262300e5997559850a99615b0e093c73fb88d11299935fb97c4
/var/lib/sysboxfs/c9fd92c15d7b4ab649ee6f9985f04e01a5fe98f19c628ef7b19e731b1f152090
/var/lib/sysboxfs/ca4ca84e3c9e3347540ff75eb1e40f7691f67696a565378ba287306239f3af4a
/var/lib/sysboxfs/cb03b0b4ed8668cab975f5a818bcbf1e8255ca3bca7a131005a7bacf0b81e83f
/var/lib/sysboxfs/cdfde7087ff07f79ccd714a8448bdb5b30ab74200d62f4eb0712d6c77b825f6a
/var/lib/sysboxfs/cf8cfaa78ed0b85529626268d50211fd30679bed255a3842ceb4a348840d89b0
/var/lib/sysboxfs/cfaa6668cc1dc960fd17c8313c54f9a40a8e57d9f676b5fe63020017cb2364ad
/var/lib/sysboxfs/d05aef0d4ff063c47ca252c0756f42a44e4feaa0243a4c457b61998d1a18713c
/var/lib/sysboxfs/d14e8bde8cd54aeaac62e59c87d7c8cb4886d1251c8d6c43e2c3cb0914921fbe
/var/lib/sysboxfs/d2ab4f0307939c910d58c56989a2702a647d5e6482cd919223fd14c3d2c1603f
/var/lib/sysboxfs/d42c4e9260879f4b6b172121a501144ca4b0ac88bbc6573c3c339d265c8a11b8
/var/lib/sysboxfs/d56ad5233bc9f6418eb7cd3e70c6158216e0720c0c0eb888803a98c09a528033
/var/lib/sysboxfs/d690dade4e41323f37814bb594e3f6eccc5dc8933d628d5bf845a8c6c22de56f
/var/lib/sysboxfs/d82944e6482dcb78ab3082c9ff5c6216085b2f9e07cee165741aebc4fc40d659
/var/lib/sysboxfs/d91fcd8ec81d6e140394dbe1ebc3aa9b74fd4878f35715c423edb86229a86d9f
/var/lib/sysboxfs/db4c808d1adf335e431d3dd8ae3f381ec60f7ab5538e27593cb2a4a640771c3f
/var/lib/sysboxfs/dcbc54a4fab865ef75a406ea48c4e391f56bcd43a4d67b343a8849dd323a6e53
/var/lib/sysboxfs/dda82ddf18ec7cd1a3bbbc2cefd9e16869d472f1816819aab20a2b3baa877729
/var/lib/sysboxfs/dddcfb87f93e95870f0fd8f4fd63551ce9078053fb9ae0ccea3a47469531fec7
/var/lib/sysboxfs/df522365c976ef85c6b9f5c493a94315f19c9ada2a8ea508a6a46620d8ed90b4
/var/lib/sysboxfs/e0c3af334e1a3f87ed8a0c8afffeaad471ce1e59450c350d69ae34dcb56d369f
/var/lib/sysboxfs/e4b10b2b386d600fd8f311d5a90fb945e1bc24b6fde0dc35f6d007449d5c4071
/var/lib/sysboxfs/e538ad54d037d2c00c9f1f816da4ee50bfe330fa4935d8962f5d89106989ce0e
/var/lib/sysboxfs/e5b92d8d83ed9baa1a6dd99b223f4f02cf23607efd9ad67f45cae3d33496a21b
/var/lib/sysboxfs/e6fe95bc8db137c084436f82cba85d979d8a6f6ea04664498b2e500f98d20ee3
/var/lib/sysboxfs/e70484ffdb2cd0591471d97e47f04fe4b7ccea9a5ed432cd8a2e7958cd7ae323
/var/lib/sysboxfs/e768ece207b3d204a23516f0182455c0afd44d36125ac48c4fc95e3fd033e7db
/var/lib/sysboxfs/e91b42dc31f2ea198c53fb766d57bf1c874682efa0c1de4cd358b17ac0dd7149
/var/lib/sysboxfs/ee2aea86cd662489bb347e955ecc227fa044d020348f952e1af98ad3d73aee38
/var/lib/sysboxfs/eec85de1a80ce5451c1e2d71714e5aa80c8554f0ae449da023f60d1d801fdc6c
/var/lib/sysboxfs/f1709285b66b6621d4ba44b87b652c76a1f21bd219ed4ccef1f710ab14457ea8
/var/lib/sysboxfs/f28c4f8102aece6284c9ad1c6197b6ecd8a9f9707bd642d792bf0b9a2b190100
/var/lib/sysboxfs/f70810f3f0a5ae9d39ef7a3bff834086a0c0651cfdea8307ae9d9308d1767f78
/var/lib/sysboxfs/fcc535ae071dac355567533ad47e69d5b1e2ad35dfb442bec09fe869b1b791b1
/var/lib/sysboxfs/ff0556e7d85bfca74e71818bf844f960d7dc707ba029e301a56c75aad60977d8
/run
/run/lock
/run/snapd/ns

We have identified the cause, a dotnet test run in a GitLab CI job using testcontainers with NUnit, which runs:
dotnet test --no-restore --logger trx --configuration Release -- NUnit.NumberOfTestWorkers=4

The developer has told me, that they are starting around 1-3 containers (Redis, MinIO, MongoDB) per test, for a test suite with 350 tests, which leads to 500-600 containers for a whole run.

Is it somehow related to the number (or size) or images created by the Docker-in-Docker engine?

It's more the number of containers, less too many different or big docker images.

And does it occur when the pod gets started or stopped, or while the pod is running?

It occurs while the pod/CI job is running.

Remarkable: Their test suite had no issues before when running on a docker-executor GitLab Runner with privileged flag enabled.

[ctalledo]

Hi @mawl, apologies for the belated response.

Looking at the diagrams, I can see the disk usage in primarily in /var/lib/containers/storage/overlay, which is where the CRI-O runtime stores the container file-systems (CRI-O is installed by the sysbox-deploy-k8s daemonset on every k8s node, replacing containerd).

This suggests that either (1) many containers are running simultaneously or (2) containers are not being removed after stopping.

Question: after installing sysbox via sysbox-deploy-k8s, do you experience the same problem when running the CI jobs without sysbox (i.e., using the default runc and --privileged)? I ask because in this case you will still be using K8s + CRI-O + runc (instead of K8s + CRI-O + sysbox), so that would help narrow the problem.

The other place I would look at is the CRI-O logs on the k8s node (e.g., journalctl -u crio), to see if there are any obvious errors.

Finally, the diagrams don't shows the storage under /var/lib/sysbox being a problem, which is where Docker Engines running inside Sysbox containers store their images. So it's not an issue with Docker Engines inside Sysbox containers pulling too many images.