Crash when checking torrent with piece size of 256.0 MiB

[YouMiNope]

qBittorrent & operating system versions

qBittorrent: 4.6.5 x64
Operating system: Windows 10 22H2

What is the problem?

I have a torrent that contains 3813 files with a total size of 279.43GiB, when checking it consumes all my 16Gb RAM and then crash. I assume there is some algorithm flaw in the checking codes that need to be fixed.

Steps to reproduce

1. check a LARGE size torrent (in my case, 279.43GiB)
2. you wait the memory fill up
3. witness your computer crash

Additional context

My RAM useage:

qbittorrent:

Log(s) & preferences file(s)

No response

[stalkerok]

This is a consequence of incorrect client configuration, also you didn't specify libtorrent version, I suspect it's 2.0.
Also

Log(s) & preferences file(s)

No response

[YouMiNope]

This is a consequence of incorrect client configuration, also you didn't specify libtorrent version, I suspect it's 2.0.

here are my supplementary informations:

crash report

qBittorrent version: v4.6.5 (64-bit)
Libtorrent version: 1.2.19.0
Qt version: 6.4.3
Boost version: 1.85.0
OpenSSL version: 1.1.1w
zlib version: 1.3.1
OS version: Windows 10 Version 22H2 10.0.19045 x86_64

Caught signal: SIGABRT

 0# 0x00007FF63F530D13 in qbittorrent
 1# 0x00007FF63F530589 in qbittorrent
 2# 0x00007FF63F52A515 in qbittorrent
 3# 0x00007FF63F820EEA in qbittorrent
 4# 0x00007FF63F8234C8 in qbittorrent
 5# 0x00007FF63F823531 in qbittorrent
 6# 0x00007FF63F818398 in qbittorrent
 7# 0x00007FF63F818FD4 in qbittorrent
 8# 0x00007FF63F819042 in qbittorrent
 9# 0x00007FF63F814F4D in qbittorrent
10# 0x00007FFC7ABB292F in ntdll
11# 0x00007FFC7AB62554 in ntdll
12# 0x00007FFC7AB622A7 in ntdll
13# 0x00007FFC781FBA99 in KERNELBASE
14# 0x00007FF63F815238 in qbittorrent
15# 0x00007FF63F807F2B in qbittorrent
16# 0x00007FF63F8130AD in qbittorrent
17# 0x00007FF63EED66A8 in qbittorrent
18# 0x00007FF63EEDB4A5 in qbittorrent
19# 0x00007FF63EEDB680 in qbittorrent
20# 0x00007FF63EEE1735 in qbittorrent
21# 0x00007FF63EEE1B98 in qbittorrent
22# 0x00007FF63EED8DB5 in qbittorrent
23# 0x00007FF63EED47C5 in qbittorrent
24# 0x00007FF63F8334CA in qbittorrent
25# 0x00007FFC7A427344 in KERNEL32
26# 0x00007FFC7AB5CC91 in ntdll

Log

(N) 2024-06-30T12:44:43 - qBittorrent v4.6.5 started
(N) 2024-06-30T12:44:43 - Using config directory: C:\Users\Administrator\AppData\Roaming\qBittorrent
(N) 2024-06-30T12:44:43 - Trying to listen on the following list of IP addresses: "0.0.0.0:8781,[::]:8781"
(I) 2024-06-30T12:44:43 - Peer ID: "-qB4650-"
(I) 2024-06-30T12:44:43 - HTTP User-Agent: "qBittorrent/4.6.5"
(I) 2024-06-30T12:44:43 - Distributed Hash Table (DHT) support: ON
(I) 2024-06-30T12:44:43 - Local Peer Discovery support: ON
(I) 2024-06-30T12:44:43 - Peer Exchange (PeX) support: ON
(I) 2024-06-30T12:44:43 - Anonymous mode: OFF
(I) 2024-06-30T12:44:43 - Encryption support: ON
(I) 2024-06-30T12:44:43 - UPnP/NAT-PMP support: ON
(I) 2024-06-30T12:44:43 - Successfully listening on IP. IP:XXXXXXXXXXXX. Port: "TCP/XXXX"
......
(N) 2024-06-30T12:44:43 - Restored torrent. Torrent: "the_LARGE_torrent"
(N) 2024-06-30T12:44:59 - Torrent resumed. Torrent: "the_LARGE_torrent"

[glassez]

here are my supplementary informations

It looks like your qBittorrent installation is broken, so it can't produce a useful stack trace.

[YouMiNope]

It looks like your qBittorrent installation is broken, so it can't produce a useful stack trace.

I think it's just some random trace due to out of memory, I tried to reproduce but the crash reporter just crashed itself, and my PC almost crash too.

This is a consequence of incorrect client configuration,

Im curious if there are some qbittorent settings that can get rid of that. How do I get the right configuration?

[xavier2k6]

@YouMiNope any crashdumps in %LOCALAPPDATA%\CrashDumps?

also, run sfc /scannow in an elevated command prompt & that should highlight/take care of any corrupt OS files etc.

[xavier2k6]

Related to #19745

[YouMiNope]

@YouMiNope any crashdumps in %LOCALAPPDATA%\CrashDumps?

also, run sfc /scannow in an elevated command prompt & that should highlight/take care of any corrupt OS files etc.

I found two crashdumps titled qbittorrent.exe that created yesterday when my PC crashed.
CrashDumps.zip

the sfc /scannow says Windows Resource Protection did not find any integrity violations.

[xavier2k6]

unknown exception (0xc0000409)

Dump gives above but no symbols/real info to go on.

E:\Program Files\qBittorrent\

You have qBittorrent installed here, does it have an associated qbittorrent.pdb file in this folder too?

[YouMiNope]

You have qBittorrent installed here, does it have an associated qbittorrent.pdb file in this folder too?

sorry I forgot to mention that I reinstalled my qbittorrent with latest v4.6.5 while the crashdumps were created when I was using v4.6.2, I don't know if the v4.6.5 .pdb will help. that file is too big to upload in github, if you want it I can email it to you.

by the way I tested libtorrent 2.0 built and it works very well.

[xavier2k6]

No need to upload pdp file, just ensure it is next to qbittorrent.exe where qbittorrent is installed.

The issue seems to be only with libtorrent 1.2.x version, It would be most helpful & highly appreciated if you could try to produce a valid stack trace (using libtorrent 1.2.x version) from instructions/method in #20869 (comment)

[YouMiNope]

Seems like I captured a crash. the log file produced by debugdiag as follows:
qbittorrent__PID__16324__Date__06_30_2024__Time_06_09_49PM__611__Log.txt

hope that will help

[scomnoob]

cant confirm that bug
5.0 beta1 on win11
RAM usage is ok on 400gb torrent while checking

[stalkerok]

@YouMiNope Can you share the torrent file or hash (if it's not a private torrent)?

[xavier2k6]

@scomnoob This type of crash only seems to manifest itself when using libtorrent 1.2.x - you are using 2.0.x (I am open to correction)

Stack trace of 0XC0000409 appears to be what was captured in #19745 (comment)

0XC0000409

When this is produced, it also gives GetLastError returns 0x000005AF which is "ERROR_COMMITMENT_LIMIT" (The paging file is too small for this operation to complete.) ref.: Win32 Error Codes

@arvidn Can you look at #19745 (comment) & also other exceptions/stack traces in log from #21011 (comment)

• qBittorrent 4.6.0 uses libtorrent 1.2.19+gitd28ee4eee8
• qBittorrent 4.6.5 uses libtorrent 1.2.19+git2316136434

[YouMiNope]

@stalkerok here is the torrent file I used

[stalkerok]

Piece size 256mb
Related #19535

[xavier2k6]

Piece size 256mb

torrent file from #19745 (comment) has same Piece size.

[2peer]

What is the expected memory complexity of the check operation?
Because I regularly see memory spikes of additional <torrent_data_size/3> to <torrent_data_size/2> during the check operation in qBittorrent. Definitely not something I am used to from other clients.

qBittorrent v4.6.5
libtorrent-rasterbar 2.0.10
Fedora 40 (64 bit)

[glassez]

What is the expected memory complexity of the check operation?

Perhaps @arvidn could comment on it.

libtorrent-rasterbar 2.0.10

Note that libtorrent 2.0 uses memory mapped files, so that memory can be consumed also for performing I/O operations.

[2peer]

Note that libtorrent 2.0 uses memory mapped files, so that memory can be consumed also for performing I/O operations.

From my understanding that would show up in VIRT mem in top, but not in RES/SHR & it shouldn't generally result in memory starving other processes. Just checked a 7.8GB torrent and the RES/SHR shot up from somewhere around 300MB to 5.3GB. When the check was finished it went back down. So something is amiss.

Thinking about the specifics of my setup (assuming this is not the most generally observed behavior), could a file access through a symlink be a part of the issue?

By the way, the torrent creation memory requirements are also horrendous on my system, but I can use other tools for that. The check is kind of unavoidable.

[xavier2k6]

I've tested & confirm this crash with torrents containing (piece size of 256.0 MiB) only with libtorrent 1.2 based builds.

As you can see from the statistics window below, the cache buffer grows & grows - it will use all available memory/RAM & use what storage space is available on a drive expanding the paging file......when there's no space left...it will eventually lead to a crash.

[glassez]

Has anyone investigated this issue in more detail? Are there any other conditions besides piece size: the full size of the torrent (and as a result the number of pieces), the size of the files in the torrent?

As I understand it, it doesn't crash immediately after checking is started, right? Then how would it behave if checking is paused at some point before it crashed, and then resumed?

[xavier2k6]

@glassez I've downloaded 3 files & created 3 torrents via qBittorrent v5 (LT2.0) with 256.0 MiB piece size:

Loaded them into qBittorrent 4.6.7 (LT1.2)

1. gimp-2.10.38-setup.exe = 328.5 MiB (2 x 256.0 MiB) Hybrid

• Will check & recheck without any issues

2. qt-everywhere-opensource-src-5.15.15.zip = 1.04GiB (5 x 256.0 MiB) V1

• will check but cache/total buffer size (statistics window) hits ~1GiB & will not free or appears to be held for a long period of time, left it for an hour!, any subsequent re-check will just keep increasing cache size by ~1GiB+ & no doubt will eventually lead to crash due to being out of space/memory (have to close application to regain memory being used)

3. ubunutu-24.04-desktop-amd64.iso = 5.69GiB (23 x 256.0 MiB) Hybrid

• same as (2) but increments of ~5.50GiB aka the size of the file/torrent created.

[glassez]

• same as (2) but increments of ~5.50GiB aka the size of the file/torrent created.

Doesn't the 1. have such symptoms, but in proportion to its size?

[glassez]

5. ubunutu-24.04-desktop-amd64.iso = 5.69GiB (23 x 256.0 MiB) Hybrid

• same as (2) but increments of ~5.50GiB aka the size of the file/torrent created.

How does it behave when created using 128 MiB piece size?

[xavier2k6]

Doesn't the 1. have such symptoms, but in proportion to its size?

No, it doesn't even seem to make a blip on cache

How does it behave when created using 128 MiB piece size?

1. Again not a blip
2. & 3. - cache/buffer hits 256MiB when checking & is freed afterwards

[stalkerok]

Could this setting be related?

Outstanding memory when checking torrents — (default: 32 MiB)

https://www.libtorrent.org/reference-Settings.html#checking_mem_usage
In libtorrent, the default is 256MiB.

[glassez]

https://www.libtorrent.org/reference-Settings.html#checking_mem_usage
In libtorrent, the default is 256MiB.

That's 256 blocks of 16 KiB each, so it's only 4 MiB.

Could this setting be related?

Outstanding memory when checking torrents — (default: 32 MiB)

You could test different values anyway.

[glassez]

Could this setting be related?

Outstanding memory when checking torrents — (default: 32 MiB)

I would rather sin against disk read buffers.

[xavier2k6]

Created another torrent:

744.0 MiB (3x 256.0 MiB) - recheck & cache/buffer hits 744.0 MiB & drops to 256.0 MiB which isn't freed.

[glassez]

The bad news is that this is a problem with libtorrent 1.2, not libtorrent 2.0. This reduces the chances that I will deal with it.

[xavier2k6]

The bad news is that this is a problem with libtorrent 1.2, not libtorrent 2.0. This reduces the chances that I will deal with it.

The problem though is that if certain users make torrents with the likes of py3createtorrent or other programs, we will more than likely see the same type issues with libtorrent 2.x as we are now with libtorrent 1.2, we know libtorrent 1.2 can only handle torrents/pieces of 128MiB & libtorrent 2.0 can only handle torrents/pieces of 256MiB - there's nothing stopping a user from creating torrents with 512/1024 etc.

[xavier2k6]

Libtorrent 2.0 seems to reject torrents with piece size of 512MiB/1GiB from even being loaded/logs show invalid or missing piece etc. however libtorrent 1.2 creates similar error in the logs but allows these torrents to be loaded in to session & they remain stuck on "checking resume data"

libtorrent 1.2 should flat out reject any torrent above 128MiB & not allow it to be loaded in to session.

[glassez]

The bad news is that this is a problem with libtorrent 1.2, not libtorrent 2.0. This reduces the chances that I will deal with it.

Good news. For some unknown reason, I still decided to make a "recon mission" to this problem. Fortunately, the cause of the problem was not too deep, so I seem to have detected it. I need to figure out some more details, but I'll deal with that later.
BTW, is there a related Issue in libtorrent repo? If not, could someone create it?

[xavier2k6]

BTW, is there a related Issue in libtorrent repo? If not, could someone create it?

I'll create one in the morning if nobody else does it in mean time.

[xavier2k6]

@glassez Upstream ticket has been filed, see: arvidn/libtorrent#7735

[Pryodon]

Can you share the torrent file or hash (if it's not a private torrent)?

I am having the same issue with the same files. I will share with you some torrents that cause this crash.
files.zip

[Pryodon]

When this is produced, it also gives GetLastError returns 0x000005AF which is "ERROR_COMMITMENT_LIMIT" (The paging file is too small for this operation to complete.) ref.: Win32 Error Codes

In order to successfully check those torrents that I just linked in my previous post, I had to create a page file of 300 GB!!! That is ridiculous to have to create a page file that huge. There is no legitimate reason that a person should have a page file that big! The issue is that there is a memory leak in qbit when checking these large files! It reads the entire file into memory and if there is a subsequent very large torrent file (such as the ones I just submitted) then it tries to also read that entire file into memory while the previous file is also still in memory and it doesn't matter how much memory you have reserved in your page file, the system will always run out of memory!

To successfully check those huge torrents, I had to check one of them (which takes an eternity). Then I have to completely shutdown qbit and wait for my computer to release all of that memory that was being used, then start qbit up again and then manually check the next file. If I attempt to check multiple files at once, it will crash guaranteed because it will always run out of page file memory no matter how big the page file is!

Download those torrents and see for yourself!

[xavier2k6]

@Pryodon I've downloaded those torrent files & indeed they were created using a piece size of 256.0MiB - the author of libtorrent has now ensured that these types of torrents will be rejected from even loading.

qBittorrent will include this block in it's next release & will also limit the creation of torrents to a MAX piece size of 128MiB

Your option now is to remove those torrents/find an alternative source for the file which do not exceed 128MiB piece size or re-create these torrents keeping to the supported piece size.

[Remzi1993]

What I don't why is qBittorrent providing both 1.2 and 2.0 libtorrent versions? Why not the latest stable version?

[qBittUser]

@Remzi1993 #21011 (comment)

Why is qBittorrent providing both 1.2 and 2.0 libtorrent versions?

Both branches are still developed, they work differently and have different settings to tweak.

So if there's any kind of issues with one, then the issue might not be producable with the other and it's giving one extra easy way to figure out regressions and what exactly still needs improvement.

Why not the latest stable version?

Everyone doesn't need the latest version features.

Many users still get faster more stable speed and more efficient computer resource usage with v1.2+.

Latest version from v2_0 branch has annoying flaws that still need fixing before it should be recommend to everyone.

There's no guarantee that in the future, the v2.1+ releases will immediately be good enough to replace v2.0+ & v1.2+.