Memory accounting error in sort

[thomasmoelhave]

I have noticed that many of my pipeline programs report that the memory limit was exceeded by 2MiB (about the memory footprint of a file_stream?). It turns out that this issue can be reproduced using tpie's own tests:

tpie/test$ ./test_sort  -s 100M -m 50M

Gives me the following output:

Path:  /var/tmp/
Num Items: 6553601
Item Size: 16
File Size: 100.00 MB

Testing sorting [===========================>                                                       ] 33% Sorting > Write sorted runs 0 secMemory limit exceeded by 2MiB (4%), while trying to allocate 2MiB. Limit is 50MiB, but 52MiB would be used.
Testing sorting [==================================================================================>] 100% 0 sec
Test ran successfully 

The same thing happens with the sorting speed tests:

tpie/test/speed_regression$ ./sort_speed_test  1 100 50
-------------------------------------------------------------------------------
Sort speed test

Hostname:       wads
Platform:       Linux 64-bit
Git branch:     
Git commit:     d055c7d91d083ecf0d5f548f7eac5e1975bd7c66
Local time:     2014-Aug-18 17:07:38
Block size:     2048 KiB
Parallel sort:  Enabled
Snappy:         Enabled

Memory (MB):    50
Data (MB):      100

            Test           Write            Sort            Hash
               1              57Memory limit exceeded by 2MiB (4%), while trying to allocate 2MiB. Limit is 50MiB, but 52MiB would be used.
             471  19543143170048
            mean         57.0000         471.000  19543143170048
          stddev            -nan            -nan            -nan
End time:       2014-Aug-18 17:07:39
Read (MB):      298.506
Written (MB):   298.507
Blocked:        111
Held:           147794
Waiting:        27841
Reading:        100179
Writing:        120786
Compressing:    11340
Uncompressing:  6636
Snappy-blocks:  3
None-blocks:    48

[Mortal]

Aha, a memory usage warning? That must be a memory accounting bug! Raise the memory limits everywhere by 2 MB!

... Just kidding. I would guess this is a regression after we merged snappy. I won't have time to look into it in the upcoming week, but I'll fix it in due time if no one else does.

[thomasmoelhave]

Excellent, thanks @Mortal!

[SSoelvsten]

I cannot seem to find a commit that tried to fix this, and it seems to be a problem that still persists to this day, even if Snappy is not activated.

./test/test_sort  -s 100M -m 50M
Path:  /var/tmp/
Num Items: 6553601
Item Size: 16
File Size: 100.00 MB

Testing sorting [===========================>                                                       ] 33% Sorting > Write sorted runs 0 secResource memory limit exceeded by 1MiB (3%), while trying to increase usage by 2MiB. Limit is 50MiB, but 51MiB would be used.
Testing sorting [==================================================================================>] 100% 0 sec
Test ran successfully

./test/speed_regression/sort_speed_test  1 100 50
-------------------------------------------------------------------------------
Sort speed test

Hostname:      _____________
Platform:       Linux 64-bit
Git branch:     refs/heads/std_uncaught_exception_deprecation
Git commit:     ef13fb620db96d5b6df41d0d92fc21776440e68a
Local time:     Error
Block size:     2048 KiB
Parallel sort:  Disabled
Snappy:         Disabled

Memory (MB):    50
Data (MB):      100

            Test           Write            Sort            Hash
               1             134Resource memory limit exceeded by 1MiB (3%), while trying to increase usage by 2MiB. Limit is 50MiB, but 51MiB would be used.
           15122  19543143170048
            mean         134.000         15122.0  19543143170048
          stddev            -nan            -nan            -nan
End time:       Error
Read (MB):      300
Written (MB):   300
Blocked:        387
Held:           635440
Waiting:        5530
Reading:        79058
Writing:        101765
Compressing:    0
Uncompressing:  0
Snappy-blocks:  49
None-blocks:    2

[SSoelvsten]

Could this potentially be the underlying problem for SSoelvsten/adiar#388 ? I've looked through my own code and cannot find any opened file streams that I am not accounting for.

EDIT: No, my problem is due to #264 .