Resource memory limit exceeded for tpie::priority_queue

[SSoelvsten]

The following program pushes the unsigned 64-bit integers 1, 2, ..., 20000000 to TPIE's external memory priority queue in reverse and then pulls them afterwards (this essentially just reverses the input). The amount of memory is 32MiB, whereas the 20000000 numbers take up 152MiB.

#include <tpie/tpie.h>
#include <tpie/priority_queue.h>

int main(int argc, char* argv[]) {
  tpie::tpie_init();
  tpie::get_memory_manager().set_limit(32 * 1024 * 1024);

  const size_t MAX = 20000000;

  {
    tpie::priority_queue<size_t, std::less<size_t>> _pq;

    std::cout << "pushing..." << std::endl;
    for (size_t k = MAX; k > 0; --k) {
      _pq.push(k);
    }

    std::cout << "pulling..." << std::endl;
    for (size_t k = 1; k <= MAX; ++k) {
      assert(_pq.top() == k);
      _pq.pop();
    }
  }
  tpie::tpie_finish();
}

The console output when the above is run is

pushing...
pulling...
Resource memory limit exceeded by 1MiB (6%), while trying to increase usage by 2MiB. Limit is 32MiB, but 33MiB would be used.
Resource memory limit exceeded by 3MiB (12%), while trying to increase usage by 2MiB. Limit is 32MiB, but 35MiB would be used.
Resource memory limit exceeded by 5MiB (18%), while trying to increase usage by 2MiB. Limit is 32MiB, but 37MiB would be used.
Resource memory limit exceeded by 7MiB (24%), while trying to increase usage by 2MiB. Limit is 32MiB, but 39MiB would be used.
Resource memory limit exceeded by 9MiB (31%), while trying to increase usage by 2MiB. Limit is 32MiB, but 41MiB would be used.
Resource memory limit exceeded by 11MiB (37%), while trying to increase usage by 2MiB. Limit is 32MiB, but 43MiB would be used.
Resource memory limit exceeded by 13MiB (43%), while trying to increase usage by 2MiB. Limit is 32MiB, but 45MiB would be used.
Resource memory limit exceeded by 15MiB (49%), while trying to increase usage by 2MiB. Limit is 32MiB, but 47MiB would be used.
Resource memory limit exceeded by 17MiB (56%), while trying to increase usage by 2MiB. Limit is 32MiB, but 49MiB would be used.
Resource memory limit exceeded by 21MiB (68%), while trying to increase usage by 2MiB. Limit is 32MiB, but 53MiB would be used.
Resource memory limit exceeded by 25MiB (81%), while trying to increase usage by 2MiB. Limit is 32MiB, but 57MiB would be used.
Resource memory limit exceeded by 29MiB (93%), while trying to increase usage by 2MiB. Limit is 32MiB, but 61MiB would be used.
Resource memory limit exceeded by 33MiB (106%), while trying to increase usage by 2MiB. Limit is 32MiB, but 65MiB would be used.
Resource memory limit exceeded by 39MiB (124%), while trying to increase usage by 2MiB. Limit is 32MiB, but 71MiB would be used.
Resource memory limit exceeded by 45MiB (143%), while trying to increase usage by 2MiB. Limit is 32MiB, but 77MiB would be used.

This is on a Fedora 36 machine and building sources with

• GCC version 12.2.1 20220819 (Red Hat 12.2.1-2)
• CMake version 3.24.2
  • CMAKE_BUILD_TYPE = Debug
  • CMAKE_C_FLAGS = -g -O2
  • CMAKE_CXX_FLAGS = -g -O2
• Boost version 1.76.0 and release 12.fc36

[SSoelvsten]

The priority queue settings, as they are printed from the tpie::priority_queue::init function are.

priority_queue: Memory limit: 25mb(26214400bytes)
m_for_queue: 26214400
memory before alloc: 29358800b
Memory used by file_stream: 131592b

fanout_overhead     131632,
sq_fanout_overhead  24,
heap_m_overhead     40,
buffer_m_overhead   24,
extra_overhead      263232,
additional_overhead 16384.

mm_avail      25934784,
setting_mmark 67538.

mm_avail      24313872,
setting_k     12156936.

mm_avail      24115872,
setting_m     602896,
setting_k     90.

priority_queue
	setting_k: 90
	setting_mmark: 67538
	setting_m: 602896

memory after alloc: 9264392b

[SSoelvsten]

If MAX is set to 2000000000 (1.5 GiB of data) then the memory limit is already exceeded during the initial pushing of elements.

pushing...
Resource memory limit exceeded by 1MiB (6%), while trying to increase usage by 2MiB. Limit is 32MiB, but 33MiB would be used.
Resource memory limit exceeded by 3MiB (12%), while trying to increase usage by 2MiB. Limit is 32MiB, but 35MiB would be used.
Resource memory limit exceeded by 5MiB (18%), while trying to increase usage by 2MiB. Limit is 32MiB, but 37MiB would be used.
Resource memory limit exceeded by 7MiB (24%), while trying to increase usage by 2MiB. Limit is 32MiB, but 39MiB would be used.
Resource memory limit exceeded by 9MiB (31%), while trying to increase usage by 2MiB. Limit is 32MiB, but 41MiB would be used.
Resource memory limit exceeded by 11MiB (37%), while trying to increase usage by 2MiB. Limit is 32MiB, but 43MiB would be used.
Resource memory limit exceeded by 13MiB (43%), while trying to increase usage by 2MiB. Limit is 32MiB, but 45MiB would be used.
Resource memory limit exceeded by 15MiB (49%), while trying to increase usage by 2MiB. Limit is 32MiB, but 47MiB would be used.
Resource memory limit exceeded by 17MiB (56%), while trying to increase usage by 2MiB. Limit is 32MiB, but 49MiB would be used.
Resource memory limit exceeded by 21MiB (68%), while trying to increase usage by 2MiB. Limit is 32MiB, but 53MiB would be used.
Resource memory limit exceeded by 25MiB (81%), while trying to increase usage by 2MiB. Limit is 32MiB, but 57MiB would be used.
Resource memory limit exceeded by 29MiB (93%), while trying to increase usage by 2MiB. Limit is 32MiB, but 61MiB would be used.
Resource memory limit exceeded by 33MiB (106%), while trying to increase usage by 2MiB. Limit is 32MiB, but 65MiB would be used.
Resource memory limit exceeded by 39MiB (124%), while trying to increase usage by 2MiB. Limit is 32MiB, but 71MiB would be used.
Resource memory limit exceeded by 45MiB (143%), while trying to increase usage by 2MiB. Limit is 32MiB, but 77MiB would be used.
Resource memory limit exceeded by 51MiB (162%), while trying to increase usage by 2MiB. Limit is 32MiB, but 83MiB would be used.
Resource memory limit exceeded by 59MiB (187%), while trying to increase usage by 2MiB. Limit is 32MiB, but 91MiB would be used.
Resource memory limit exceeded by 67MiB (212%), while trying to increase usage by 2MiB. Limit is 32MiB, but 99MiB would be used.
Resource memory limit exceeded by 77MiB (243%), while trying to increase usage by 2MiB. Limit is 32MiB, but 109MiB would be used.
Resource memory limit exceeded by 87MiB (274%), while trying to increase usage by 2MiB. Limit is 32MiB, but 119MiB would be used.
Resource memory limit exceeded by 99MiB (312%), while trying to increase usage by 2MiB. Limit is 32MiB, but 131MiB would be used.
Resource memory limit exceeded by 113MiB (356%), while trying to increase usage by 2MiB. Limit is 32MiB, but 145MiB would be used.
Resource memory limit exceeded by 129MiB (406%), while trying to increase usage by 2MiB. Limit is 32MiB, but 161MiB would be used.
Resource memory limit exceeded by 147MiB (462%), while trying to increase usage by 2MiB. Limit is 32MiB, but 179MiB would be used.