Resource memory limit exceeded

[SSoelvsten]

When running the following benchmarks with the minimum of 128 MiB of memory, we get the following error message.

Resource memory limit exceeded by _MiB (_%), while trying to increase usage by 2MiB. Limit is 128MiB, but ___MiB would be used.

Specifically, this happens in the following benchmarks.

• EPFL: random_control/arbiter_depth
• Queens (BDD): 15
• Tic-Tac-Toe (BDD / ZDD): 23, 24, 25...

This seems like there is a missing accounting for some file stream. After lots of digging (and squashing a few bugs that might have been the culprit) this turns out to be an issue in TPIE. I have reported this at thomasmoelhave/tpie#264 .

[SSoelvsten]

If running with CMAKE_BUILD_TYPE set to Debug, then the assertions for the internal memory versions getting enough space is not satisfied. Maybe some of our memory_usage functions are not valid? Digging a bit deeper, we see this is due to the following two bugs of mine:

• The memory_fits of the internal_sorter is not always smaller than the memory_fits of the same internal_priority_queue.
• The levelized priority queue does not change its m_overflow_queue computation based on whether it is internal or external memory.

Fixing this does not seem to address this bug.

As I was looking at this, I also noticed the following issues in the memory computation

• The m_overflow_queue computation does not take the merger into account.

Fixing this neither solves this issue.

[SSoelvsten]

This seems to be a problem with TPIE's external memory priority queue taking up more space than it is given when it has to deal with too large instances.

thomasmoelhave/tpie#264

[nhusung]

I tried to use the bdd-benchmark set to compare a new BDD library to existing ones. However, I’m experiencing problems with Adiar:

$ build/src/adiar_queens_bdd -N 11 -M 4096
11-Queens (Adiar [BDD] 4096 MiB):

   Adiar [BDD] initialisation:
   | time (ms):              1

   Decision diagram construction:
Assert failed:  memory_fits and memory_usage should agree.
Expected:       unsafe_memory_usage(ret) <= memory_bytes
Source:         /[...]/bdd-benchmark/external/adiar/src/adiar/../adiar/internal/data_structures/sorter.h, line 56

Interestingly, I couldn’t observe a fail of this assertion in debug mode. Even in release mode, when inserting a debug print just above the mentioned line, the assert did not fail. Is this problem related?

(I’m on Linux 6.4 and use clang 15.0.7)

[SSoelvsten]

This issue, #388, is due to the bug #264 of my dependency on the TPIE library

That debug statement on the other hand is to ensure that the number memory_bytes which is obtained from calling a function tpie::___::memory_fits(...) (how much fits into x bytes of memory?) agrees(ish) with its sibling function tpie::___::memory_usage(...) (how much memory does x elements occupy?).

I'll move your problem into a separate issue, if you don't mind. 🙂