Realm: assert at startup

[mariodirenzo]

Because of changes in the master branch between a8906edfa and cc5db48d4, I cannot start any legion program on a system I utilize for the CI of HTR++.
In particular, I get the following assertion

/home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc:789: Realm::HardwareTopology::HardwareTopology(const std::vector<Proc>&, const std::vector<MemoryInfo>&, size_t): Assertion `num_logical_cores_per_physical_core == proc.shares_fpu.size() + 1' failed.

This is the backtrace

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fffe2e4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fffe2e288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fffe2e2881b in __assert_fail_base (fmt=0x7fffe2fd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555558b8b708 "num_logical_cores_per_physical_core == proc.shares_fpu.size() + 1", file=file@entry=0x555558b8b508 "/home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc", line=line@entry=789,
    function=function@entry=0x555558b8b668 "Realm::HardwareTopology::HardwareTopology(const std::vector<Proc>&, const std::vector<MemoryInfo>&, size_t)") at ./assert/assert.c:96
#6  0x00007fffe2e3b517 in __assert_fail (assertion=0x555558b8b708 "num_logical_cores_per_physical_core == proc.shares_fpu.size() + 1", file=0x555558b8b508 "/home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc", line=789, function=0x555558b8b668 "Realm::HardwareTopology::HardwareTopology(const std::vector<Proc>&, const std::vector<MemoryInfo>&, size_t)")
    at ./assert/assert.c:105
#7  0x0000555557ff20b4 in Realm::HardwareTopology::HardwareTopology (this=0x7fffffffb9c0, logical_cores=std::vector of length 32, capacity 32 = {...}, memories=std::vector of length 1, capacity 1 = {...}, host_memory=202388615168) at /home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc:789
#8  0x0000555557ff1c47 in Realm::HardwareTopology::create_topology () at /home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc:760
#9  0x0000555557f7bf75 in Realm::RuntimeImpl::create_configs (this=0x5555598960f0, argc=2, argv=0x7fffffffd9f8) at /home/mdirenzo/legion/src/runtime/realm/runtime_impl.cc:2910
#10 0x0000555557f70bd1 in Realm::Runtime::create_configs (this=0x7fffffffbb38, argc=2, argv=0x7fffffffd9f8) at /home/mdirenzo/legion/src/runtime/realm/runtime_impl.cc:777
#11 0x0000555556c35065 in Legion::Internal::Runtime::initialize (argc=0x7fffffffd62c, argv=0x7fffffffd620, parse=true, filter=false) at /home/mdirenzo/legion/src/runtime/legion/runtime.cc:34160
#12 0x0000555556c346e1 in Legion::Internal::Runtime::start (argc=2, argv=0x7fffffffd9f8, background=false, supply_default_mapper=true, filter=false) at /home/mdirenzo/legion/src/runtime/legion/runtime.cc:34017
#13 0x0000555556b013e6 in Legion::Runtime::start (argc=2, argv=0x7fffffffd9f8, background=false, default_mapper=true, filter=false) at /home/mdirenzo/legion/src/runtime/legion/legion.cc:7311
#14 0x00007fffe2e2a1ca in __libc_start_call_main (main=main@entry=0x555556a416f0 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffd9f8) at ../sysdeps/nptl/libc_start_call_main.h:58
#15 0x00007fffe2e2a28b in __libc_start_main_impl (main=0x555556a416f0 <main>, argc=2, argv=0x7fffffffd9f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd9e8) at ../csu/libc-start.c:360
#16 0x0000555556a420a5 in _start ()

and

(gdb) f 7
#7  0x0000555557ff20b4 in Realm::HardwareTopology::HardwareTopology (this=0x7fffffffb9c0, logical_cores=std::vector of length 32, capacity 32 = {...}, memories=std::vector of length 1, capacity 1 = {...}, host_memory=202388615168) at /home/mdirenzo/legion/src/runtime/realm/hardware_topology.cc:789
789	        assert(num_logical_cores_per_physical_core == proc.shares_fpu.size() + 1);
(gdb) p num_logical_cores_per_physical_core
$1 = 2
(gdb) p proc.shares_fpu
$2 = std::set with 0 elements

Do you have any idea of what is going on?

@elliottslaughter, can you please add this issue to #1032 ?

[elliottslaughter]

@eddy16112 have you touched the machine topology recently?

[eddy16112]

Yes, I recently refactored the topology. @mariodirenzo Could you please try to remove the assertion, and run with -ll:show_rsrv? I would like to see the topology of the machine.

[mariodirenzo]

This is the core map generated with the legion release 2024.12. I hit the assertion before obtaining the output on the new version of the runtime

core map {
  domain 0 {
    core 0 { ids=<0> alu=<1> fpu=<1> ldst=<1> }
    core 1 { ids=<1> alu=<0> fpu=<0> ldst=<0> }
    core 2 { ids=<2> alu=<3> fpu=<3> ldst=<3> }
    core 3 { ids=<3> alu=<2> fpu=<2> ldst=<2> }
    core 4 { ids=<4> alu=<5> fpu=<5> ldst=<5> }
    core 5 { ids=<5> alu=<4> fpu=<4> ldst=<4> }
    core 6 { ids=<6> alu=<7> fpu=<7> ldst=<7> }
    core 7 { ids=<7> alu=<6> fpu=<6> ldst=<6> }
    core 8 { ids=<8> alu=<9> fpu=<9> ldst=<9> }
    core 9 { ids=<9> alu=<8> fpu=<8> ldst=<8> }
    core 10 { ids=<10> alu=<11> fpu=<11> ldst=<11> }
    core 11 { ids=<11> alu=<10> fpu=<10> ldst=<10> }
    core 12 { ids=<12> alu=<13> fpu=<13> ldst=<13> }
    core 13 { ids=<13> alu=<12> fpu=<12> ldst=<12> }
    core 14 { ids=<14> alu=<15> fpu=<15> ldst=<15> }
    core 15 { ids=<15> alu=<14> fpu=<14> ldst=<14> }
    core 16 { ids=<16> }
    core 17 { ids=<17> }
    core 18 { ids=<18> }
    core 19 { ids=<19> }
    core 20 { ids=<20> }
    core 21 { ids=<21> }
    core 22 { ids=<22> }
    core 23 { ids=<23> }
    core 24 { ids=<24> }
    core 25 { ids=<25> }
    core 26 { ids=<26> }
    core 27 { ids=<27> }
    core 28 { ids=<28> }
    core 29 { ids=<29> }
    core 30 { ids=<30> }
    core 31 { ids=<31> }
  }
}
GPU ctxsync 0x56f9289c1bd0: allocated <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31>
dedicated worker (generic) #1: allocated <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31>
dedicated worker (generic) #2: allocated <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31>
utility proc 1d00000000000000: allocated <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31>
CPU proc 1d00000000000001: allocated <0>
GPU proc 1d00000000000002: allocated <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31>

[eddy16112]

I see the problem, why some cores (core 0-15) have hyperthreading enabled, while others (core 16-31) are not? Currently, I put a verification to check if all cores have the same hyperthreading configuration.

[mariodirenzo]

Not sure about it. Though deactivating the hyperthreading of the system fixes the issue.