Realm: core->get_resource("cpu", ...) ignores cpuset

[manopapad]

If I use taskset to restrict the cores available to Realm, it will correctly abide by this restriction when creating the processors, but the query API will wrongfully report that all cores are available.

Unconstrained execution

(b/env) iblis:~/b/legate> LEGION_DEFAULT_ARGS="-ll:show_rsrv" legate b.py
found 32 cores
core map {
  domain 0 {
    core 0 { ids=<0> alu=<32> fpu=<32> ldst=<32> }
    core 1 { ids=<1> alu=<33> fpu=<33> ldst=<33> }
    core 2 { ids=<2> alu=<34> fpu=<34> ldst=<34> }
    core 3 { ids=<3> alu=<35> fpu=<35> ldst=<35> }
    core 4 { ids=<4> alu=<36> fpu=<36> ldst=<36> }
    core 5 { ids=<5> alu=<37> fpu=<37> ldst=<37> }
    core 6 { ids=<6> alu=<38> fpu=<38> ldst=<38> }
    core 7 { ids=<7> alu=<39> fpu=<39> ldst=<39> }
    core 8 { ids=<8> alu=<40> fpu=<40> ldst=<40> }
    core 9 { ids=<9> alu=<41> fpu=<41> ldst=<41> }
    core 10 { ids=<10> alu=<42> fpu=<42> ldst=<42> }
    core 11 { ids=<11> alu=<43> fpu=<43> ldst=<43> }
    core 12 { ids=<12> alu=<44> fpu=<44> ldst=<44> }
    core 13 { ids=<13> alu=<45> fpu=<45> ldst=<45> }
    core 14 { ids=<14> alu=<46> fpu=<46> ldst=<46> }
    core 15 { ids=<15> alu=<47> fpu=<47> ldst=<47> }
    core 16 { ids=<16> alu=<48> fpu=<48> ldst=<48> }
    core 17 { ids=<17> alu=<49> fpu=<49> ldst=<49> }
    core 18 { ids=<18> alu=<50> fpu=<50> ldst=<50> }
    core 19 { ids=<19> alu=<51> fpu=<51> ldst=<51> }
    core 20 { ids=<20> alu=<52> fpu=<52> ldst=<52> }
    core 21 { ids=<21> alu=<53> fpu=<53> ldst=<53> }
    core 22 { ids=<22> alu=<54> fpu=<54> ldst=<54> }
    core 23 { ids=<23> alu=<55> fpu=<55> ldst=<55> }
    core 24 { ids=<24> alu=<56> fpu=<56> ldst=<56> }
    core 25 { ids=<25> alu=<57> fpu=<57> ldst=<57> }
    core 26 { ids=<26> alu=<58> fpu=<58> ldst=<58> }
    core 27 { ids=<27> alu=<59> fpu=<59> ldst=<59> }
    core 28 { ids=<28> alu=<60> fpu=<60> ldst=<60> }
    core 29 { ids=<29> alu=<61> fpu=<61> ldst=<61> }
    core 30 { ids=<30> alu=<62> fpu=<62> ldst=<62> }
    core 31 { ids=<31> alu=<63> fpu=<63> ldst=<63> }
    core 32 { ids=<32> alu=<0> fpu=<0> ldst=<0> }
    core 33 { ids=<33> alu=<1> fpu=<1> ldst=<1> }
    core 34 { ids=<34> alu=<2> fpu=<2> ldst=<2> }
    core 35 { ids=<35> alu=<3> fpu=<3> ldst=<3> }
    core 36 { ids=<36> alu=<4> fpu=<4> ldst=<4> }
    core 37 { ids=<37> alu=<5> fpu=<5> ldst=<5> }
    core 38 { ids=<38> alu=<6> fpu=<6> ldst=<6> }
    core 39 { ids=<39> alu=<7> fpu=<7> ldst=<7> }
    core 40 { ids=<40> alu=<8> fpu=<8> ldst=<8> }
    core 41 { ids=<41> alu=<9> fpu=<9> ldst=<9> }
    core 42 { ids=<42> alu=<10> fpu=<10> ldst=<10> }
    core 43 { ids=<43> alu=<11> fpu=<11> ldst=<11> }
    core 44 { ids=<44> alu=<12> fpu=<12> ldst=<12> }
    core 45 { ids=<45> alu=<13> fpu=<13> ldst=<13> }
    core 46 { ids=<46> alu=<14> fpu=<14> ldst=<14> }
    core 47 { ids=<47> alu=<15> fpu=<15> ldst=<15> }
    core 48 { ids=<48> alu=<16> fpu=<16> ldst=<16> }
    core 49 { ids=<49> alu=<17> fpu=<17> ldst=<17> }
    core 50 { ids=<50> alu=<18> fpu=<18> ldst=<18> }
    core 51 { ids=<51> alu=<19> fpu=<19> ldst=<19> }
    core 52 { ids=<52> alu=<20> fpu=<20> ldst=<20> }
    core 53 { ids=<53> alu=<21> fpu=<21> ldst=<21> }
    core 54 { ids=<54> alu=<22> fpu=<22> ldst=<22> }
    core 55 { ids=<55> alu=<23> fpu=<23> ldst=<23> }
    core 56 { ids=<56> alu=<24> fpu=<24> ldst=<24> }
    core 57 { ids=<57> alu=<25> fpu=<25> ldst=<25> }
    core 58 { ids=<58> alu=<26> fpu=<26> ldst=<26> }
    core 59 { ids=<59> alu=<27> fpu=<27> ldst=<27> }
    core 60 { ids=<60> alu=<28> fpu=<28> ldst=<28> }
    core 61 { ids=<61> alu=<29> fpu=<29> ldst=<29> }
    core 62 { ids=<62> alu=<30> fpu=<30> ldst=<30> }
    core 63 { ids=<63> alu=<31> fpu=<31> ldst=<31> }
  }
}
GPU ctxsync 0x56def0846fd0: allocated <27,28,29,30,31,59,60,61,62,63>
utility proc 1d00000000000000: allocated <27,28,29,30,31,59,60,61,62,63>
GPU ctxsync 0x56def0842740: allocated <27,28,29,30,31,59,60,61,62,63>
dedicated worker (generic) #2: allocated <27,28,29,30,31,59,60,61,62,63>
dedicated worker (generic) #1: allocated <27,28,29,30,31,59,60,61,62,63>
utility proc 1d00000000000001: allocated <27,28,29,30,31,59,60,61,62,63>
CPU proc 1d00000000000002: allocated <26>
OMP0 proc 1d00000000000003 (master): allocated <0>
OMP0 proc 1d00000000000003 (worker 1): allocated <1>
OMP0 proc 1d00000000000003 (worker 2): allocated <2>
OMP0 proc 1d00000000000003 (worker 3): allocated <3>
OMP0 proc 1d00000000000003 (worker 4): allocated <4>
OMP0 proc 1d00000000000003 (worker 5): allocated <5>
OMP0 proc 1d00000000000003 (worker 6): allocated <6>
OMP0 proc 1d00000000000003 (worker 7): allocated <7>
OMP0 proc 1d00000000000003 (worker 8): allocated <8>
OMP0 proc 1d00000000000003 (worker 9): allocated <9>
OMP0 proc 1d00000000000003 (worker 10): allocated <10>
OMP0 proc 1d00000000000003 (worker 11): allocated <11>
OMP0 proc 1d00000000000003 (worker 12): allocated <12>
OMP0 proc 1d00000000000004 (master): allocated <13>
OMP0 proc 1d00000000000004 (worker 1): allocated <14>
OMP0 proc 1d00000000000004 (worker 2): allocated <15>
OMP0 proc 1d00000000000004 (worker 3): allocated <16>
OMP0 proc 1d00000000000004 (worker 4): allocated <17>
OMP0 proc 1d00000000000004 (worker 5): allocated <18>
OMP0 proc 1d00000000000004 (worker 6): allocated <19>
OMP0 proc 1d00000000000004 (worker 7): allocated <20>
OMP0 proc 1d00000000000004 (worker 8): allocated <21>
OMP0 proc 1d00000000000004 (worker 9): allocated <22>
OMP0 proc 1d00000000000004 (worker 10): allocated <23>
OMP0 proc 1d00000000000004 (worker 11): allocated <24>
OMP0 proc 1d00000000000004 (worker 12): allocated <25>
GPU proc 1d00000000000005: allocated <27,28,29,30,31,59,60,61,62,63>
GPU proc 1d00000000000006: allocated <27,28,29,30,31,59,60,61,62,63>

Constrained execution

(b/env) iblis:~/b/legate> LEGION_DEFAULT_ARGS="-ll:show_rsrv" taskset --cpu-list 0-4 legate b.py
found 32 cores
[0 - 763994f02440]    0.000000 {4}{openmp}: not enough cores in NUMA domain 0 (5 < 26)
[0 - 763994f02440]    0.000207 {5}{numa}: can't read '/sys/devices/system/node/node-1/distance': No such file or directory
[0 - 763994f02440]    0.000624 {4}{threads}: reservation ('OMP-1 proc 1d00000000000003 (worker 4)') cannot be satisfied
core map {
  domain 0 {
    core 0 { ids=<0> }
    core 1 { ids=<1> }
    core 2 { ids=<2> }
    core 3 { ids=<3> }
    core 4 { ids=<4> }
  }
}
GPU ctxsync 0x6121c8756970: allocated <>
utility proc 1d00000000000000: allocated <>
GPU ctxsync 0x6121c87520e0: allocated <>
dedicated worker (generic) #2: allocated <>
dedicated worker (generic) #1: allocated <>
utility proc 1d00000000000001: allocated <>
CPU proc 1d00000000000002: allocated <>
OMP-1 proc 1d00000000000003 (master): allocated <>
OMP-1 proc 1d00000000000003 (worker 1): allocated <>
OMP-1 proc 1d00000000000003 (worker 2): allocated <>
OMP-1 proc 1d00000000000003 (worker 3): allocated <>
OMP-1 proc 1d00000000000003 (worker 4): allocated <>
OMP-1 proc 1d00000000000003 (worker 5): allocated <>
OMP-1 proc 1d00000000000003 (worker 6): allocated <>
OMP-1 proc 1d00000000000003 (worker 7): allocated <>
OMP-1 proc 1d00000000000003 (worker 8): allocated <>
OMP-1 proc 1d00000000000003 (worker 9): allocated <>
OMP-1 proc 1d00000000000003 (worker 10): allocated <>
OMP-1 proc 1d00000000000003 (worker 11): allocated <>
OMP-1 proc 1d00000000000003 (worker 12): allocated <>
OMP-1 proc 1d00000000000004 (master): allocated <>
OMP-1 proc 1d00000000000004 (worker 1): allocated <>
OMP-1 proc 1d00000000000004 (worker 2): allocated <>
OMP-1 proc 1d00000000000004 (worker 3): allocated <>
OMP-1 proc 1d00000000000004 (worker 4): allocated <>
OMP-1 proc 1d00000000000004 (worker 5): allocated <>
OMP-1 proc 1d00000000000004 (worker 6): allocated <>
OMP-1 proc 1d00000000000004 (worker 7): allocated <>
OMP-1 proc 1d00000000000004 (worker 8): allocated <>
OMP-1 proc 1d00000000000004 (worker 9): allocated <>
OMP-1 proc 1d00000000000004 (worker 10): allocated <>
OMP-1 proc 1d00000000000004 (worker 11): allocated <>
OMP-1 proc 1d00000000000004 (worker 12): allocated <>
GPU proc 1d00000000000005: allocated <>
GPU proc 1d00000000000006: allocated <>

[eddy16112]

Right, because the resource discovery does not read the cpu core map of the current process. I am working on a branch https://gitlab.com/StanfordLegion/legion/-/commits/topology, which will unify the topology discovery and it can be used by core and numa modules, as well as coremap creation. The bug will be fixed in that branch.