Error in mlx5dv_create_qp in the DC transport #5749

lyu · 2020-09-29T17:31:35Z

Describe the bug

ucx_info and ucx_perftest reports dc_mlx5.c:329 UCX ERROR mlx5dv_create_qp(mlx5_0:1, DCI): failed: Invalid argument.

Steps to Reproduce

UCX version: UCT version=1.10.0 revision c7add93
UCX build config: --prefix=$PREFIX --enable-debug --enable-assertions --enable-params-check --enable-frame-pointer --enable-backtrace-detail

Setup and versions

lsb_release -a:

LSB Version:	:core-4.1-aarch64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 8.1.1911 (Core) 
Release:	8.1.1911
Codename:	Core

ofed_info -s: MLNX_OFED_LINUX-5.1-0.6.6.0
rpm -q rdma-core: rdma-core-51mlnx1-1.51066.aarch64
rpm -q libibverbs: libibverbs-51mlnx1-1.51066.aarch64

Additional information (depending on the issue)

For ucx_info -d, this happens when it tries to print info about the dc_mlx5 transport.
For ucx_perftest, it happens when running any UCP test without any environment variable set.

All issues go away if I add --without-dc to the configure script.

This doesn't happen with UCX 1.9.0, dc transport will be enabled and work correctly.

This also doesn't happen when built against MLNX_OFED_LINUX-4.5-1.0.1.0 on another ThunderX2 machine, but it looks like dc is automatically disabled there.

The text was updated successfully, but these errors were encountered:

yosefe · 2020-09-29T17:34:10Z

@lyu this is MLNX_OFED issue. Can you pls try MLNX_OFED 5.1-2 or higher?

lyu · 2020-09-29T17:39:31Z

@yosefe Sorry can't do, I don't have admin access.

tonycurtis · 2020-09-30T18:22:25Z

Similar issues here with TX2 and MLNX_OFED_LINUX-5.1-0.6.6.0

I just built UCX --without-dc for now.

alex--m · 2021-08-21T07:11:53Z

@yosefe it reproduces on the course setup in HUJI, which doesn't have MOFED (I can give you access to it, it's 4 hosts with CX3). Looks like UCX incorrectly detects DC is supported on the device when in fact it isn't, and fails the entire worker when DCT creation fails. This is still relevant to the current master branch.

BTW - a simpler workaround (without rebuilding UCX) would be setting UCX_TLS (this is what I do on that setup until this is resolved).

yosefe · 2021-08-21T12:02:40Z

@alex--m can you pls upload the output of "ucx_info -dvb" command, with UCX_LOG_LEVEL=data?

alex--m · 2021-08-22T07:08:36Z

[1629615907.871546] [mlx-stud-01:26236:0]           stats.c:861  UCX  TRACE statistics disabled
[1629615907.871566] [mlx-stud-01:26236:0]        memtrack.c:379  UCX  TRACE memtrack disabled
[1629615907.871587] [mlx-stud-01:26236:0]           debug.c:1197 UCX  DEBUG using signal stack 0x7f23d820d000 size 141824
[1629615907.884343] [mlx-stud-01:26236:0]            init.c:115  UCX  DEBUG /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/libucs.so.0 loaded at 0x7f23d8230000
[1629615907.884375] [mlx-stud-01:26236:0]            init.c:116  UCX  DEBUG cmd line: ./build/bin/ucx_info -dvb 
[1629615907.884394] [mlx-stud-01:26236:0]          module.c:69   UCX  DEBUG ucs library path: /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/libucs.so.0
[1629615907.884403] [mlx-stud-01:26236:0]          module.c:251  UCX  DEBUG loading modules for ucs
# UCT version=1.12.0 revision dd824bc
# configured with: --enable-gtest --enable-examples --with-valgrind --enable-profiling --enable-frame-pointer --enable-stats --enable-fault-injection --enable-debug-data --enable-mt --prefix=/cs/labs/amnon/alexam02/ucg/ucx-orig/build
#define UCX_CONFIG_H              
#define ENABLE_ASSERT             1
#define ENABLE_BUILTIN_MEMCPY     1
#define ENABLE_DEBUG_DATA         1
#define ENABLE_FAULT_INJECTION    1
#define ENABLE_MT                 1
#define ENABLE_PARAMS_CHECK       1
#define ENABLE_STATS              1
#define HAVE_1_ARG_BFD_SECTION_SIZE 0
#define HAVE_ALLOCA               1
#define HAVE_ALLOCA_H             1
#define HAVE_ATTRIBUTE_NOOPTIMIZE 1
#define HAVE_CLEARENV             1
#define HAVE_CPLUS_DEMANGLE       1
#define HAVE_CPU_SET_T            1
#define HAVE_DC_DV                1
#define HAVE_DECL_ASPRINTF        1
#define HAVE_DECL_BASENAME        1
#define HAVE_DECL_BFD_GET_SECTION_FLAGS 1
#define HAVE_DECL_BFD_GET_SECTION_VMA 1
#define HAVE_DECL_BFD_SECTION_FLAGS 0
#define HAVE_DECL_BFD_SECTION_VMA 1
#define HAVE_DECL_CPU_ISSET       1
#define HAVE_DECL_CPU_ZERO        1
#define HAVE_DECL_ETHTOOL_CMD_SPEED 1
#define HAVE_DECL_FMEMOPEN        1
#define HAVE_DECL_FUSE_MOUNT      0
#define HAVE_DECL_FUSE_OPEN_CHANNEL 0
#define HAVE_DECL_FUSE_UNMOUNT    0
#define HAVE_DECL_F_SETOWN_EX     1
#define HAVE_DECL_IBV_ACCESS_ON_DEMAND 1
#define HAVE_DECL_IBV_ACCESS_RELAXED_ORDERING 0
#define HAVE_DECL_IBV_ADVISE_MR   1
#define HAVE_DECL_IBV_ALLOC_DM    1
#define HAVE_DECL_IBV_ALLOC_TD    1
#define HAVE_DECL_IBV_CMD_MODIFY_QP 0
#define HAVE_DECL_IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN 1
#define HAVE_DECL_IBV_CREATE_QP_EX 1
#define HAVE_DECL_IBV_CREATE_SRQ  1
#define HAVE_DECL_IBV_CREATE_SRQ_EX 1
#define HAVE_DECL_IBV_EVENT_GID_CHANGE 1
#define HAVE_DECL_IBV_EVENT_TYPE_STR 1
#define HAVE_DECL_IBV_EXP_ACCESS_ALLOCATE_MR 0
#define HAVE_DECL_IBV_EXP_ACCESS_ON_DEMAND 0
#define HAVE_DECL_IBV_EXP_ALLOC_DM 0
#define HAVE_DECL_IBV_EXP_ATOMIC_HCA_REPLY_BE 0
#define HAVE_DECL_IBV_EXP_CQ_IGNORE_OVERRUN 0
#define HAVE_DECL_IBV_EXP_CQ_MODERATION 0
#define HAVE_DECL_IBV_EXP_CREATE_QP 0
#define HAVE_DECL_IBV_EXP_CREATE_SRQ 0
#define HAVE_DECL_IBV_EXP_DCT_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_PCI_ATOMIC_CAPS 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_RESERVED_2 0
#define HAVE_DECL_IBV_EXP_DEVICE_DC_TRANSPORT 0
#define HAVE_DECL_IBV_EXP_DEVICE_MR_ALLOCATE 0
#define HAVE_DECL_IBV_EXP_MR_FIXED_BUFFER_SIZE 0
#define HAVE_DECL_IBV_EXP_MR_INDIRECT_KLMS 0
#define HAVE_DECL_IBV_EXP_ODP_SUPPORT_IMPLICIT 0
#define HAVE_DECL_IBV_EXP_POST_SEND 0
#define HAVE_DECL_IBV_EXP_PREFETCH_MR 0
#define HAVE_DECL_IBV_EXP_PREFETCH_WRITE_ACCESS 0
#define HAVE_DECL_IBV_EXP_QPT_DC_INI 0
#define HAVE_DECL_IBV_EXP_QP_CREATE_UMR 0
#define HAVE_DECL_IBV_EXP_QP_INIT_ATTR_ATOMICS_ARG 0
#define HAVE_DECL_IBV_EXP_QP_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_QUERY_DEVICE 0
#define HAVE_DECL_IBV_EXP_QUERY_GID_ATTR 0
#define HAVE_DECL_IBV_EXP_REG_MR  0
#define HAVE_DECL_IBV_EXP_SEND_EXT_ATOMIC_INLINE 0
#define HAVE_DECL_IBV_EXP_SETENV  0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_CMP_AND_SWP 0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_FETCH_AND_ADD 0
#define HAVE_DECL_IBV_EXP_WR_NOP  0
#define HAVE_DECL_IBV_GET_ASYNC_EVENT 1
#define HAVE_DECL_IBV_GET_DEVICE_NAME 1
#define HAVE_DECL_IBV_LINK_LAYER_ETHERNET 1
#define HAVE_DECL_IBV_LINK_LAYER_INFINIBAND 1
#define HAVE_DECL_IBV_ODP_SUPPORT_IMPLICIT 0
#define HAVE_DECL_IBV_QPF_GRH_REQUIRED 1
#define HAVE_DECL_IBV_QUERY_DEVICE_EX 1
#define HAVE_DECL_IBV_QUERY_GID   1
#define HAVE_DECL_IBV_WC_STATUS_STR 1
#define HAVE_DECL_INOTIFY_ADD_WATCH 1
#define HAVE_DECL_INOTIFY_INIT    1
#define HAVE_DECL_IN_ATTRIB       1
#define HAVE_DECL_IPPROTO_TCP     1
#define HAVE_DECL_MADV_FREE       1
#define HAVE_DECL_MADV_REMOVE     1
#define HAVE_DECL_MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE 1
#define HAVE_DECL_MLX5DV_CREATE_QP 1
#define HAVE_DECL_MLX5DV_DCTYPE_DCT 1
#define HAVE_DECL_MLX5DV_DEVX_SUBSCRIBE_DEVX_EVENT 0
#define HAVE_DECL_MLX5DV_INIT_OBJ 1
#define HAVE_DECL_MLX5DV_IS_SUPPORTED 1
#define HAVE_DECL_MLX5DV_OBJ_AH   1
#define HAVE_DECL_MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE 1
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_BF 0
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_NC 0
#define HAVE_DECL_POSIX_MADV_DONTNEED 1
#define HAVE_DECL_PR_SET_PTRACER  1
#define HAVE_DECL_SOL_SOCKET      1
#define HAVE_DECL_SO_KEEPALIVE    1
#define HAVE_DECL_SPEED_UNKNOWN   1
#define HAVE_DECL_STRERROR_R      1
#define HAVE_DECL_SYS_BRK         1
#define HAVE_DECL_SYS_IPC         0
#define HAVE_DECL_SYS_MADVISE     1
#define HAVE_DECL_SYS_MMAP        1
#define HAVE_DECL_SYS_MREMAP      1
#define HAVE_DECL_SYS_MUNMAP      1
#define HAVE_DECL_SYS_SHMAT       1
#define HAVE_DECL_SYS_SHMDT       1
#define HAVE_DECL_TCP_KEEPCNT     1
#define HAVE_DECL_TCP_KEEPIDLE    1
#define HAVE_DECL_TCP_KEEPINTVL   1
#define HAVE_DECL___PPC_GET_TIMEBASE_FREQ 0
#define HAVE_DETAILED_BACKTRACE   1
#define HAVE_DEVX                 1
#define HAVE_DLFCN_H              1
#define HAVE_HW_TIMER             1
#define HAVE_IB                   1
#define HAVE_IBV_DM               1
#define HAVE_IN6_ADDR_S6_ADDR32   1
#define HAVE_INFINIBAND_MLX5DV_H  1
#define HAVE_INFINIBAND_TM_TYPES_H 1
#define HAVE_INOTIFY              1
#define HAVE_INTTYPES_H           1
#define HAVE_IP_IP_DST            1
#define HAVE_JNI_H                1
#define HAVE_JNI_MD_H             1
#define HAVE_LIBGEN_H             1
#define HAVE_LIBRT                1
#define HAVE_LINUX_FUTEX_H        1
#define HAVE_LINUX_IP_H           1
#define HAVE_LINUX_MMAN_H         1
#define HAVE_MALLOC_H             1
#define HAVE_MALLOC_HOOK          1
#define HAVE_MALLOC_TRIM          1
#define HAVE_MEMALIGN             1
#define HAVE_MEMORY_H             1
#define HAVE_MLX5_HW              1
#define HAVE_MLX5_HW_UD           1
#define HAVE_MREMAP               1
#define HAVE_NETINET_IP_H         1
#define HAVE_NET_ETHERNET_H       1
#define HAVE_NUMA                 1
#define HAVE_NUMAIF_H             1
#define HAVE_NUMA_H               1
#define HAVE_ODP                  1
#define HAVE_POSIX_MEMALIGN       1
#define HAVE_PREFETCH             1
#define HAVE_PROFILING            1
#define HAVE_SCHED_GETAFFINITY    1
#define HAVE_SCHED_SETAFFINITY    1
#define HAVE_SIGACTION_SA_RESTORER 1
#define HAVE_SIGEVENT_SIGEV_UN_TID 1
#define HAVE_SIGHANDLER_T         1
#define HAVE_STDINT_H             1
#define HAVE_STDLIB_H             1
#define HAVE_STRERROR_R           1
#define HAVE_STRINGS_H            1
#define HAVE_STRING_H             1
#define HAVE_STRUCT_BITMASK       1
#define HAVE_STRUCT_DL_PHDR_INFO  1
#define HAVE_STRUCT_IBV_TM_CAPS_FLAGS 1
#define HAVE_STRUCT_MLX5DV_CQ_CQ_UAR 1
#define HAVE_SYS_EPOLL_H          1
#define HAVE_SYS_EVENTFD_H        1
#define HAVE_SYS_STAT_H           1
#define HAVE_SYS_TYPES_H          1
#define HAVE_SYS_UIO_H            1
#define HAVE_TL_DC                1
#define HAVE_TL_RC                1
#define HAVE_TL_UD                1
#define HAVE_UCM_PTMALLOC286      1
#define HAVE_UNISTD_H             1
#define HAVE___CLEAR_CACHE        1
#define HAVE___CURBRK             1
#define HAVE___SIGHANDLER_T       1
#define IBV_HW_TM                 1
#define LT_OBJDIR                 ".libs/"
#define PACKAGE                   "ucx"
#define PACKAGE_BUGREPORT         ""
#define PACKAGE_NAME              "ucx"
#define PACKAGE_STRING            "ucx 1.12"
#define PACKAGE_TARNAME           "ucx"
#define PACKAGE_URL               ""
#define PACKAGE_VERSION           "1.12"
#define STDC_HEADERS              1
#define STRERROR_R_CHAR_P         1
#define UCM_BISTRO_HOOKS          1
#define UCS_MAX_LOG_LEVEL         UCS_LOG_LEVEL_TRACE_POLL
#define UCT_TCP_EP_KEEPALIVE      1
#define UCT_UD_EP_DEBUG_HOOKS     1
#define UCX_CONFIGURE_FLAGS       "--enable-gtest --enable-examples --with-valgrind --enable-profiling --enable-frame-pointer --enable-stats --enable-fault-injection --enable-debug-data --enable-mt --prefix=/cs/labs/amnon/alexam02/ucg/ucx-orig/build"
#define UCX_MODULE_SUBDIR         "ucx"
#define VERSION                   "1.12"
#define restrict                  __restrict
#define test_MODULES              ":module"
#define ucm_MODULES               ""
#define ucs_MODULES               ""
#define uct_MODULES               ":ib:cma"
#define uct_cuda_MODULES          ""
#define uct_ib_MODULES            ""
#define uct_rocm_MODULES          ""
#define ucx_perftest_MODULES      ""
[1629615907.884543] [mlx-stud-01:26236:0]          module.c:251  UCX  DEBUG loading modules for uct
[1629615907.887885] [mlx-stud-01:26236:0]          module.c:180  UCX  TRACE loaded /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/ucx/libuct_ib.so.0.0.0 [0x558730579fa0]
[1629615907.887895] [mlx-stud-01:26236:0]          module.c:186  UCX  TRACE not calling constructor 'ucs_module_global_init' in /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/ucx/libuct_ib.so.0
[1629615907.888751] [mlx-stud-01:26236:0]          module.c:180  UCX  TRACE loaded /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/ucx/libuct_cma.so.0.0.0 [0x55873057c810]
[1629615907.888759] [mlx-stud-01:26236:0]          module.c:186  UCX  TRACE not calling constructor 'ucs_module_global_init' in /cs/labs/amnon/alexam02/ucg/ucx-orig/build/lib/ucx/libuct_cma.so.0
[1629615907.888804] [mlx-stud-01:26236:0]             sys.c:1354 UCX  DEBUG failed to stat(/proc/self/ns/pid): No such file or directory
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 8201860K
#           remote key: 40 bytes
#           rkey_ptr is supported
#
#      Transport: posix
#         Device: memory
#  System device: <unknown>
[1629615907.888919] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating mm_recv_fifo: host memory length 8447 flags 0x3e0
[1629615907.888924] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615907.889126] [mlx-stud-01:26236:0]             sys.c:652  UCX  TRACE   detected huge page size: 2097152
[1629615907.889144] [mlx-stud-01:26236:0]        mm_posix.c:531  UCX  DEBUG   allocated posix shared memory at 0x7f23d81e8000 length 12288
[1629615907.889148] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 12288 bytes at 0x7f23d81e8000 using posix
[1629615907.889193] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8288
[1629615907.889203] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating mm_recv_desc: host memory length 4259952 flags 0x3e0
[1629615907.889205] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615907.891300] [mlx-stud-01:26236:0]        mm_posix.c:326  UCX  DEBUG   shared memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=5) failed: Invalid argument
[1629615907.891315] [mlx-stud-01:26236:0]        mm_posix.c:531  UCX  DEBUG   allocated posix shared memory at 0x7f23d70ba000 length 4263936
[1629615907.891319] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 4263936 bytes at 0x7f23d70ba000 using posix
[1629615907.891332] [mlx-stud-01:26236:0]           mpool.c:218  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x7f23d70ba018 of 4263912 bytes with 512 elements
[1629615907.892341] [mlx-stud-01:26236:0]        mm_iface.c:603  UCX  DEBUG created mm iface 0x55873057f880 FIFO id 0xc0000000c000667c va 0x7f23d81e8000 size 12288 (128 x 64 elems)
#
#      capabilities:
[1629615907.892378] [mlx-stud-01:26236:0]             sys.c:1354 UCX  DEBUG failed to stat(/proc/self/ns/ipc): No such file or directory
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
[1629615907.893258] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool mm_recv_desc destroyed
#
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 28 bytes
#           rkey_ptr is supported
#
#      Transport: sysv
#         Device: memory
#  System device: <unknown>
[1629615907.893367] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating mm_recv_fifo: host memory length 8447 flags 0x3e0
[1629615907.893371] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615907.893378] [mlx-stud-01:26236:0]         mm_sysv.c:94   UCX  DEBUG   mm failed to allocate 8447 bytes with hugetlb
[1629615907.893411] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 12288 bytes at 0x7f23d81e8000 using sysv
[1629615907.893440] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8288
[1629615907.893445] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating mm_recv_desc: host memory length 4259952 flags 0x3e0
[1629615907.893448] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615907.893476] [mlx-stud-01:26236:0]         mm_sysv.c:94   UCX  DEBUG   mm failed to allocate 4259952 bytes with hugetlb
[1629615907.893498] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 4263936 bytes at 0x7f23d70ba000 using sysv
[1629615907.893511] [mlx-stud-01:26236:0]           mpool.c:218  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x7f23d70ba018 of 4263912 bytes with 512 elements
[1629615907.895105] [mlx-stud-01:26236:0]        mm_iface.c:603  UCX  DEBUG created mm iface 0x558730580230 FIFO id 0x30032 va 0x7f23d81e8000 size 12288 (128 x 64 elems)
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
[1629615907.895401] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool mm_recv_desc destroyed
#
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 16 bytes
#
#      Transport: self
#         Device: memory0
#  System device: <unknown>
[1629615907.895471] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool self_msg_desc: align 64, maxelems 4294967295, elemsize 8200
[1629615907.895474] [mlx-stud-01:26236:0]            self.c:222  UCX  DEBUG created self iface id 0x251cca833a57c43e send_size 8192
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
[1629615907.895491] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool self_msg_desc destroyed
#
[1629615907.895542] [mlx-stud-01:26236:0]            sock.c:90   UCX  DEBUG ioctl(req=35093, ifr_name=ibp7s0d1) failed: Cannot assign requested address
[1629615907.895602] [mlx-stud-01:26236:0]            sock.c:90   UCX  DEBUG ioctl(req=35093, ifr_name=enp5s0f1) failed: Cannot assign requested address
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 16 bytes
#
[1629615907.896115] [mlx-stud-01:26236:0]            time.c:22   UCX  DEBUG measured arch clock speed: 2670000000.00 Hz
#      Transport: tcp
#         Device: ib0
#  System device: <unknown>
[1629615907.896132] [mlx-stud-01:26236:0]       tcp_iface.c:547  UCX  DEBUG using TCP port range: 0-0
[1629615907.896136] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp: align 64, maxelems 4294967295, elemsize 8205
[1629615907.896138] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp: align 64, maxelems 4294967295, elemsize 131090
[1629615907.898873] [mlx-stud-01:26236:0]           async.c:231  UCX  DEBUG added async handler 0x55873057cfe0 [id=4 ref 1] uct_tcp_iface_connect_handler() to hash
[1629615907.898994] [mlx-stud-01:26236:0]           async.c:509  UCX  DEBUG listening to async event fd 4 events 0x5 mode thread_spinlock
[1629615907.899005] [mlx-stud-01:26236:0]       tcp_iface.c:497  UCX  DEBUG tcp_iface 0x55873057d4b0: listening for connections (fd=4) on 10.164.164.101:54659
#
#      capabilities:
#            bandwidth: 6239.81/ppn + 0.00 MB/sec
#              latency: 5210 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
[1629615907.901740] [mlx-stud-01:26236:0]       tcp_iface.c:772  UCX  DEBUG tcp_iface 0x55873057d4b0: destroying
[1629615907.901746] [mlx-stud-01:26236:0]           async.c:156  UCX  DEBUG removed async handler 0x55873057cfe0 [id=4 ref 1] uct_tcp_iface_connect_handler() from hash
[1629615907.901748] [mlx-stud-01:26236:0]           async.c:562  UCX  DEBUG removing async handler 0x55873057cfe0 [id=4 ref 1] uct_tcp_iface_connect_handler()
[1629615907.901903] [mlx-stud-01:26236:0]           async.c:582  UCX  TRACE waiting for 0x55873057cfe0 [id=4 ref 1] uct_tcp_iface_connect_handler() completion (called=0)
[1629615907.901908] [mlx-stud-01:26236:0]           async.c:171  UCX  DEBUG release async handler 0x55873057cfe0 [id=4 ref 0] uct_tcp_iface_connect_handler()
[1629615907.901913] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp destroyed
[1629615907.901916] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp destroyed
#
#      Transport: tcp
#         Device: eth0
#  System device: <unknown>
[1629615907.901970] [mlx-stud-01:26236:0]       tcp_iface.c:547  UCX  DEBUG using TCP port range: 0-0
[1629615907.901978] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp: align 64, maxelems 4294967295, elemsize 8205
[1629615907.901980] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp: align 64, maxelems 4294967295, elemsize 131090
[1629615907.902015] [mlx-stud-01:26236:0]           async.c:231  UCX  DEBUG added async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() to hash
[1629615907.902070] [mlx-stud-01:26236:0]           async.c:509  UCX  DEBUG listening to async event fd 4 events 0x5 mode thread_spinlock
[1629615907.902074] [mlx-stud-01:26236:0]       tcp_iface.c:497  UCX  DEBUG tcp_iface 0x55873057d4b0: listening for connections (fd=4) on 132.65.164.101:44115
#
#      capabilities:
#            bandwidth: 113.16/ppn + 0.00 MB/sec
#              latency: 5776 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
[1629615907.902151] [mlx-stud-01:26236:0]       tcp_iface.c:772  UCX  DEBUG tcp_iface 0x55873057d4b0: destroying
[1629615907.902155] [mlx-stud-01:26236:0]           async.c:156  UCX  DEBUG removed async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() from hash
[1629615907.902157] [mlx-stud-01:26236:0]           async.c:562  UCX  DEBUG removing async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler()
[1629615907.902258] [mlx-stud-01:26236:0]           async.c:582  UCX  TRACE waiting for 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() completion (called=0)
[1629615907.902263] [mlx-stud-01:26236:0]           async.c:171  UCX  DEBUG release async handler 0x55873064bec0 [id=4 ref 0] uct_tcp_iface_connect_handler()
[1629615907.902266] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp destroyed
[1629615907.902269] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp destroyed
#
#      Transport: tcp
#         Device: lo
#  System device: <unknown>
[1629615907.902318] [mlx-stud-01:26236:0]       tcp_iface.c:547  UCX  DEBUG using TCP port range: 0-0
[1629615907.902324] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp: align 64, maxelems 4294967295, elemsize 8205
[1629615907.902327] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp: align 64, maxelems 4294967295, elemsize 131090
[1629615907.902365] [mlx-stud-01:26236:0]           async.c:231  UCX  DEBUG added async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() to hash
[1629615907.902421] [mlx-stud-01:26236:0]           async.c:509  UCX  DEBUG listening to async event fd 4 events 0x5 mode thread_spinlock
[1629615907.902425] [mlx-stud-01:26236:0]       tcp_iface.c:497  UCX  DEBUG tcp_iface 0x55873057d4b0: listening for connections (fd=4) on 127.0.0.1:46701
#
#      capabilities:
[1629615907.902442] [mlx-stud-01:26236:0]            sock.c:90   UCX  DEBUG ioctl(req=35142, ifr_name=lo) failed: Operation not supported
[1629615907.902461] [mlx-stud-01:26236:0]         tcp_net.c:61   UCX  DEBUG speed of lo is UNKNOWN, assuming 100 Mbps
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
[1629615907.902523] [mlx-stud-01:26236:0]       tcp_iface.c:772  UCX  DEBUG tcp_iface 0x55873057d4b0: destroying
[1629615907.902527] [mlx-stud-01:26236:0]           async.c:156  UCX  DEBUG removed async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() from hash
[1629615907.902529] [mlx-stud-01:26236:0]           async.c:562  UCX  DEBUG removing async handler 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler()
[1629615907.902592] [mlx-stud-01:26236:0]           async.c:582  UCX  TRACE waiting for 0x55873064bec0 [id=4 ref 1] uct_tcp_iface_connect_handler() completion (called=0)
[1629615907.902595] [mlx-stud-01:26236:0]           async.c:171  UCX  DEBUG release async handler 0x55873064bec0 [id=4 ref 0] uct_tcp_iface_connect_handler()
[1629615907.902597] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_rx_buf_mp destroyed
[1629615907.902599] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_tcp_iface_tx_buf_mp destroyed
#
[1629615907.902628] [mlx-stud-01:26236:0]      tcp_sockcm.c:215  UCX  DEBUG created tcp_sockcm 0x558730618df0
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
[1629615907.902682] [mlx-stud-01:26236:0]          module.c:251  UCX  DEBUG loading modules for uct_ib
[1629615907.902951] [mlx-stud-01:26236:0]           ib_md.c:1588 UCX  TRACE opening IB device mlx5_0
[1629615907.917534] [mlx-stud-01:26236:0]    ib_mlx5dv_md.c:630  UCX  DEBUG mlx5dv_open_device(mlx5_0) failed: Bad file descriptor
[1629615907.917539] [mlx-stud-01:26236:0]           ib_md.c:1648 UCX  DEBUG mlx5_0: md open by 'uct_ib_mlx5_devx_md_ops' failed, trying next
[1629615907.929781] [mlx-stud-01:26236:0]       ib_device.c:554  UCX  DEBUG PF: mlx5_0 vendor_id: 0x15b3 device_id: 4113
[1629615907.929788] [mlx-stud-01:26236:0]    ib_mlx5dv_md.c:863  UCX  DEBUG checking for DC support on mlx5_0
[1629615907.937372] [mlx-stud-01:26236:0]    ib_mlx5dv_md.c:937  UCX  DEBUG DC is supported on mlx5_0
[1629615908.026471] [mlx-stud-01:26236:0]           async.c:231  UCX  DEBUG added async handler 0x55873064bf90 [id=4 ref 1] uct_ib_async_event_handler() to hash
[1629615908.026567] [mlx-stud-01:26236:0]           async.c:509  UCX  DEBUG listening to async event fd 4 events 0x1 mode thread_spinlock
[1629615908.026581] [mlx-stud-01:26236:0]       ib_device.c:668  UCX  DEBUG initialized device 'mlx5_0' (InfiniBand channel adapter) with 2 ports
[1629615908.027462] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.027478] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.027502] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rcache_mp: align 8, maxelems 4294967295, elemsize 144
[1629615908.030501] [mlx-stud-01:26236:0]           async.c:231  UCX  DEBUG added async handler 0x5587307970a0 [id=8 ref 1] ucs_rcache_invalidate_handler() to hash
[1629615908.030515] [mlx-stud-01:26236:0]           async.c:509  UCX  DEBUG listening to async event fd 8 events 0x1 mode thread_spinlock
[1629615908.030711] [mlx-stud-01:26236:0]          module.c:251  UCX  DEBUG loading modules for ucm
[1629615908.030717] [mlx-stud-01:26236:0]           ib_md.c:1356 UCX  DEBUG mlx5_0: using registration cache
[1629615908.030849] [mlx-stud-01:26236:0]           ib_md.c:1554 UCX  TRACE mlx5_0: PCIe gen2 16x, effective throughput 6877.201 MB/s 57.690 Gb/s
[1629615908.030852] [mlx-stud-01:26236:0]           ib_md.c:1641 UCX  DEBUG mlx5_0: md open by 'uct_ib_mlx5_md_ops' is successful
[1629615908.043460] [mlx-stud-01:26236:0]            topo.c:100  UCX  DEBUG bus id 0x70000 doesn't exist. sys_dev = 0
[1629615908.043465] [mlx-stud-01:26236:0]       ib_device.c:1137 UCX  DEBUG mlx5_0 bus id 0:7:0.0 sys_dev 0
[1629615908.043468] [mlx-stud-01:26236:0]       ib_device.c:768  UCX  TRACE mlx5_0:2 is not active (state: 1)
[1629615908.043472] [mlx-stud-01:26236:0]       ib_device.c:1168 UCX  TRACE mlx5_0:2 does not support flags 0x0: Destination is unreachable
[1629615908.043562] [mlx-stud-01:26236:0]            topo.c:92   UCX  DEBUG bus id 0x70000 exists. sys_dev = 0
[1629615908.043566] [mlx-stud-01:26236:0]       ib_device.c:1137 UCX  DEBUG mlx5_0 bus id 0:7:0.0 sys_dev 0
[1629615908.043568] [mlx-stud-01:26236:0]       ib_device.c:768  UCX  TRACE mlx5_0:2 is not active (state: 1)
[1629615908.043570] [mlx-stud-01:26236:0]       ib_device.c:1168 UCX  TRACE mlx5_0:2 does not support flags 0x4: Destination is unreachable
[1629615908.043668] [mlx-stud-01:26236:0]            topo.c:92   UCX  DEBUG bus id 0x70000 exists. sys_dev = 0
[1629615908.043672] [mlx-stud-01:26236:0]       ib_device.c:1137 UCX  DEBUG mlx5_0 bus id 0:7:0.0 sys_dev 0
[1629615908.043674] [mlx-stud-01:26236:0]       ib_device.c:768  UCX  TRACE mlx5_0:2 is not active (state: 1)
[1629615908.043677] [mlx-stud-01:26236:0]       ib_device.c:1168 UCX  TRACE mlx5_0:2 does not support flags 0xc4: Destination is unreachable
[1629615908.043762] [mlx-stud-01:26236:0]            topo.c:92   UCX  DEBUG bus id 0x70000 exists. sys_dev = 0
[1629615908.043766] [mlx-stud-01:26236:0]       ib_device.c:1137 UCX  DEBUG mlx5_0 bus id 0:7:0.0 sys_dev 0
[1629615908.043768] [mlx-stud-01:26236:0]       ib_device.c:768  UCX  TRACE mlx5_0:2 is not active (state: 1)
[1629615908.043771] [mlx-stud-01:26236:0]       ib_device.c:1168 UCX  TRACE mlx5_0:2 does not support flags 0x0: Destination is unreachable
[1629615908.043848] [mlx-stud-01:26236:0]            topo.c:92   UCX  DEBUG bus id 0x70000 exists. sys_dev = 0
[1629615908.043851] [mlx-stud-01:26236:0]       ib_device.c:1137 UCX  DEBUG mlx5_0 bus id 0:7:0.0 sys_dev 0
[1629615908.043853] [mlx-stud-01:26236:0]       ib_device.c:768  UCX  TRACE mlx5_0:2 is not active (state: 1)
[1629615908.043856] [mlx-stud-01:26236:0]       ib_device.c:1168 UCX  TRACE mlx5_0:2 does not support flags 0x4: Destination is unreachable
[1629615908.043868] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.043876] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#
# Memory domain: mlx5_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 24 bytes
#           local memory handle is required for zcopy
#
[1629615908.043903] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.043910] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#      Transport: rc_verbs
#         Device: mlx5_0:1
#  System device: 0000:07:00.0 (0)
[1629615908.044005] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.044013] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.044485] [mlx-stud-01:26236:0]        ib_iface.c:858  UCX  DEBUG using pkey[0] 0xffff on mlx5_0:1
[1629615908.046011] [mlx-stud-01:26236:0]        ib_iface.c:1471 UCX  DEBUG created uct_ib_iface_t headroom_ofs 12 payload_ofs 12 hdr_ofs 11 data_sz 8256
[1629615908.046051] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_recv_desc: align 64, maxelems 4294967295, elemsize 8276
[1629615908.046078] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_send_desc: align 64, maxelems 4294967295, elemsize 8328
[1629615908.046118] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool send-ops-mpool: align 64, maxelems 4294967295, elemsize 56
[1629615908.048492] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool pending-ops: align 1, maxelems 4294967295, elemsize 64
[1629615908.048501] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_verbs_short_desc: align 64, maxelems 4294967295, elemsize 200
[1629615908.053088] [mlx-stud-01:26236:0]        ib_iface.c:1001 UCX  DEBUG iface=0x558730592ed0: created RC QP 0x3370 on mlx5_0:1 TX wr:409 sge:5 inl:124 resp:64 RX wr:0 sge:0 resp:64
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 700 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 127
#           connection: to ep
#      device priority: 20
#     device num paths: 1
#              max eps: 256
#       device address: 3 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
[1629615908.057087] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_verbs_short_desc destroyed
[1629615908.058710] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool send-ops-mpool destroyed
[1629615908.058715] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_send_desc destroyed
[1629615908.058718] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_recv_desc destroyed
[1629615908.058720] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool pending-ops destroyed
#
#
[1629615908.059759] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.059770] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#      Transport: rc_mlx5
#         Device: mlx5_0:1
#  System device: 0000:07:00.0 (0)
[1629615908.059880] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.059889] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.060174] [mlx-stud-01:26236:0]        ib_iface.c:858  UCX  DEBUG using pkey[0] 0xffff on mlx5_0:1
[1629615908.060252] [mlx-stud-01:26236:0]       ib_device.c:1405 UCX  DEBUG max IB CQE size is 128
[1629615908.062659] [mlx-stud-01:26236:0]        ib_iface.c:1471 UCX  DEBUG created uct_ib_iface_t headroom_ofs 12 payload_ofs 12 hdr_ofs 10 data_sz 8256
[1629615908.062670] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_recv_desc: align 64, maxelems 4294967295, elemsize 8276
[1629615908.062674] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_send_desc: align 64, maxelems 4294967295, elemsize 8328
[1629615908.062717] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool send-ops-mpool: align 64, maxelems 4294967295, elemsize 56
[1629615908.064902] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool pending-ops: align 1, maxelems 4294967295, elemsize 64
[1629615908.064920] [mlx-stud-01:26236:0]         ib_mlx5.c:898  UCX  DEBUG SL=0 (AR support - unknown) was selected on mlx5_0:1, SLs with AR support = { <none> }, SLs without AR support = { <none> }
[1629615908.065015] [mlx-stud-01:26236:0]  rc_mlx5_common.c:737  UCX  DEBUG ibv_alloc_dm(dev=mlx5_0 length=2048) failed: Invalid argument
[1629615908.065020] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_mlx5_atomic_desc: align 64, maxelems 4294967295, elemsize 80
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 700 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 220
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 234
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 186
#           connection: to ep
#      device priority: 20
#     device num paths: 1
#              max eps: 256
#       device address: 3 bytes
#           ep address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
[1629615908.065067] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_mlx5_atomic_desc destroyed
[1629615908.066577] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool send-ops-mpool destroyed
[1629615908.066581] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_send_desc destroyed
[1629615908.066584] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_recv_desc destroyed
[1629615908.066586] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool pending-ops destroyed
#
#
[1629615908.067940] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.067951] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#      Transport: dc_mlx5
#         Device: mlx5_0:1
#  System device: 0000:07:00.0 (0)
[1629615908.068093] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.068102] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.068407] [mlx-stud-01:26236:0]        ib_iface.c:858  UCX  DEBUG using pkey[0] 0xffff on mlx5_0:1
[1629615908.070270] [mlx-stud-01:26236:0]        ib_iface.c:1471 UCX  DEBUG created uct_ib_iface_t headroom_ofs 12 payload_ofs 12 hdr_ofs 10 data_sz 8256
[1629615908.070281] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_recv_desc: align 64, maxelems 4294967295, elemsize 8276
[1629615908.070285] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_send_desc: align 64, maxelems 4294967295, elemsize 8328
[1629615908.070333] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool send-ops-mpool: align 64, maxelems 4294967295, elemsize 56
[1629615908.072564] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool pending-ops: align 1, maxelems 4294967295, elemsize 112
[1629615908.072572] [mlx-stud-01:26236:0]         ib_mlx5.c:898  UCX  DEBUG SL=0 (AR support - unknown) was selected on mlx5_0:1, SLs with AR support = { <none> }, SLs without AR support = { <none> }
[1629615908.072719] [mlx-stud-01:26236:0]  rc_mlx5_common.c:737  UCX  DEBUG ibv_alloc_dm(dev=mlx5_0 length=2048) failed: Invalid argument
[1629615908.072724] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool rc_mlx5_atomic_desc: align 64, maxelems 4294967295, elemsize 80
[1629615908.072742] [mlx-stud-01:26236:0]         dc_mlx5.c:503  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument
[1629615908.072764] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_mlx5_atomic_desc destroyed
[1629615908.074284] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool send-ops-mpool destroyed
[1629615908.074288] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_send_desc destroyed
[1629615908.074290] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rc_recv_desc destroyed
[1629615908.074292] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool pending-ops destroyed
#   < failed to open interface >
#
[1629615908.075630] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.075639] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#      Transport: ud_verbs
#         Device: mlx5_0:1
#  System device: 0000:07:00.0 (0)
[1629615908.075714] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.075721] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.076027] [mlx-stud-01:26236:0]        ib_iface.c:858  UCX  DEBUG using pkey[0] 0xffff on mlx5_0:1
[1629615908.077406] [mlx-stud-01:26236:0]        ib_iface.c:1471 UCX  DEBUG created uct_ib_iface_t headroom_ofs 88 payload_ofs 88 hdr_ofs 40 data_sz 4096
[1629615908.082621] [mlx-stud-01:26236:0]        ib_iface.c:1001 UCX  DEBUG iface=0x558730619880: created UD QP 0x3373 on mlx5_0:1 TX wr:341 sge:6 inl:124 resp:0 RX wr:4096 sge:1 resp:0
[1629615908.084415] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool ud_recv_skb: align 64, maxelems 4294967295, elemsize 4192
[1629615908.084424] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating ud_recv_skb: host memory length 540784 flags 0x3e0
[1629615908.084427] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method huge
[1629615908.084433] [mlx-stud-01:26236:0]         uct_mem.c:283  UCX  TRACE   failed to allocate 540784 bytes from hugetlb: User-defined limit was reached
[1629615908.084435] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method thp
[1629615908.084468] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615908.084485] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG   mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.084494] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG   mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.084500] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method mmap
[1629615908.084512] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 544768 bytes at 0x7f23d6c45000 using mmap
[1629615908.084521] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.084529] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.084561] [mlx-stud-01:26236:0]           mpool.c:218  UCX  DEBUG mpool rcache_mp: allocated chunk 0x7f23d79cf008 of 151544 bytes with 1052 elements
[1629615908.084974] [mlx-stud-01:26236:0]           ib_md.c:577  UCX  TRACE ibv_reg_mr(0x55873057cef0, 0x7f23d6c45000, 544768) took 0.256 msec
[1629615908.084980] [mlx-stud-01:26236:0]           ib_md.c:817  UCX  DEBUG registered memory 0x7f23d6c45000..0x7f23d6cca000 on mlx5_0 lkey 0x922b93 rkey 0x922b93 access 0xf flags 0x3e4
[1629615908.084992] [mlx-stud-01:26236:0]          rcache.c:945  UCX  TRACE mlx5_0: created region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.084997] [mlx-stud-01:26236:0]           mpool.c:218  UCX  DEBUG mpool ud_recv_skb: allocated chunk 0x7f23d6c45018 of 544744 bytes with 128 elements
[1629615908.085006] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool ud_tx_skb: align 64, maxelems 4294967295, elemsize 4168
[1629615908.085073] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80::f452:1403:18:8470 to hash on device mlx5_0 port 1 index 0)
[1629615908.085125] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 1)
[1629615908.085169] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 2)
[1629615908.085212] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 3)
[1629615908.085254] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 4)
[1629615908.085297] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 5)
[1629615908.085339] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 6)
[1629615908.085382] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x558730619880: adding gid fe80:: to hash on device mlx5_0 port 1 index 7)
[1629615908.085876] [mlx-stud-01:26236:0]     timer_wheel.c:41   UCX  DEBUG high res timer created log=23 resolution=3141.800749 usec wanted: 2500.000000 usec
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 730 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 3952
#           connection: to ep, to iface
#      device priority: 20
#     device num paths: 1
#              max eps: inf
#       device address: 3 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
[1629615908.085928] [mlx-stud-01:26236:0]        ud_iface.c:615  UCX  DEBUG iface(0x558730619880): cep cleanup
[1629615908.085931] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool ud_tx_skb destroyed
[1629615908.085938] [mlx-stud-01:26236:0]          rcache.c:331  UCX  TRACE mlx5_0: lru add region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.085943] [mlx-stud-01:26236:0]          rcache.c:419  UCX  TRACE mlx5_0: put region, flags 0x1 region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.085953] [mlx-stud-01:26236:0]          rcache.c:456  UCX  TRACE mlx5_0: invalidate region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 1 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.085962] [mlx-stud-01:26236:0]          rcache.c:419  UCX  TRACE mlx5_0: put region, flags 0xa region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 1 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.085966] [mlx-stud-01:26236:0]          rcache.c:430  UCX  TRACE mlx5_0: put on GC list region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.085992] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool ud_recv_skb destroyed
[1629615908.092124] [mlx-stud-01:26236:0]        ud_iface.c:622  UCX  DEBUG iface(0x558730619880): ptr_array cleanup
#
#
[1629615908.093127] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.093137] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
#      Transport: ud_mlx5
#         Device: mlx5_0:1
#  System device: 0000:07:00.0 (0)
[1629615908.093210] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.093218] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.093559] [mlx-stud-01:26236:0]        ib_iface.c:858  UCX  DEBUG using pkey[0] 0xffff on mlx5_0:1
[1629615908.094946] [mlx-stud-01:26236:0]        ib_iface.c:1471 UCX  DEBUG created uct_ib_iface_t headroom_ofs 88 payload_ofs 88 hdr_ofs 40 data_sz 4096
[1629615908.100203] [mlx-stud-01:26236:0]        ib_iface.c:1001 UCX  DEBUG iface=0x5587306175d0: created UD QP 0x3374 on mlx5_0:1 TX wr:341 sge:6 inl:124 resp:0 RX wr:4096 sge:1 resp:0
[1629615908.100213] [mlx-stud-01:26236:0]         ib_mlx5.c:577  UCX  DEBUG tx wq 65536 bytes [bb=64, nwqe=1024] mmio_mode bf_post
[1629615908.101347] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool ud_recv_skb: align 64, maxelems 4294967295, elemsize 4192
[1629615908.101352] [mlx-stud-01:26236:0]         uct_mem.c:106  UCX  TRACE allocating ud_recv_skb: host memory length 540784 flags 0x3e0
[1629615908.101355] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method huge
[1629615908.101359] [mlx-stud-01:26236:0]         uct_mem.c:283  UCX  TRACE   failed to allocate 540784 bytes from hugetlb: User-defined limit was reached
[1629615908.101362] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method thp
[1629615908.101387] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method md
[1629615908.101405] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG   mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.101414] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG   mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.101419] [mlx-stud-01:26236:0]         uct_mem.c:110  UCX  TRACE   trying allocation method mmap
[1629615908.101429] [mlx-stud-01:26236:0]         uct_mem.c:304  UCX  TRACE   allocated 544768 bytes at 0x7f23d6c45000 using mmap
[1629615908.101439] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: cuda GPUDirect RDMA is disabled
[1629615908.101446] [mlx-stud-01:26236:0]           ib_md.c:300  UCX  DEBUG mlx5_0: rocm GPUDirect RDMA is disabled
[1629615908.101455] [mlx-stud-01:26236:0]          rcache.c:375  UCX  TRACE mlx5_0: destroy region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.101565] [mlx-stud-01:26236:0]          rcache.c:345  UCX  TRACE mlx5_0: lru remove region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922b93 rkey 0x922b93 atomic_rkey 0xffffffff
[1629615908.101786] [mlx-stud-01:26236:0]           ib_md.c:577  UCX  TRACE ibv_reg_mr(0x55873057cef0, 0x7f23d6c45000, 544768) took 0.208 msec
[1629615908.101792] [mlx-stud-01:26236:0]           ib_md.c:817  UCX  DEBUG registered memory 0x7f23d6c45000..0x7f23d6cca000 on mlx5_0 lkey 0x922a90 rkey 0x922a90 access 0xf flags 0x3e4
[1629615908.101798] [mlx-stud-01:26236:0]          rcache.c:945  UCX  TRACE mlx5_0: created region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.101802] [mlx-stud-01:26236:0]           mpool.c:218  UCX  DEBUG mpool ud_recv_skb: allocated chunk 0x7f23d6c45018 of 544744 bytes with 128 elements
[1629615908.101810] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool ud_tx_skb: align 64, maxelems 4294967295, elemsize 4168
[1629615908.101871] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80::f452:1403:18:8470 to hash on device mlx5_0 port 1 index 0)
[1629615908.101916] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 1)
[1629615908.101958] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 2)
[1629615908.102000] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 3)
[1629615908.102042] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 4)
[1629615908.102083] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 5)
[1629615908.102125] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 6)
[1629615908.102167] [mlx-stud-01:26236:0]        ud_iface.c:398  UCX  DEBUG iface 0x5587306175d0: adding gid fe80:: to hash on device mlx5_0 port 1 index 7)
[1629615908.102173] [mlx-stud-01:26236:0]         ib_mlx5.c:898  UCX  DEBUG SL=0 (AR support - unknown) was selected on mlx5_0:1, SLs with AR support = { <none> }, SLs without AR support = { <none> }
[1629615908.102255] [mlx-stud-01:26236:0]     timer_wheel.c:41   UCX  DEBUG high res timer created log=23 resolution=3141.800749 usec wanted: 2500.000000 usec
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 730 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 20
#     device num paths: 1
#              max eps: inf
#       device address: 3 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
[1629615908.102287] [mlx-stud-01:26236:0]        ud_iface.c:615  UCX  DEBUG iface(0x5587306175d0): cep cleanup
[1629615908.102290] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool ud_tx_skb destroyed
[1629615908.102295] [mlx-stud-01:26236:0]          rcache.c:331  UCX  TRACE mlx5_0: lru add region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.102300] [mlx-stud-01:26236:0]          rcache.c:419  UCX  TRACE mlx5_0: put region, flags 0x1 region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 2 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.102307] [mlx-stud-01:26236:0]          rcache.c:456  UCX  TRACE mlx5_0: invalidate region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] gt rw ref 1 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.102313] [mlx-stud-01:26236:0]          rcache.c:419  UCX  TRACE mlx5_0: put region, flags 0xa region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 1 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.102317] [mlx-stud-01:26236:0]          rcache.c:430  UCX  TRACE mlx5_0: put on GC list region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.102340] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool ud_recv_skb destroyed
[1629615908.107487] [mlx-stud-01:26236:0]        ud_iface.c:622  UCX  DEBUG iface(0x5587306175d0): ptr_array cleanup
#
[1629615908.108488] [mlx-stud-01:26236:0]           async.c:156  UCX  DEBUG removed async handler 0x5587307970a0 [id=8 ref 1] ucs_rcache_invalidate_handler() from hash
[1629615908.108493] [mlx-stud-01:26236:0]           async.c:562  UCX  DEBUG removing async handler 0x5587307970a0 [id=8 ref 1] ucs_rcache_invalidate_handler()
[1629615908.108505] [mlx-stud-01:26236:0]           async.c:582  UCX  TRACE waiting for 0x5587307970a0 [id=8 ref 1] ucs_rcache_invalidate_handler() completion (called=0)
[1629615908.108508] [mlx-stud-01:26236:0]           async.c:171  UCX  DEBUG release async handler 0x5587307970a0 [id=8 ref 0] ucs_rcache_invalidate_handler()
[1629615908.108520] [mlx-stud-01:26236:0]          rcache.c:375  UCX  TRACE mlx5_0: destroy region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.108615] [mlx-stud-01:26236:0]          rcache.c:345  UCX  TRACE mlx5_0: lru remove region 0x558730712010 [0x7f23d6c45000..0x7f23d6cca000] g- rw ref 0 lkey 0x922a90 rkey 0x922a90 atomic_rkey 0xffffffff
[1629615908.108709] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool rcache_mp destroyed
[1629615908.109097] [mlx-stud-01:26236:0]       ib_device.c:686  UCX  DEBUG destroying ib device mlx5_0
[1629615908.109129] [mlx-stud-01:26236:0]           async.c:156  UCX  DEBUG removed async handler 0x55873064bf90 [id=4 ref 1] uct_ib_async_event_handler() from hash
[1629615908.109133] [mlx-stud-01:26236:0]           async.c:562  UCX  DEBUG removing async handler 0x55873064bf90 [id=4 ref 1] uct_ib_async_event_handler()
[1629615908.109293] [mlx-stud-01:26236:0]           async.c:582  UCX  TRACE waiting for 0x55873064bf90 [id=4 ref 1] uct_ib_async_event_handler() completion (called=0)
[1629615908.109298] [mlx-stud-01:26236:0]           async.c:171  UCX  DEBUG release async handler 0x55873064bf90 [id=4 ref 0] uct_ib_async_event_handler()
[1629615908.111936] [mlx-stud-01:26236:0]          cma_md.c:46   UCX  DEBUG could not read '/proc/sys/kernel/yama/ptrace_scope' - assuming Yama security is not enforced
[1629615908.111963] [mlx-stud-01:26236:0]          cma_md.c:46   UCX  DEBUG could not read '/proc/sys/kernel/yama/ptrace_scope' - assuming Yama security is not enforced
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#
#      Transport: cma
#         Device: memory
#  System device: <unknown>
[1629615908.112023] [mlx-stud-01:26236:0]           mpool.c:101  UCX  DEBUG mpool uct_scopy_iface_tx_mp: align 64, maxelems 4294967295, elemsize 736
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 400 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check
[1629615908.112047] [mlx-stud-01:26236:0]           mpool.c:155  UCX  DEBUG mpool uct_scopy_iface_tx_mp destroyed
#

alex--m · 2021-08-22T07:09:26Z

Also, this ibstat output may be helpful:

CA 'mlx5_0'
	CA type: MT4113
	Number of ports: 2
	Firmware version: 10.16.1020
	Hardware version: 0
	Node GUID: 0xf452140300188470
	System image GUID: 0xf452140300188470
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 13
		LMC: 0
		SM lid: 33
		Capability mask: 0x26516848
		Port GUID: 0xf452140300188470
		Link layer: InfiniBand
	Port 2:
		State: Down
		Physical state: Disabled
		Rate: 10
		Base lid: 65535
		LMC: 0
		SM lid: 0
		Capability mask: 0x26516848
		Port GUID: 0xf452140300188478
		Link layer: InfiniBand

yosefe · 2021-08-22T08:57:13Z

@alex--m i see there is a COnnect-IB card (which supports DC) and DC iface create does not have any error

alex--m · 2021-08-22T10:35:10Z

@yosefe this looks problematic (and also OMPI crashes during MPI_Init() on worker creation following a similar DCT failure):

[1629615908.072742] [mlx-stud-01:26236:0]         dc_mlx5.c:503  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument

yosefe · 2021-08-22T11:27:33Z

@alex--m seems it could be issue with scatter-to-cqe initialization on DC
What is the version of rdma-core and kernel / MLNX_OFED in the machine?

alex--m · 2021-08-22T11:55:24Z

Kernel 5.4.81, no MOFED, rdma-core version unclear (no privs to check, ofed_info not installed) - any Ideas how to query?
I got this from ibstat, if it helps: /sbin/ibstat BUILD VERSION: 2.1.0 Build date: Dec 14 2018 07:54:41

P.S. I can help you or a member of your team to connect to those servers, if it helps.

yosefe · 2021-08-22T12:07:07Z

Kernel 5.4.81, no MOFED, rdma-core version unclear (no privs to check, ofed_info not installed) - any Ideas how to query?

rpm -q rdma-core
rpm -q libibverbs
rpm -q libmlx5

P.S. I can help you or a member of your team to connect to those servers, if it helps.

Yes can u pls send access info by mail?

alex--m · 2021-08-22T12:14:16Z

I don't have access to 'rpm' executable (it's a netboot image). I'll send info shortly.

zerothi · 2022-01-05T13:02:02Z

I have the same problems, let me know if further details are required.

UCX version: 1.11.2

Error:

[1641382252.649812] [n-62-31-9:154364:0]         dc_mlx5.c:505  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument
[n-62-31-9:154364] ../../../../../ompi/mca/pml/ucx/pml_ucx.c:309  Error: Failed to create UCP worker

Output of lsb_release -a

LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	Scientific
Description:	Scientific Linux release 7.7 (Nitrogen)
Release:	7.7
Codename:	Nitrogen

Output of `ibstat`

CA 'mlx5_0'
	CA type: MT4115
	Number of ports: 1
	Firmware version: 12.23.1020
	Hardware version: 0
	Node GUID: 0x98039b030074c28c
	System image GUID: 0x98039b030074c28c
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 464
		LMC: 0
		SM lid: 271
		Capability mask: 0x2659e848
		Port GUID: 0x98039b030074c28c
		Link layer: InfiniBand

Output of `ucx_info -dvb`

# UCT version=1.11.2 revision ef2bbcf
# configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --with-verbs --with-mlx5-dv --with-ib-hw-tm --with-rc --with-ud --with-dc --with-dm --with-mcpu --with-march --disable-backtrace-detail --enable-devel-headers --enable-optimizations --enable-shared --disable-static --with-knem=/dtu/sw/dcc/SL73/2021-nov/XeonGold6126/gnu/11.2.0/knem/1.1.4 --prefix=/dtu/sw/dcc/SL73/2021-nov/XeonGold6126/gnu/11.2.0/ucx/1.11.2
#define UCX_CONFIG_H              
#define ENABLE_BUILTIN_MEMCPY     1
#define ENABLE_DEBUG_DATA         0
#define ENABLE_MT                 0
#define ENABLE_PARAMS_CHECK       0
#define HAVE_ALLOCA               1
#define HAVE_ALLOCA_H             1
#define HAVE_ATTRIBUTE_NOOPTIMIZE 1
#define HAVE_CLEARENV             1
#define HAVE_CPU_SET_T            1
#define HAVE_DC_DV                1
#define HAVE_DECL_ASPRINTF        1
#define HAVE_DECL_BASENAME        1
#define HAVE_DECL_CPU_ISSET       1
#define HAVE_DECL_CPU_ZERO        1
#define HAVE_DECL_ETHTOOL_CMD_SPEED 1
#define HAVE_DECL_FMEMOPEN        1
#define HAVE_DECL_FUSE_MOUNT      0
#define HAVE_DECL_FUSE_OPEN_CHANNEL 0
#define HAVE_DECL_FUSE_UNMOUNT    0
#define HAVE_DECL_F_SETOWN_EX     1
#define HAVE_DECL_IBV_ACCESS_ON_DEMAND 1
#define HAVE_DECL_IBV_ACCESS_RELAXED_ORDERING 0
#define HAVE_DECL_IBV_ADVISE_MR   1
#define HAVE_DECL_IBV_ALLOC_DM    1
#define HAVE_DECL_IBV_ALLOC_TD    1
#define HAVE_DECL_IBV_CMD_MODIFY_QP 0
#define HAVE_DECL_IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN 1
#define HAVE_DECL_IBV_CREATE_QP_EX 1
#define HAVE_DECL_IBV_CREATE_SRQ  1
#define HAVE_DECL_IBV_CREATE_SRQ_EX 1
#define HAVE_DECL_IBV_EVENT_GID_CHANGE 1
#define HAVE_DECL_IBV_EVENT_TYPE_STR 1
#define HAVE_DECL_IBV_EXP_ACCESS_ALLOCATE_MR 0
#define HAVE_DECL_IBV_EXP_ACCESS_ON_DEMAND 0
#define HAVE_DECL_IBV_EXP_ALLOC_DM 0
#define HAVE_DECL_IBV_EXP_ATOMIC_HCA_REPLY_BE 0
#define HAVE_DECL_IBV_EXP_CQ_IGNORE_OVERRUN 0
#define HAVE_DECL_IBV_EXP_CQ_MODERATION 0
#define HAVE_DECL_IBV_EXP_CREATE_QP 0
#define HAVE_DECL_IBV_EXP_CREATE_SRQ 0
#define HAVE_DECL_IBV_EXP_DCT_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_PCI_ATOMIC_CAPS 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_RESERVED_2 0
#define HAVE_DECL_IBV_EXP_DEVICE_DC_TRANSPORT 0
#define HAVE_DECL_IBV_EXP_DEVICE_MR_ALLOCATE 0
#define HAVE_DECL_IBV_EXP_MR_FIXED_BUFFER_SIZE 0
#define HAVE_DECL_IBV_EXP_MR_INDIRECT_KLMS 0
#define HAVE_DECL_IBV_EXP_ODP_SUPPORT_IMPLICIT 0
#define HAVE_DECL_IBV_EXP_POST_SEND 0
#define HAVE_DECL_IBV_EXP_PREFETCH_MR 0
#define HAVE_DECL_IBV_EXP_PREFETCH_WRITE_ACCESS 0
#define HAVE_DECL_IBV_EXP_QPT_DC_INI 0
#define HAVE_DECL_IBV_EXP_QP_CREATE_UMR 0
#define HAVE_DECL_IBV_EXP_QP_INIT_ATTR_ATOMICS_ARG 0
#define HAVE_DECL_IBV_EXP_QP_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_QUERY_DEVICE 0
#define HAVE_DECL_IBV_EXP_QUERY_GID_ATTR 0
#define HAVE_DECL_IBV_EXP_REG_MR  0
#define HAVE_DECL_IBV_EXP_SEND_EXT_ATOMIC_INLINE 0
#define HAVE_DECL_IBV_EXP_SETENV  0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_CMP_AND_SWP 0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_FETCH_AND_ADD 0
#define HAVE_DECL_IBV_EXP_WR_NOP  0
#define HAVE_DECL_IBV_GET_ASYNC_EVENT 1
#define HAVE_DECL_IBV_GET_DEVICE_NAME 1
#define HAVE_DECL_IBV_LINK_LAYER_ETHERNET 1
#define HAVE_DECL_IBV_LINK_LAYER_INFINIBAND 1
#define HAVE_DECL_IBV_ODP_SUPPORT_IMPLICIT 0
#define HAVE_DECL_IBV_QPF_GRH_REQUIRED 1
#define HAVE_DECL_IBV_QUERY_DEVICE_EX 1
#define HAVE_DECL_IBV_QUERY_GID   1
#define HAVE_DECL_IBV_WC_STATUS_STR 1
#define HAVE_DECL_INOTIFY_ADD_WATCH 1
#define HAVE_DECL_INOTIFY_INIT    1
#define HAVE_DECL_IN_ATTRIB       1
#define HAVE_DECL_IPPROTO_TCP     1
#define HAVE_DECL_MADV_FREE       0
#define HAVE_DECL_MADV_REMOVE     1
#define HAVE_DECL_MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE 1
#define HAVE_DECL_MLX5DV_CREATE_QP 1
#define HAVE_DECL_MLX5DV_DCTYPE_DCT 1
#define HAVE_DECL_MLX5DV_DEVX_SUBSCRIBE_DEVX_EVENT 0
#define HAVE_DECL_MLX5DV_INIT_OBJ 1
#define HAVE_DECL_MLX5DV_IS_SUPPORTED 1
#define HAVE_DECL_MLX5DV_OBJ_AH   1
#define HAVE_DECL_MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE 1
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_BF 0
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_NC 0
#define HAVE_DECL_POSIX_MADV_DONTNEED 1
#define HAVE_DECL_PR_SET_PTRACER  1
#define HAVE_DECL_SOL_SOCKET      1
#define HAVE_DECL_SO_KEEPALIVE    1
#define HAVE_DECL_SPEED_UNKNOWN   1
#define HAVE_DECL_STRERROR_R      1
#define HAVE_DECL_SYS_BRK         1
#define HAVE_DECL_SYS_IPC         0
#define HAVE_DECL_SYS_MADVISE     1
#define HAVE_DECL_SYS_MMAP        1
#define HAVE_DECL_SYS_MREMAP      1
#define HAVE_DECL_SYS_MUNMAP      1
#define HAVE_DECL_SYS_SHMAT       1
#define HAVE_DECL_SYS_SHMDT       1
#define HAVE_DECL_TCP_KEEPCNT     1
#define HAVE_DECL_TCP_KEEPIDLE    1
#define HAVE_DECL_TCP_KEEPINTVL   1
#define HAVE_DECL___PPC_GET_TIMEBASE_FREQ 0
#define HAVE_DEVX                 1
#define HAVE_DLFCN_H              1
#define HAVE_HW_TIMER             1
#define HAVE_IB                   1
#define HAVE_IBV_DM               1
#define HAVE_IN6_ADDR_S6_ADDR32   1
#define HAVE_INFINIBAND_MLX5DV_H  1
#define HAVE_INFINIBAND_TM_TYPES_H 1
#define HAVE_INOTIFY              1
#define HAVE_INTTYPES_H           1
#define HAVE_IP_IP_DST            1
#define HAVE_LIBGEN_H             1
#define HAVE_LIBRT                1
#define HAVE_LINUX_FUTEX_H        1
#define HAVE_LINUX_IP_H           1
#define HAVE_LINUX_MMAN_H         1
#define HAVE_MALLOC_GET_STATE     1
#define HAVE_MALLOC_H             1
#define HAVE_MALLOC_HOOK          1
#define HAVE_MALLOC_SET_STATE     1
#define HAVE_MALLOC_TRIM          1
#define HAVE_MEMALIGN             1
#define HAVE_MEMORY_H             1
#define HAVE_MLX5_HW              1
#define HAVE_MLX5_HW_UD           1
#define HAVE_MREMAP               1
#define HAVE_NETINET_IP_H         1
#define HAVE_NET_ETHERNET_H       1
#define HAVE_NUMA                 1
#define HAVE_NUMAIF_H             1
#define HAVE_NUMA_H               1
#define HAVE_ODP                  1
#define HAVE_POSIX_MEMALIGN       1
#define HAVE_PREFETCH             1
#define HAVE_SCHED_GETAFFINITY    1
#define HAVE_SCHED_SETAFFINITY    1
#define HAVE_SIGACTION_SA_RESTORER 1
#define HAVE_SIGEVENT_SIGEV_UN_TID 1
#define HAVE_SIGHANDLER_T         1
#define HAVE_STDINT_H             1
#define HAVE_STDLIB_H             1
#define HAVE_STRERROR_R           1
#define HAVE_STRINGS_H            1
#define HAVE_STRING_H             1
#define HAVE_STRUCT_BITMASK       1
#define HAVE_STRUCT_IBV_TM_CAPS_FLAGS 1
#define HAVE_STRUCT_MLX5DV_CQ_CQ_UAR 1
#define HAVE_SYS_EPOLL_H          1
#define HAVE_SYS_EVENTFD_H        1
#define HAVE_SYS_STAT_H           1
#define HAVE_SYS_TYPES_H          1
#define HAVE_SYS_UIO_H            1
#define HAVE_TL_DC                1
#define HAVE_TL_RC                1
#define HAVE_TL_UD                1
#define HAVE_UCM_PTMALLOC286      1
#define HAVE_UNISTD_H             1
#define HAVE___CLEAR_CACHE        1
#define HAVE___CURBRK             1
#define HAVE___SIGHANDLER_T       1
#define IBV_HW_TM                 1
#define LT_OBJDIR                 ".libs/"
#define NVALGRIND                 1
#define PACKAGE                   "ucx"
#define PACKAGE_BUGREPORT         ""
#define PACKAGE_NAME              "ucx"
#define PACKAGE_STRING            "ucx 1.11"
#define PACKAGE_TARNAME           "ucx"
#define PACKAGE_URL               ""
#define PACKAGE_VERSION           "1.11"
#define STDC_HEADERS              1
#define STRERROR_R_CHAR_P         1
#define UCM_BISTRO_HOOKS          1
#define UCS_MAX_LOG_LEVEL         UCS_LOG_LEVEL_DEBUG
#define UCT_TCP_EP_KEEPALIVE      1
#define UCT_UD_EP_DEBUG_HOOKS     0
#define UCX_CONFIGURE_FLAGS       "--disable-logging --disable-debug --disable-assertions --disable-params-check --with-verbs --with-mlx5-dv --with-ib-hw-tm --with-rc --with-ud --with-dc --with-dm --with-mcpu --with-march --disable-backtrace-detail --enable-devel-headers --enable-optimizations --enable-shared --disable-static --with-knem=/dtu/sw/dcc/SL73/2021-nov/XeonGold6126/gnu/11.2.0/knem/1.1.4 --prefix=/dtu/sw/dcc/SL73/2021-nov/XeonGold6126/gnu/11.2.0/ucx/1.11.2"
#define UCX_MODULE_SUBDIR         "ucx"
#define VERSION                   "1.11"
#define restrict                  __restrict
#define test_MODULES              ":module"
#define ucm_MODULES               ""
#define ucs_MODULES               ""
#define uct_MODULES               ":ib:cma:knem"
#define uct_cuda_MODULES          ""
#define uct_ib_MODULES            ""
#define uct_rocm_MODULES          ""
#define ucx_perftest_MODULES      ""
#
# Memory domain: posix
#     Component: posix
#             allocate: unlimited
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#      Transport: posix
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#      Transport: sysv
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: self
#         Device: memory0
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: tcp
#         Device: eth1
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 1131.64/ppn + 0.00 MB/sec
#              latency: 5258 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: lo
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: ib0
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 6239.81/ppn + 0.00 MB/sec
#              latency: 5210 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: mlx5_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#
#      Transport: rc_verbs
#         Device: mlx5_0:1
#  System device: 0000:06:00.0 (0)
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 700 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 4 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 4 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 30
#     device num paths: 1
#              max eps: 256
#       device address: 3 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: rc_mlx5
#         Device: mlx5_0:1
#  System device: 0000:06:00.0 (0)
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 700 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 220
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 234
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 186
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 30
#     device num paths: 1
#              max eps: 256
#       device address: 3 bytes
#           ep address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: dc_mlx5
#         Device: mlx5_0:1
#  System device: 0000:06:00.0 (0)
[1641387609.196524] [n-62-31-9:181277:0]         dc_mlx5.c:505  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument
#   < failed to open interface >
#
#      Transport: ud_verbs
#         Device: mlx5_0:1
#  System device: 0000:06:00.0 (0)
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 730 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 3952
#           connection: to ep, to iface
#      device priority: 30
#     device num paths: 1
#              max eps: inf
#       device address: 3 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_0:1
#  System device: 0000:06:00.0 (0)
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 730 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 30
#     device num paths: 1
#              max eps: inf
#       device address: 3 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#
#      Transport: cma
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 400 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check
#

Artemy-Mellanox · 2022-01-05T14:33:13Z

could you please check dmesg for syndrome - this is how-to do it
https://github.com/openucx/ucx/wiki/How-to-get-FW-syndrome-when-using-DEVX

zerothi · 2022-01-05T21:20:19Z

Thanks, but I don't have sudo access, so changing the dynamic debug isn't going to work?

zerothi · 2022-01-10T22:05:00Z

could you please check dmesg for syndrome - this is how-to do it https://github.com/openucx/ucx/wiki/How-to-get-FW-syndrome-when-using-DEVX

@Artemy-Mellanox I got help from our sys-admin, here is the attached output:

Output of dmesg

Linux n-62-31-9 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 23 21:51:54 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
echo 'file drivers/infiniband/hw/mlx5/* -p' | sudo tee /sys/kernel/debug/dynamic_debug/control

[564689.013592] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.013843] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.013907] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.013993] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.014059] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.014135] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.014197] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.014260] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.014334] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.014779] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.014842] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.014904] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.014982] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.015044] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.015109] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.015182] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.015244] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.015309] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.015727] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.015789] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.015851] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.015926] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.016303] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.016383] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.016444] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.016508] infiniband mlx5_0: alloLinux n-62-31-9 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 23 21:51:54 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
echo 'file drivers/infiniband/hw/mlx5/* -p' | sudo tee /sys/kernel/debug/dynamic_debug/control

[564689.013592] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.013843] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.013907] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.013993] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.014059] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.014135] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.014197] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.014260] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.014334] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.014779] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.014842] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.014904] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.014982] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.015044] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.015109] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.015182] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.015244] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.015309] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.015727] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.015789] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.015851] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.015926] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.016303] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.016383] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.016444] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.016508] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.016581] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.017043] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.017402] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.017475] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.017536] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.017597] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.017673] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.017735] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.017797] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.017873] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.018509] infiniband mlx5_0: print_lib_caps:1551:(pid 202927): MLX5_LIB_CAP_4K_UAR = y
[564689.018616] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x0, pfn 0x0000000021ffe011
[564689.018628] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x1, pfn 0x0000000021ffe012
[564689.018632] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x2, pfn 0x0000000021ffe013
[564689.018636] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x3, pfn 0x0000000021ffe014
[564689.018640] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x4, pfn 0x0000000021ffe015
[564689.018644] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x5, pfn 0x0000000021ffe016
[564689.018647] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x6, pfn 0x0000000021ffe017
[564689.018651] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x7, pfn 0x0000000021ffe018
[564689.021054] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1779000, size 128, npages 1, page_shift 12, ncont 1
[564689.021057] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1be66e5000
[564689.021492] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.021518] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ea2cc0000
[564689.022494] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa300
[564689.060185] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523817000, size 524288, npages 128, page_shift 12, ncont 128
[564689.060189] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f54896000
[564689.060191] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f437f0000
[564689.060192] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2f5b4b2000
[564689.060194] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2f56620000
[564689.060196] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2f35b78000
[564689.060197] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2ea05cc000
[564689.060199] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2e86b56000
[564689.060201] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2f35b7e000
[564689.060202] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x28019bb000
[564689.060204] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2ef3aa1000
[564689.060206] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2f38dac000
[564689.060207] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2f390a9000
[564689.060209] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2f07dd6000
[564689.060211] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2f5b5f6000
[564689.060212] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2e89ec0000
[564689.060214] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2d9eddb000
[564689.060216] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x1ed69ac000
[564689.060217] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x1bc172f000
[564689.060219] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2f00394000
[564689.060220] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2e9d2e6000
[564689.060222] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2effbfb000
[564689.060224] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2f00609000
[564689.060225] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x1be2a57000
[564689.060227] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2f5a327000
[564689.060229] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x1e745ca000
[564689.060230] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2e48f8b000
[564689.060232] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x2da57a3000
[564689.060233] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2f58fc9000
[564689.060235] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2ea082e000
[564689.060237] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x1dbf546000
[564689.060238] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2a5abf3000
[564689.060240] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2cf49d3000
[564689.060242] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[32] 0x2f31e11000
[564689.060244] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[33] 0x2f5cc60000
[564689.060245] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[34] 0x2ea6014000
[564689.060247] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[35] 0x2de1f44000
[564689.060248] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[36] 0x2f32411000
[564689.060250] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[37] 0x2efea37000
[564689.060252] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[38] 0x2efb5ea000
[564689.060253] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[39] 0x2f47004000
[564689.060255] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[40] 0x2e28fe2000
[564689.060257] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[41] 0x2471733000
[564689.060258] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[42] 0x2f00c6e000
[564689.060260] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[43] 0x2e65160000
[564689.060262] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[44] 0x2ef9fa4000
[564689.060263] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[45] 0x2ee8144000
[564689.060265] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[46] 0x2ddf5d7000
[564689.060266] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[47] 0x2f450a1000
[564689.060268] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[48] 0x2ee1e39000
[564689.060270] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[49] 0x2efda96000
[564689.060271] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[50] 0x2f44391000
[564689.060273] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[51] 0x2e8b3ee000
[564689.060275] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[52] 0x2891ff0000
[564689.060276] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[53] 0x2f5b3ae000
[564689.060278] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[54] 0x2eca4a9000
[564689.060280] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[55] 0x2e9db34000
[564689.060281] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[56] 0x293d6fe000
[564689.060283] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[57] 0x2de8fc0000
[564689.060285] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[58] 0x2f3e81e000
[564689.060286] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[59] 0x2f3776e000
[564689.060288] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[60] 0x2f33edf000
[564689.060290] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[61] 0x1f7957a000
[564689.060291] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[62] 0x2a51620000
[564689.060293] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[63] 0x2ecc695000
[564689.060294] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[64] 0x1e8d0c0000
[564689.060296] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[65] 0x2f59735000
[564689.060298] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[66] 0x2edc143000
[564689.060299] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[67] 0x20bb54e000
[564689.060301] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[68] 0x2f50ee6000
[564689.060303] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[69] 0x2f033ce000
[564689.060304] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[70] 0x2ef843d000
[564689.060306] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[71] 0x2e67f5c000
[564689.060308] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[72] 0x2f37e76000
[564689.060309] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[73] 0x2674ec8000
[564689.060311] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[74] 0x2e6e236000
[564689.060312] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[75] 0x2f403e5000
[564689.060314] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[76] 0x2f3c199000
[564689.060316] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[77] 0x2e82c95000
[564689.060317] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[78] 0x213928d000
[564689.060319] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[79] 0x2f04f7c000
[564689.060321] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[80] 0x2f34f3c000
[564689.060322] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[81] 0x2588026000
[564689.060324] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[82] 0x2f58404000
[564689.060326] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[83] 0x2ef31c9000
[564689.060327] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[84] 0x2f3363c000
[564689.060329] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[85] 0x2f5c088000
[564689.060331] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[86] 0x2f3d5d5000
[564689.060332] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[87] 0x1f13233000
[564689.060334] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[88] 0x2f5870e000
[564689.060335] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[89] 0x2f460e2000
[564689.060337] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[90] 0x1bfe631000
[564689.060339] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[91] 0x1d4667f000
[564689.060340] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[92] 0x2e9b540000
[564689.060342] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[93] 0x2f390aa000
[564689.060344] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[94] 0x2f3d45a000
[564689.060345] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[95] 0x2eb61af000
[564689.060347] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[96] 0x2b38b85000
[564689.060348] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[97] 0x2f51858000
[564689.060350] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[98] 0x2f0a863000
[564689.060352] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[99] 0x2f35fd7000
[564689.060353] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[100] 0x2f50526000
[564689.060355] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[101] 0x2eac802000
[564689.060357] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[102] 0x2e777ce000
[564689.060358] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[103] 0x1f39b6f000
[564689.060360] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[104] 0x2e3f94a000
[564689.060362] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[105] 0x2f5ad3b000
[564689.060363] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[106] 0x1e8cf7c000
[564689.060365] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[107] 0x2f33f23000
[564689.060366] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[108] 0x2f5a00e000
[564689.060368] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[109] 0x2f3c5ed000
[564689.060370] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[110] 0x2ee34fa000
[564689.060371] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[111] 0x2f5ab78000
[564689.060373] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[112] 0x2e53349000
[564689.060375] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[113] 0x2eded22000
[564689.060376] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[114] 0x1cabbef000
[564689.060378] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[115] 0x2ed6b49000
[564689.060380] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[116] 0x2e885fd000
[564689.060381] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[117] 0x2e2e1f9000
[564689.060383] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[118] 0x25a59c5000
[564689.060385] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[119] 0x2ed4efb000
[564689.060386] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[120] 0x2f38fba000
[564689.060388] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[121] 0x2f3a91e000
[564689.060389] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[122] 0x2f371be000
[564689.060391] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[123] 0x2f362a6000
[564689.060393] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[124] 0x2ee6224000
[564689.060394] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[125] 0x2efc477000
[564689.060396] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[126] 0x2a75b37000
[564689.060398] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2e36325000
[564689.067632] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.067797] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f85237d5000, size 262144, npages 64, page_shift 12, ncont 64
[564689.067799] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2d42a5f000
[564689.067801] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2ee0c6b000
[564689.067803] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x1ec97e8000
[564689.067804] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2e98ece000
[564689.067806] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2ed3195000
[564689.067807] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x1e11871000
[564689.067809] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2e93e13000
[564689.067811] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2e4387e000
[564689.067812] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2f40f08000
[564689.067814] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2e48f96000
[564689.067816] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2f5bf5b000
[564689.067817] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2f3958e000
[564689.067819] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2551e5d000
[564689.067821] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2f5527f000
[564689.067822] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2f0367a000
[564689.067824] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2ef927f000
[564689.067826] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x2eae3df000
[564689.067827] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2e2491a000
[564689.067829] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2ee12bc000
[564689.067831] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2eefa56000
[564689.067832] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2e22a25000
[564689.067834] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2f50af4000
[564689.067836] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2f4bd04000
[564689.067837] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2e33a7e000
[564689.067839] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x1be21e2000
[564689.067840] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2ef9ff1000
[564689.067842] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x2efec24000
[564689.067844] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2efc906000
[564689.067845] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2ee4a29000
[564689.067847] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x1d452bf000
[564689.067857] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2f34e93000
[564689.067863] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f36377000
[564689.067868] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[32] 0x2deeb3e000
[564689.067872] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[33] 0x2ef62aa000
[564689.067877] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[34] 0x2eba201000
[564689.067881] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[35] 0x2e1b4be000
[564689.067885] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[36] 0x2f54f48000
[564689.067889] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[37] 0x2ef8f69000
[564689.067893] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[38] 0x2ed2623000
[564689.067898] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[39] 0x2e94d3a000
[564689.067902] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[40] 0x2f581c9000
[564689.067906] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[41] 0x2efb026000
[564689.067911] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[42] 0x2f3ca3e000
[564689.067915] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[43] 0x2e10d57000
[564689.067919] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[44] 0x2ed8164000
[564689.067924] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[45] 0x2f3ed13000
[564689.067927] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[46] 0x2f32034000
[564689.067929] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[47] 0x2f592f1000
[564689.067931] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[48] 0x2f554b9000
[564689.067932] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[49] 0x2f084d0000
[564689.067934] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[50] 0x1e635a5000
[564689.067935] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[51] 0x1bbaebc000
[564689.067937] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[52] 0x2ec07c9000
[564689.067939] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[53] 0x22037bd000
[564689.067940] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[54] 0x2f0ac79000
[564689.067942] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[55] 0x2f3ee26000
[564689.067944] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[56] 0x22af089000
[564689.067945] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[57] 0x2ecfe5a000
[564689.067947] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[58] 0x2e4728c000
[564689.067949] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[59] 0x2f3b898000
[564689.067953] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[60] 0x2f59981000
[564689.067957] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[61] 0x2865146000
[564689.067962] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[62] 0x2ebc88f000
[564689.067966] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[63] 0x2e63719000
[564689.071425] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.071632] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
[564689.071634] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f5927d000
[564689.071636] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2f51d1f000
[564689.071638] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2e02779000
[564689.071639] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2cf46ef000
[564689.071641] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2f567a1000
[564689.071643] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2efd644000
[564689.071644] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x19c6286000
[564689.071646] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2e50188000
[564689.071648] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2f3cae0000
[564689.071649] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2253dba000
[564689.071651] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2ebe57b000
[564689.071653] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2e74c83000
[564689.071654] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2de9e93000
[564689.071656] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2f5a01b000
[564689.071657] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x1c16df8000
[564689.071659] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x2f335ec000
[564689.071661] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2e98575000
[564689.071662] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2f348e1000
[564689.071664] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2f7ac00000
[564689.071666] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2ec66cf000
[564689.071667] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2ee712a000
[564689.071669] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2ed3c74000
[564689.071671] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x1c92ddd000
[564689.071672] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x2e9b52e000
[564689.071674] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2f59a35000
[564689.071675] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x2e9c044000
[564689.071677] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2e6d90e000
[564689.071679] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2efaa17000
[564689.071680] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x2f0325b000
[564689.071682] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x1bfd791000
[564689.071683] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.073624] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa302
[564689.073766] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (2048)
[564689.073769] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.073777] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17e7000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.073779] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ef10da000
[564689.073781] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2279fc5000
[564689.073783] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2efe00a000
[564689.073785] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2c2333e000
[564689.073786] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2eaa05f000
[564689.073788] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2f44367000
[564689.073790] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2f58d62000
[564689.073791] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2f5c4c9000
[564689.073793] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2f3b971000
[564689.073794] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x21d5423000
[564689.073796] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2f3d5bf000
[564689.073798] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2f47f43000
[564689.073799] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2f3c19f000
[564689.073801] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2eba4e6000
[564689.073803] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2e27c3f000
[564689.073804] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2dc3232000
[564689.073806] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x2eb21aa000
[564689.073807] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2ef1a32000
[564689.073809] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x1ccf798000
[564689.073811] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x26a73ef000
[564689.073812] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x1c12baf000
[564689.073814] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2efad9e000
[564689.073815] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2f50bdd000
[564689.073817] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2e76cdc000
[564689.073819] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x2e90c1f000
[564689.073821] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2e812d8000
[564689.073822] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x2f43d97000
[564689.073824] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2f0a867000
[564689.073825] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2e9f6eb000
[564689.073827] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x2e82aa9000
[564689.073829] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2f3493d000
[564689.073830] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ee81f6000
[564689.075758] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c69, mlx qpn 0x8c69, rcqn 0xc2, scqn 0xc1
[564689.079417] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.079420] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f38fba000
 ... shortening too long lines pas[i] continued
[564689.079897] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2f3a91e000
[564689.089958] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.090258] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523715000, size 524288, npages 128, page_shift 12, ncont 128
[564689.090261] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1d94a45000
 ... shortening too long lines pas[i] continued
[564689.090470] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2ede513000
[564689.094881] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.094940] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
 ... shortening too long lines pas[i] continued
[564689.094991] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.096061] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa304
[564689.098097] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.098100] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f35b84000
 ... shortening too long lines pas[i] continued
[564689.098517] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2eefc29000
[564689.098883] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.099063] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.099066] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.099291] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.099473] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.099578] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3f31d000
[564689.099579] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x1459287000
[564689.099581] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2f58bd7000
[564689.099583] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2efba75000
[564689.099584] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x1eddf80000
[564689.099586] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2f41b4e000
[564689.099588] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2f3963a000
[564689.099589] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2f45144000
[564689.099591] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2effa99000
[564689.099593] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2eb5562000
[564689.099594] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2ee1d17000
[564689.099596] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2eee576000
[564689.099598] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2efda93000
[564689.099599] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2ee1225000
[564689.099601] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2ee0f8f000
[564689.099603] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2f07905000
[564689.099604] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x2ec6e4f000
[564689.099606] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2eb5462000
[564689.099607] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2f32220000
[564689.099609] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2f3a34e000
[564689.099611] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2efa097000
[564689.099612] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2e10353000
[564689.099614] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2865140000
[564689.099616] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2e9b86d000
[564689.099617] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x2edf7bc000
[564689.099619] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2e3a347000
[564689.099620] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x1c16cb1000
[564689.099622] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2f35285000
[564689.099624] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2ee5d2d000
[564689.099625] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x22232eb000
[564689.099627] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2956600000
[564689.099629] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2efd35e000
[564689.100372] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa305
[564689.101741] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 32768, npages 8, page_shift 12, ncont 8
[564689.101745] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
[564689.101747] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f5b50d000
[564689.101749] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2e542fa000
[564689.101751] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2efa514000
[564689.101753] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2f3b825000
[564689.101755] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2bd0d2e000
[564689.101757] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2eac5ba000
[564689.101759] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2e80902000
[564689.101923] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.101988] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.101990] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.102239] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.102406] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.102436] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.102439] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.102447] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x1795000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.102449] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ec879d000
 ... shortening too long lines pas[i] continued
[564689.102500] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x1be25e6000
[564689.103210] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6a, mlx qpn 0x8c6a, rcqn 0xc2, scqn 0xc1
[564689.104619] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.104708] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.104710] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.104713] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xc7bfe
[564689.104715] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2b38b85003
 ... shortening too long lines pas[i] continued
[564689.105009] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[132] 0x2ee4a29003
[564689.108492] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 131072, npages 32, page_shift 12, ncont 32
[564689.108495] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
 ... shortening too long lines pas[i] continued
[564689.108557] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed8cd4000
[564689.108779] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.108845] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.108848] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.109094] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.109263] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.109292] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.109294] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.109302] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17ad000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.109304] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ed9578000
 ... shortening too long lines pas[i] continued
[564689.109366] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed3c74000
[564689.109653] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6b, mlx qpn 0x8c6b, rcqn 0xc2, scqn 0xc1
[564689.110084] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.110192] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.110195] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.110197] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xca05c
[564689.110199] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee4a29003
 ... shortening too long lines pas[i] continued
[564689.110456] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[132] 0x2b38b85003
cate_uars:1613:(pid 202927): allocated uar 23
[564689.016581] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.017043] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.017402] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.017475] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.017536] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.017597] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.017673] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.017735] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.017797] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.017873] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.018509] infiniband mlx5_0: print_lib_caps:1551:(pid 202927): MLX5_LIB_CAP_4K_UAR = y
[564689.018616] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x0, pfn 0x0000000021ffe011
[564689.018628] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x1, pfn 0x0000000021ffe012
[564689.018632] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x2, pfn 0x0000000021ffe013
[564689.018636] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x3, pfn 0x0000000021ffe014
[564689.018640] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x4, pfn 0x0000000021ffe015
[564689.018644] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x5, pfn 0x0000000021ffe016
[564689.018647] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x6, pfn 0x0000000021ffe017
[564689.018651] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x7, pfn 0x0000000021ffe018
[564689.021054] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1779000, size 128, npages 1, page_shift 12, ncont 1
[564689.021057] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1be66e5000
[564689.021492] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.021518] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ea2cc0000
[564689.022494] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa300
[564689.060185] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523817000, size 524288, npages 128, page_shift 12, ncont 128
[564689.060189] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f54896000
 ... shortening too long lines pas[i] continued
[564689.060398] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2e36325000
[564689.067632] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.067797] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f85237d5000, size 262144, npages 64, page_shift 12, ncont 64
[564689.067799] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2d42a5f000
 ... shortening too long lines pas[i] continued
[564689.067966] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[63] 0x2e63719000
[564689.071425] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.071632] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
 ... shortening too long lines pas[i] continued
[564689.071683] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.073624] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa302
[564689.073766] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (2048)
[564689.073769] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.073777] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17e7000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.073779] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ef10da000
 ... shortening too long lines pas[i] continued
[564689.073830] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ee81f6000
[564689.075758] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c69, mlx qpn 0x8c69, rcqn 0xc2, scqn 0xc1
[564689.079417] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.079420] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f38fba000
 ... shortening too long lines pas[i] continued
[564689.079897] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2f3a91e000
[564689.089958] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.090258] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523715000, size 524288, npages 128, page_shift 12, ncont 128
[564689.090261] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1d94a45000
 ... shortening too long lines pas[i] continued
[564689.090470] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2ede513000
[564689.094881] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.094940] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
 ... shortening too long lines pas[i] continued
[564689.094991] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.096061] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa304
[564689.098097] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.098100] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f35b84000
 ... shortening too long lines pas[i] continued
[564689.098517] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2eefc29000
[564689.098883] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.099063] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.099066] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
... shortening too long lines pas[i] continued
[564689.099291] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.099473] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.099578] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3f31d000
[564689.099579] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x1459287000
[564689.099581] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2f58bd7000
[564689.099583] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2efba75000
[564689.099584] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x1eddf80000
[564689.099586] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2f41b4e000
[564689.099588] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2f3963a000
[564689.099589] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2f45144000
[564689.099591] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2effa99000
[564689.099593] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2eb5562000
[564689.099594] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x2ee1d17000
[564689.099596] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2eee576000
[564689.099598] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2efda93000
[564689.099599] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2ee1225000
[564689.099601] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2ee0f8f000
[564689.099603] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2f07905000
[564689.099604] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x2ec6e4f000
[564689.099606] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2eb5462000
[564689.099607] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2f32220000
[564689.099609] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2f3a34e000
[564689.099611] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2efa097000
[564689.099612] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x2e10353000
[564689.099614] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2865140000
[564689.099616] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2e9b86d000
[564689.099617] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x2edf7bc000
[564689.099619] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2e3a347000
[564689.099620] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x1c16cb1000
[564689.099622] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x2f35285000
[564689.099624] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2ee5d2d000
[564689.099625] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x22232eb000
[564689.099627] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2956600000
[564689.099629] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2efd35e000
[564689.100372] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa305
[564689.101741] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 32768, npages 8, page_shift 12, ncont 8
[564689.101745] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
[564689.101747] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f5b50d000
[564689.101749] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2e542fa000
[564689.101751] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2efa514000
[564689.101753] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2f3b825000
[564689.101755] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2bd0d2e000
[564689.101757] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2eac5ba000
[564689.101759] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2e80902000
[564689.101923] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.101988] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.101990] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.102239] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.102406] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.102436] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.102439] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.102447] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x1795000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.102449] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ec879d000
 ... shortening too long lines pas[i] continued
[564689.102500] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x1be25e6000
[564689.103210] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6a, mlx qpn 0x8c6a, rcqn 0xc2, scqn 0xc1
[564689.104619] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.104708] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.104710] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.104713] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xc7bfe
[564689.104715] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2b38b85003
[564689.104717] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f51858003
[564689.104718] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2ef843d003
[564689.104720] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2e67f5c003
[564689.104722] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2f37e76003
[564689.104723] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2674ec8003
[564689.104725] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2e6e236003
[564689.104727] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2f403e5003
[564689.104728] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[8] 0x2f3c199003
[564689.104730] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[9] 0x2e82c95003
[564689.104731] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[10] 0x213928d003
[564689.104733] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[11] 0x2f04f7c003
[564689.104735] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[12] 0x2f34f3c003
[564689.104736] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[13] 0x2588026003
[564689.104738] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[14] 0x2f58404003
[564689.104740] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[15] 0x2ef31c9003
[564689.104741] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[16] 0x293d6fe003
[564689.104743] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[17] 0x2de8fc0003
[564689.104745] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[18] 0x2f3e81e003
[564689.104746] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[19] 0x2f3776e003
[564689.104748] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[20] 0x2f33edf003
[564689.104750] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[21] 0x1f7957a003
[564689.104751] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[22] 0x2a51620003
[564689.104753] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[23] 0x2ecc695003
[564689.104755] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[24] 0x1e8d0c0003
[564689.104756] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[25] 0x2f59735003
[564689.104758] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[26] 0x2edc143003
[564689.104759] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[27] 0x20bb54e003
[564689.104761] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[28] 0x2f50ee6003
[564689.104769] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[29] 0x2f033ce003
[564689.104773] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[30] 0x2f450a1003Linux n-62-31-9 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 23 21:51:54 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
echo 'file drivers/infiniband/hw/mlx5/* -p' | sudo tee /sys/kernel/debug/dynamic_debug/control

[564689.013592] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.013843] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.013907] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.013993] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.014059] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.014135] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.014197] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.014260] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.014334] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.014779] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.014842] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.014904] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.014982] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.015044] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.015109] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.015182] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.015244] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.015309] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.015727] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support no, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.015789] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.015851] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.015926] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.016303] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.016383] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.016444] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.016508] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.016581] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.017043] infiniband mlx5_0: calc_total_bfregs:1596:(pid 202927): uar_4k: fw support no, lib support yes, user requested 16 bfregs, allocated 16, total bfregs 1040, using 520 sys pages
[564689.017402] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 17
[564689.017475] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 18
[564689.017536] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 19
[564689.017597] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 20
[564689.017673] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 21
[564689.017735] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 22
[564689.017797] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 23
[564689.017873] infiniband mlx5_0: allocate_uars:1613:(pid 202927): allocated uar 24
[564689.018509] infiniband mlx5_0: print_lib_caps:1551:(pid 202927): MLX5_LIB_CAP_4K_UAR = y
[564689.018616] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x0, pfn 0x0000000021ffe011
[564689.018628] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x1, pfn 0x0000000021ffe012
[564689.018632] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x2, pfn 0x0000000021ffe013
[564689.018636] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x3, pfn 0x0000000021ffe014
[564689.018640] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x4, pfn 0x0000000021ffe015
[564689.018644] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x5, pfn 0x0000000021ffe016
[564689.018647] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x6, pfn 0x0000000021ffe017
[564689.018651] infiniband mlx5_0: uar_mmap:2138:(pid 202927): uar idx 0x7, pfn 0x0000000021ffe018
[564689.021054] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1779000, size 128, npages 1, page_shift 12, ncont 1
[564689.021057] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1be66e5000
[564689.021492] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.021518] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ea2cc0000
[564689.022494] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa300
[564689.060185] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523817000, size 524288, npages 128, page_shift 12, ncont 128
[564689.060189] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f54896000
 ... shortening too long lines pas[i] continued
[564689.060398] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2e36325000
[564689.067632] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.067797] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f85237d5000, size 262144, npages 64, page_shift 12, ncont 64
[564689.067799] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2d42a5f000
 ... shortening too long lines pas[i] continued
[564689.067966] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[63] 0x2e63719000
[564689.071425] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.071632] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
 ... shortening too long lines pas[i] continued
[564689.071683] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.073624] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa302
[564689.073766] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (2048)
[564689.073769] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.073777] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17e7000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.073779] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ef10da000
 ... shortening too long lines pas[i] continued
[564689.073830] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ee81f6000
[564689.075758] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c69, mlx qpn 0x8c69, rcqn 0xc2, scqn 0xc1
[564689.079417] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.079420] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f38fba000
 ... shortening too long lines pas[i] continued
[564689.079897] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2f3a91e000
[564689.089958] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.090258] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523715000, size 524288, npages 128, page_shift 12, ncont 128
[564689.090261] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x1d94a45000
 ... shortening too long lines pas[i] continued
[564689.090470] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2ede513000
[564689.094881] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.094940] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x29e130c000
 ... shortening too long lines pas[i] continued
[564689.094991] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2f33aa0000
[564689.096061] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa304
[564689.098097] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x7f8523797000, size 1048576, npages 256, page_shift 12, ncont 256
[564689.098100] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f35b84000
 ... shortening too long lines pas[i] continued
[564689.098517] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[255] 0x2eefc29000
[564689.098883] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.099063] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.099066] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.099291] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.099473] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.099578] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3f31d000
 ... shortening too long lines pas[i] continued
[564689.099629] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2efd35e000
[564689.100372] infiniband mlx5_0: mlx5_ib_create_srq:337:(pid 202927): create SRQ with srqn 0xa305
[564689.101741] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 32768, npages 8, page_shift 12, ncont 8
[564689.101745] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
[564689.101747] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[1] 0x2f5b50d000
[564689.101749] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[2] 0x2e542fa000
[564689.101751] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[3] 0x2efa514000
[564689.101753] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[4] 0x2f3b825000
[564689.101755] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[5] 0x2bd0d2e000
[564689.101757] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[6] 0x2eac5ba000
[564689.101759] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[7] 0x2e80902000
[564689.101923] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.101988] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.101990] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.102239] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.102406] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.102436] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.102439] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.102447] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x1795000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.102449] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ec879d000
 ... shortening too long lines pas[i] continued
[564689.102500] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x1be25e6000
[564689.103210] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6a, mlx qpn 0x8c6a, rcqn 0xc2, scqn 0xc1
[564689.104619] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.104708] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.104710] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.104713] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xc7bfe
[564689.104715] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2b38b85003
 ... shortening too long lines pas[i] continued
[564689.105009] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[132] 0x2ee4a29003
[564689.108492] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 131072, npages 32, page_shift 12, ncont 32
[564689.108495] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
 ... shortening too long lines pas[i] continued
[564689.108557] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed8cd4000
[564689.108779] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.108845] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.108848] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
 ... shortening too long lines pas[i] continued
[564689.109094] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.109263] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.109292] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.109294] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.109302] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17ad000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.109304] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ed9578000
... shortening too long lines pas[i] continued
[564689.109366] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed3c74000
[564689.109653] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6b, mlx qpn 0x8c6b, rcqn 0xc2, scqn 0xc1
[564689.110084] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.110192] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.110195] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.110197] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xca05c
[564689.110199] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee4a29003
... shortening too long lines pas[i] continued
[564689.105009] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[132] 0x2ee4a29003
[564689.108492] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x1784000, size 131072, npages 32, page_shift 12, ncont 32
[564689.108495] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2f3ac83000
... shortening too long lines pas[i] continued
[564689.108557] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed8cd4000
[564689.108779] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc1
[564689.108845] infiniband mlx5_0: create_cq_user:726:(pid 202927): addr 0x17e1000, size 524288, npages 128, page_shift 12, ncont 128
[564689.108848] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee6224000
... shortening too long lines pas[i] continued
[564689.109094] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[127] 0x2efc477000
[564689.109263] infiniband mlx5_0: mlx5_ib_create_cq:962:(pid 202927): cqn 0xc2
[564689.109292] infiniband mlx5_0: create_qp_common:1968:(pid 202927): requested sq_wqe_count (1024)
[564689.109294] infiniband mlx5_0: create_user_qp:822:(pid 202927): bfregn 0xc, uar_index 0x0
[564689.109302] infiniband mlx5_0: mlx5_ib_umem_get:687:(pid 202927): addr 0x17ad000, size 131072, npages 32, page_shift 12, ncont 32, offset 0
[564689.109304] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ed9578000
... shortening too long lines pas[i] continued
[564689.109366] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[31] 0x2ed3c74000
[564689.109653] infiniband mlx5_0: mlx5_ib_create_qp:2557:(pid 202927): ib qpnum 0x8c6b, mlx qpn 0x8c6b, rcqn 0xc2, scqn 0xc1
[564689.110084] infiniband mlx5_0: mlx5_ib_reg_user_mr:1310:(pid 202927): start 0x7f8523813000, virt_addr 0x7f8523813000, length 0x85000, access_flags 0xf
[564689.110192] infiniband mlx5_0: mr_umem_get:872:(pid 202927): npages 133, ncont 133, order 8, page_shift 12
[564689.110195] infiniband mlx5_0: alloc_cached_mr:506:(pid 202927): order 8, cache index 6
[564689.110197] infiniband mlx5_0: mlx5_ib_reg_user_mr:1363:(pid 202927): mkey 0xca05c
[564689.110199] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[0] 0x2ee4a29003
... shortening too long lines pas[i] continued
[564689.110456] infiniband mlx5_0: __mlx5_ib_populate_pas:196:(pid 202927): pas[132] 0x2b38b85003

Artemy-Mellanox · 2022-01-11T04:37:25Z

was there some problem with enabling dynamic debug?
in attached output I see command file drivers/infiniband/hw/mlx5/* +p while we need func mlx5_cmd_check +p to get syndrome

zerothi · 2022-01-11T07:08:36Z

was there some problem with enabling dynamic debug? in attached output I see command file drivers/infiniband/hw/mlx5/* +p while we need func mlx5_cmd_check +p to get syndrome

I am not sure :) I'll ask again.

sb22bs · 2022-01-13T07:34:40Z

Hi

here is the "sys-admin".
the dynamic debug on functions doesn't seem to work (for me).
The problem seems to got introduced in ucx-1.10.0 and the problem only
happens on x86-64 without ofed. (but maybe I'm missing something.)

aarch64 without ofed: okay
x86_64 with ofed: okay
x86_64 without ofed: ucx-1.9.0: okay
x86_64 without ofed: ucx-1.10.0: problem

Hope it helps..... Best regard, Sebastian

ofed-x86-64.txt
sys-arm64.txt
sys-x86-64.txt
ucx-1.9.0-data-with-ofed.txt
ucx-1.9.0-data-without-ofed-arm64.txt
ucx-1.9.0-data-without-ofed.txt
ucx-1.10.0-data-with-ofed.txt
ucx-1.10.0-data-without-ofed-arm64.txt
ucx-1.10.0-data-without-ofed.txt

zerothi · 2022-02-02T13:44:48Z

@Artemy-Mellanox any update here?

sb22bs · 2022-02-16T16:38:34Z

I have an update......(when not using OFED), then upgrading the mellanox-firmware solves the problem.

# diff -u between old-firmware and new-firmware of ucx_info -dvb:
diff -u ucx1120-n-62-31-18.txt ucx1120-n-62-31-20.txt
--- ucx1120-n-62-31-18.txt	2022-02-16 17:33:11.032505000 +0100
+++ ucx1120-n-62-31-20.txt	2022-02-16 17:32:58.995822000 +0100
@@ -1,20 +1,20 @@
-Linux n-62-31-18 3.10.0-1160.53.1.el7.x86_64 #1 SMP Tue Jan 11 08:25:52 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
+Linux n-62-31-20 3.10.0-1160.53.1.el7.x86_64 #1 SMP Tue Jan 11 08:25:52 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
 CA 'mlx5_0'
 	CA type: MT4115
 	Number of ports: 1
-	Firmware version: 12.23.1020
+	Firmware version: 12.28.2006
 	Hardware version: 0
-	Node GUID: 0x98039b030074c3f4
-	System image GUID: 0x98039b030074c3f4
+	Node GUID: 0x98039b030074c3d8
+	System image GUID: 0x98039b030074c3d8
 	Port 1:
 		State: Active
 		Physical state: LinkUp
 		Rate: 56
-		Base lid: 498
+		Base lid: 492
 		LMC: 0
 		SM lid: 271
 		Capability mask: 0x2659e848
-		Port GUID: 0x98039b030074c3f4
+		Port GUID: 0x98039b030074c3d8
 		Link layer: InfiniBand
 # UCT version=1.12.0 revision d367332
 # configured with: --prefix=/zhome/31/b/80425/local/ucx-1.12.0 --without-go
@@ -227,7 +227,7 @@
 #
 # Memory domain: posix
 #     Component: posix
-#             allocate: <= 197863628K
+#             allocate: <= 98780104K
 #           remote key: 24 bytes
 #           rkey_ptr is supported
 #
@@ -473,7 +473,7 @@
 #     device num paths: 1
 #              max eps: 256
 #       device address: 3 bytes
-#           ep address: 4 bytes
+#           ep address: 5 bytes
 #       error handling: peer failure, ep_check
 #
 #
@@ -502,9 +502,16 @@
 #         am_align_mtu: <= 4K
 #            am header: <= 186
 #               domain: device
-#           atomic_add: 64 bit
-#          atomic_fadd: 64 bit
-#         atomic_cswap: 64 bit
+#           atomic_add: 32, 64 bit
+#           atomic_and: 32, 64 bit
+#            atomic_or: 32, 64 bit
+#           atomic_xor: 32, 64 bit
+#          atomic_fadd: 32, 64 bit
+#          atomic_fand: 32, 64 bit
+#           atomic_for: 32, 64 bit
+#          atomic_fxor: 32, 64 bit
+#          atomic_swap: 32, 64 bit
+#         atomic_cswap: 32, 64 bit
 #           connection: to ep
 #      device priority: 30
 #     device num paths: 1
@@ -518,8 +525,45 @@
 #         Device: mlx5_0:1
 #           Type: network
 #  System device: mlx5_0 (0)
-[1645029191.774016] [n-62-31-18:237757:0]         dc_mlx5.c:517  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument
-#   < failed to open interface >
+#
+#      capabilities:
+#            bandwidth: 6433.22/ppn + 0.00 MB/sec
+#              latency: 760 nsec
+#             overhead: 40 nsec
+#            put_short: <= 172
+#            put_bcopy: <= 8256
+#            put_zcopy: <= 1G, up to 11 iov
+#  put_opt_zcopy_align: <= 512
+#        put_align_mtu: <= 4K
+#            get_bcopy: <= 8256
+#            get_zcopy: 65..1G, up to 11 iov
+#  get_opt_zcopy_align: <= 512
+#        get_align_mtu: <= 4K
+#             am_short: <= 186
+#             am_bcopy: <= 8254
+#             am_zcopy: <= 8254, up to 3 iov
+#   am_opt_zcopy_align: <= 512
+#         am_align_mtu: <= 4K
+#            am header: <= 138
+#               domain: device
+#           atomic_add: 32, 64 bit
+#           atomic_and: 32, 64 bit
+#            atomic_or: 32, 64 bit
+#           atomic_xor: 32, 64 bit
+#          atomic_fadd: 32, 64 bit
+#          atomic_fand: 32, 64 bit
+#           atomic_for: 32, 64 bit
+#          atomic_fxor: 32, 64 bit
+#          atomic_swap: 32, 64 bit
+#         atomic_cswap: 32, 64 bit
+#           connection: to iface
+#      device priority: 30
+#     device num paths: 1
+#              max eps: inf
+#       device address: 3 bytes
+#        iface address: 5 bytes
+#       error handling: buffer (zcopy), remote access, peer failure, ep_check
+#
 #
 #      Transport: ud_verbs
 #         Device: mlx5_0:1

Best regards, Sebastian

Artemy-Mellanox · 2022-02-22T20:51:01Z

Maybe you can attach from old kernel fw config:
sudo mlxconfig -d /dev/mst/mt4115_pciconf0 q

sb22bs · 2022-02-23T09:43:40Z

Hi

okay...here you go... - installation is without ofed and a fresh
installation of mft-4.5.0-31.x86_64

# (Linux n-62-31-24 3.10.0-1160.53.1.el7.x86_64 #1 SMP Tue Jan 11 08:25:52 CST 2022 x86_64 x86_64 x86_64 GNU/Linux) 
$ ibstat
CA 'mlx5_0'
	CA type: MT4115
	Number of ports: 1
	Firmware version: 12.23.1020
	Hardware version: 0
	Node GUID: 0x98039b030074c280
	System image GUID: 0x98039b030074c280
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 461
		LMC: 0
		SM lid: 271
		Capability mask: 0x2659e848
		Port GUID: 0x98039b030074c280
		Link layer: InfiniBand

$ mlxconfig -d /dev/mst/mt4115_pciconf0 q

Device #1:
----------

Device type:    ConnectX4
PCI device:     /dev/mst/mt4115_pciconf0

Configurations:                              Current
         NUM_OF_VFS                          8
         NUM_PFS                             1
         FPP_EN                              True(1)
         SRIOV_EN                            False(0)
         PF_LOG_BAR_SIZE                     5
         VF_LOG_BAR_SIZE                     1
         NUM_PF_MSIX                         63
         NUM_VF_MSIX                         11
         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0)
         CQE_COMPRESSION                     BALANCED(0)
         LRO_LOG_TIMEOUT0                    6
         LRO_LOG_TIMEOUT1                    7
         LRO_LOG_TIMEOUT2                    8
         LRO_LOG_TIMEOUT3                    13
         LOG_DCR_HASH_TABLE_SIZE             14
         DCR_LIFO_SIZE                       16384
         ROCE_NEXT_PROTOCOL                  254
         LLDP_NB_DCBX_P1                     False(0)
         LLDP_NB_RX_MODE_P1                  OFF(0)
         LLDP_NB_TX_MODE_P1                  OFF(0)
         CLAMP_TGT_RATE_AFTER_TIME_INC_P1    True(1)
         CLAMP_TGT_RATE_P1                   False(0)
         RPG_TIME_RESET_P1                   300
         RPG_BYTE_RESET_P1                   32767
         RPG_THRESHOLD_P1                    1
         RPG_MAX_RATE_P1                     0
         RPG_AI_RATE_P1                      5
         RPG_HAI_RATE_P1                     50
         RPG_GD_P1                           11
         RPG_MIN_DEC_FAC_P1                  50
         RPG_MIN_RATE_P1                     1
         RATE_TO_SET_ON_FIRST_CNP_P1         0
         DCE_TCP_G_P1                        1019
         DCE_TCP_RTT_P1                      1
         RATE_REDUCE_MONITOR_PERIOD_P1       4
         INITIAL_ALPHA_VALUE_P1              1023
         MIN_TIME_BETWEEN_CNPS_P1            0
         CNP_802P_PRIO_P1                    6
         CNP_DSCP_P1                         48
         LINK_TYPE_P1                        IB(1)
         KEEP_ETH_LINK_UP_P1                 True(1)
         KEEP_IB_LINK_UP_P1                  False(0)
         KEEP_LINK_UP_ON_BOOT_P1             False(0)
         KEEP_LINK_UP_ON_STANDBY_P1          False(0)
         ROCE_CC_PRIO_MASK_P1                255
         ROCE_CC_ALGORITHM_P1                ECN(0)
         DCBX_IEEE_P1                        True(1)
         DCBX_CEE_P1                         True(1)
         DCBX_WILLING_P1                     True(1)
         NUM_OF_VL_P1                        4_VLS(3)
         NUM_OF_TC_P1                        8_TCS(0)
         NUM_OF_PFC_P1                       8
         DUP_MAC_ACTION_P1                   LAST_CFG(0)
         PORT_OWNER                          True(1)
         ALLOW_RD_COUNTERS                   True(1)
         IP_VER                              IPv4(0)
         BOOT_VLAN                           1
         BOOT_VLAN_EN                        False(0)
         BOOT_OPTION_ROM_EN                  True(1)
         BOOT_PKEY                           0

Artemy-Mellanox · 2022-02-24T07:48:36Z

Can you please run failing ucx_info with strace UCX_LOG_LEVEL=debug strace -o ./strace.log -s 256 ucx_info -d
and attach ./strace.log

sb22bs · 2022-02-24T07:57:58Z

Yes....attached.
strace.log

Thanks :)

Artemy-Mellanox · 2022-02-24T08:12:33Z

Can you please send output of rpm -qi libibverbs

sb22bs · 2022-02-24T08:13:32Z

here it is:

$ rpm -qi libibverbs
Name        : libibverbs
Version     : 22.1
Release     : 3.el7
Architecture: x86_64
Install Date: Mon 22 Nov 2021 02:35:03 AM CET
Group       : Unspecified
Size        : 746964
License     : GPLv2 or BSD
Signature   : DSA/SHA1, Fri 09 Aug 2019 03:47:58 PM CEST, Key ID b0b4183f192a7d7d
Source RPM  : rdma-core-22.1-3.el7.src.rpm
Build Date  : Thu 08 Aug 2019 07:42:27 PM CEST
Build Host  : sl7.fnal.gov
Relocations : (not relocatable)
Packager    : Scientific Linux
Vendor      : Scientific Linux
URL         : https://github.com/linux-rdma/rdma-core
Summary     : A library and drivers for direct userspace use of RDMA (InfiniBand/iWARP/RoCE) hardware
Description :
libibverbs is a library that allows userspace processes to use RDMA
"verbs" as described in the InfiniBand Architecture Specification and
the RDMA Protocol Verbs Specification.  This includes direct hardware
access from userspace to InfiniBand/iWARP adapters (kernel bypass) for
fast path operations.

Device-specific plug-in ibverbs userspace drivers are included:

- libbxnt_re: Broadcom NetXtreme-E RoCE HCA
- libcxgb3: Chelsio T3 iWARP HCA
- libcxgb4: Chelsio T4 iWARP HCA
- libhfi1: Intel Omni-Path HFI
- libhns: HiSilicon Hip06 SoC
- libi40iw: Intel Ethernet Connection X722 RDMA
- libipathverbs: QLogic InfiniPath HCA
- libmlx4: Mellanox ConnectX-3 InfiniBand HCA
- libmlx5: Mellanox Connect-IB/X-4+ InfiniBand HCA
- libmthca: Mellanox InfiniBand HCA
- libnes: NetEffect RNIC
- libocrdma: Emulex OneConnect RDMA/RoCE Device
- libqedr: QLogic QL4xxx RoCE HCA
- librxe: A software implementation of the RoCE protocol
- libvmw_pvrdma: VMware paravirtual RDMA device

Artemy-Mellanox · 2022-02-24T10:23:11Z

I have an update......(when not using OFED), then upgrading the mellanox-firmware solves the problem.

Can you please verify that in this case it's still same libibverbs version and not some other, probably installed by OFED

sb22bs · 2022-02-24T11:02:27Z

This machine is and was OFED-free.

Just compiled a fresh openucx on this box....

$ ~/local/openucx-test-1.12.0-n-62-31-24/bin/ucx_info -d  | grep dc
#      Transport: dc_mlx5
[1645700161.828228] [n-62-31-24:19214:0]         dc_mlx5.c:517  UCX  ERROR mlx5dv_create_qp(DCT) failed: Invalid argument

But I would have expected, that libibverbs would show up in the ldd-output....(?)

$ ldd ~/local/openucx-test-1.12.0-n-62-31-24/bin/ucx_info
	linux-vdso.so.1 =>  (0x00007ffd4df72000)
	libucp.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libucp.so.0 (0x00007fb2e544b000)
	libuct.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libuct.so.0 (0x00007fb2e5402000)
	libucs.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libucs.so.0 (0x00007fb2e51bc000)
	libnuma.so.1 => /lib64/libnuma.so.1 (0x00007fb2e4fb1000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fb2e4d9b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fb2e4b97000)
	libucm.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libucm.so.0 (0x00007fb2e53b9000)
	libatomic.so.1 => /appl/gcc/11.2.0-binutils-2.37/lib64/libatomic.so.1 (0x00007fb2e53ae000)
	liblsf.so => /lsf/10.1/linux3.10-glibc2.17-x86_64/lib/liblsf.so (0x00007fb2e4455000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fb2e4153000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fb2e3f39000)
	libnss_nis.so.2 => /lib64/libnss_nis.so.2 (0x00007fb2e3d2d000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb2e3b11000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fb2e3909000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fb2e353b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb2e534e000)
	libgcc_s.so.1 => /appl/gcc/11.2.0-binutils-2.37/lib64/libgcc_s.so.1 (0x00007fb2e538f000)
	libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00007fb2e3328000)

# and there are no static-libibverbs-libraries installed....
$ ls -adlrt /usr/lib64/libibverbs*
-rwxr-xr-x 1 root root 105728 Aug  8  2019 /usr/lib64/libibverbs.so.1.5.22.1
lrwxrwxrwx 1 root root     22 Nov 22 02:35 /usr/lib64/libibverbs.so.1 -> libibverbs.so.1.5.22.1
drwxr-xr-x 2 root root   4096 Nov 22 02:35 /usr/lib64/libibverbs
lrwxrwxrwx 1 root root     15 Nov 22 02:51 /usr/lib64/libibverbs.so -> libibverbs.so.1


# ucx-configure-config.log-warnings:
 $ grep WARN config.log 
configure:24402: WARNING: GO support was explicitly disabled.
configure:24668: WARNING: Disabling Java support - java or mvn not in path.
configure:25095: WARNING: CUDA not found
configure:25304: WARNING: ROCm not found
configure:25442: WARNING: HIP Runtime not found
configure:26732: WARNING: GDR_COPY not found
configure:28164: WARNING: Compiling without extended atomics support
configure:28755: WARNING: RDMACM requested but librdmacm is not found or does not provide rdma_establish() API
configure:34151: WARNING: unrecognized options: --enable-backtrace-detail

Artemy-Mellanox · 2022-02-24T11:11:47Z

please try ldd ~/local/openucx-test-1.12.0-n-62-31-24/lib/ucx/libuct_ib.so

sb22bs · 2022-02-24T11:24:46Z

okay....and it looks all consistent...

hpc-node:n-62-31-24(sebo) $ ldd ~/local/openucx-test-1.12.0-n-62-31-24/lib/ucx/libuct_ib.so
	linux-vdso.so.1 =>  (0x00007ffef8b7b000)
	libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f53e2d7c000)
	libmlx5.so.1 => /lib64/libmlx5.so.1 (0x00007f53e2b54000)
	libuct.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libuct.so.0 (0x00007f53e308d000)
	libucs.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libucs.so.0 (0x00007f53e29c2000)
	libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f53e27b7000)
	libucm.so.0 => /zhome/31/b/80425/local/openucx-test-1.12.0-n-62-31-24/lib/libucm.so.0 (0x00007f53e306e000)
	libatomic.so.1 => /appl/gcc/11.2.0-binutils-2.37/lib64/libatomic.so.1 (0x00007f53e3064000)
	liblsf.so => /lsf/10.1/linux3.10-glibc2.17-x86_64/lib/liblsf.so (0x00007f53e2075000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f53e1d73000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f53e1b59000)
	libnss_nis.so.2 => /lib64/libnss_nis.so.2 (0x00007f53e194d000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f53e1737000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f53e1533000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f53e1317000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f53e110f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f53e0d41000)
	libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007f53e0ad4000)
	libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f53e08b3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f53e2f95000)
	libgcc_s.so.1 => /appl/gcc/11.2.0-binutils-2.37/lib64/libgcc_s.so.1 (0x00007f53e3043000)
	libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00007f53e06a0000)
~
hpc-node:n-62-31-24(sebo) $ rpm -V libibverbs
~
hpc-node:n-62-31-24(sebo) $ rpm -Vv libibverbs
.........    /etc/libibverbs.d
.........  c /etc/libibverbs.d/bnxt_re.driver
.........  c /etc/libibverbs.d/cxgb3.driver
.........  c /etc/libibverbs.d/cxgb4.driver
.........  c /etc/libibverbs.d/hfi1verbs.driver
.........  c /etc/libibverbs.d/hns.driver
.........  c /etc/libibverbs.d/i40iw.driver
.........  c /etc/libibverbs.d/ipathverbs.driver
.........  c /etc/libibverbs.d/mlx4.driver
.........  c /etc/libibverbs.d/mlx5.driver
.........  c /etc/libibverbs.d/mthca.driver
.........  c /etc/libibverbs.d/nes.driver
.........  c /etc/libibverbs.d/ocrdma.driver
.........  c /etc/libibverbs.d/qedr.driver
.........  c /etc/libibverbs.d/rxe.driver
.........  c /etc/libibverbs.d/vmw_pvrdma.driver
.........    /usr/bin/rxe_cfg
.........    /usr/lib64/libibverbs
.........    /usr/lib64/libibverbs.so.1
.........    /usr/lib64/libibverbs.so.1.5.22.1
.........    /usr/lib64/libibverbs/libbnxt_re-rdmav22.so
.........    /usr/lib64/libibverbs/libcxgb3-rdmav22.so
.........    /usr/lib64/libibverbs/libcxgb4-rdmav22.so
.........    /usr/lib64/libibverbs/libhfi1verbs-rdmav22.so
.........    /usr/lib64/libibverbs/libhns-rdmav22.so
.........    /usr/lib64/libibverbs/libi40iw-rdmav22.so
.........    /usr/lib64/libibverbs/libipathverbs-rdmav22.so
.........    /usr/lib64/libibverbs/libmlx4-rdmav22.so
.........    /usr/lib64/libibverbs/libmlx5-rdmav22.so
.........    /usr/lib64/libibverbs/libmthca-rdmav22.so
.........    /usr/lib64/libibverbs/libnes-rdmav22.so
.........    /usr/lib64/libibverbs/libocrdma-rdmav22.so
.........    /usr/lib64/libibverbs/libqedr-rdmav22.so
.........    /usr/lib64/libibverbs/librxe-rdmav22.so
.........    /usr/lib64/libibverbs/libvmw_pvrdma-rdmav22.so
.........    /usr/lib64/libmlx4.so.1
.........    /usr/lib64/libmlx4.so.1.0.22.1
.........    /usr/lib64/libmlx5.so.1
.........    /usr/lib64/libmlx5.so.1.8.22.1
.........  d /usr/share/doc/rdma-core-22.1/libibverbs.md
.........  d /usr/share/doc/rdma-core-22.1/rxe.md
.........  d /usr/share/doc/rdma-core-22.1/tag_matching.md
.........  d /usr/share/man/man7/mlx4dv.7.gz
.........  d /usr/share/man/man7/mlx5dv.7.gz
.........  d /usr/share/man/man7/rxe.7.gz
.........  d /usr/share/man/man8/rxe_cfg.8.gz
~
hpc-node:n-62-31-24(sebo) $ rpm -qi libibverbs
Name        : libibverbs
Version     : 22.1
Release     : 3.el7
Architecture: x86_64
Install Date: Mon 22 Nov 2021 02:35:03 AM CET
Group       : Unspecified
Size        : 746964
License     : GPLv2 or BSD
Signature   : DSA/SHA1, Fri 09 Aug 2019 03:47:58 PM CEST, Key ID b0b4183f192a7d7d
Source RPM  : rdma-core-22.1-3.el7.src.rpm
Build Date  : Thu 08 Aug 2019 07:42:27 PM CEST
Build Host  : sl7.fnal.gov
Relocations : (not relocatable)
Packager    : Scientific Linux
Vendor      : Scientific Linux
URL         : https://github.com/linux-rdma/rdma-core
Summary     : A library and drivers for direct userspace use of RDMA (InfiniBand/iWARP/RoCE) hardware
Description :
libibverbs is a library that allows userspace processes to use RDMA
"verbs" as described in the InfiniBand Architecture Specification and
the RDMA Protocol Verbs Specification.  This includes direct hardware
access from userspace to InfiniBand/iWARP adapters (kernel bypass) for
fast path operations.

Device-specific plug-in ibverbs userspace drivers are included:

- libbxnt_re: Broadcom NetXtreme-E RoCE HCA
- libcxgb3: Chelsio T3 iWARP HCA
- libcxgb4: Chelsio T4 iWARP HCA
- libhfi1: Intel Omni-Path HFI
- libhns: HiSilicon Hip06 SoC
- libi40iw: Intel Ethernet Connection X722 RDMA
- libipathverbs: QLogic InfiniPath HCA
- libmlx4: Mellanox ConnectX-3 InfiniBand HCA
- libmlx5: Mellanox Connect-IB/X-4+ InfiniBand HCA
- libmthca: Mellanox InfiniBand HCA
- libnes: NetEffect RNIC
- libocrdma: Emulex OneConnect RDMA/RoCE Device
- libqedr: QLogic QL4xxx RoCE HCA
- librxe: A software implementation of the RoCE protocol
- libvmw_pvrdma: VMware paravirtual RDMA device

$ ls -adlrt /lib64
lrwxrwxrwx. 1 root root 9 Nov 22 02:16 /lib64 -> usr/lib64

 $ ls -arldt /lib64/libibverbs*
-rwxr-xr-x 1 root root 105728 Aug  8  2019 /lib64/libibverbs.so.1.5.22.1
lrwxrwxrwx 1 root root     22 Nov 22 02:35 /lib64/libibverbs.so.1 -> libibverbs.so.1.5.22.1
drwxr-xr-x 2 root root   4096 Nov 22 02:35 /lib64/libibverbs
lrwxrwxrwx 1 root root     15 Nov 22 02:51 /lib64/libibverbs.so -> libibverbs.so.1

Artemy-Mellanox · 2022-02-28T12:48:00Z

Looks like this libibverbs version has bug, could you please update it, version 22.4-2 or later should have a fix to this.

sb22bs · 2022-02-28T18:02:59Z

Yes.... thanks a lot...confirmed....

using the updating the infiniband-packages which are coming with Scientific-Linux-7.9 are
fixing the issue....:

$ rpm -Fvh *.rpm                                                                                                                                                                  
Preparing...                          ################################# [100%]                                                                                                                              
Updating / installing...                                                                                                                                                                                    
   1:rdma-core-22.4-5.el7             ################################# [  6%]                                                                                                                              
   2:libibverbs-22.4-5.el7            ################################# [ 13%]                                                                                                                              
   3:librdmacm-22.4-5.el7             ################################# [ 19%]                                                                                                                              
   4:libibumad-22.4-5.el7             ################################# [ 25%]                                                                                                                              
   5:ibacm-22.4-5.el7                 ################################# [ 31%]                                                                                                                              
   6:rdma-core-devel-22.4-5.el7       ################################# [ 38%]                                                                                                                              
   7:librdmacm-utils-22.4-5.el7       ################################# [ 44%]                                                                                                                              
   8:libibverbs-utils-22.4-5.el7      ################################# [ 50%]                                                                                                                              
Cleaning up / removing...                                                                                                                                                                                   
   9:rdma-core-devel-22.1-3.el7       ################################# [ 56%]                                                                                                                              
  10:ibacm-22.1-3.el7                 ################################# [ 63%]                                                                                                                              
  11:librdmacm-utils-22.1-3.el7       ################################# [ 69%]                                                                                                                              
  12:librdmacm-22.1-3.el7             ################################# [ 75%]                                                                                                                              
  13:libibumad-22.1-3.el7             ################################# [ 81%]                                                                                                                              
  14:libibverbs-utils-22.1-3.el7      ################################# [ 88%]                                                                                                                              
  15:libibverbs-22.1-3.el7            ################################# [ 94%]                                                                                                                              
  16:rdma-core-22.1-3.el7             ################################# [100%]            

$ ~/local/openucx-test-1.12.0-n-62-31-24/bin/ucx_info -dvb
[...]
#      Transport: dc_mlx5
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
#
#      capabilities:
#            bandwidth: 6433.22/ppn + 0.00 MB/sec
#              latency: 760 nsec
#             overhead: 40 nsec
#            put_short: <= 172
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 11 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 11 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 186
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 138
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to iface
#      device priority: 30
#     device num paths: 1
#              max eps: inf
#       device address: 3 bytes
#        iface address: 5 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
[...]

 $ /usr/sbin/ibstat
CA 'mlx5_0'
	CA type: MT4115
	Number of ports: 1
	Firmware version: 12.23.1020
	Hardware version: 0
	Node GUID: 0x98039b030074c280
	System image GUID: 0x98039b030074c280
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 461
		LMC: 0
		SM lid: 271
		Capability mask: 0x2659e848
		Port GUID: 0x98039b030074c280
		Link layer: InfiniBand

Case closed.... (at least from our side...)

Thanks again + Best regards, Sebastian

lyu added the Bug label Sep 29, 2020

yosefe added the External label Sep 29, 2020

yosefe pinned this issue Sep 29, 2020

Artemy-Mellanox mentioned this issue Mar 7, 2022

UCP/EP: Add eager multi-fragment overhead - v1.12.x #8014

Merged

Artemy-Mellanox closed this as completed Mar 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in mlx5dv_create_qp in the DC transport #5749

Error in mlx5dv_create_qp in the DC transport #5749

lyu commented Sep 29, 2020

yosefe commented Sep 29, 2020

lyu commented Sep 29, 2020

tonycurtis commented Sep 30, 2020

alex--m commented Aug 21, 2021 •

edited

Loading

yosefe commented Aug 21, 2021

alex--m commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

zerothi commented Jan 5, 2022 •

edited

Loading

Artemy-Mellanox commented Jan 5, 2022

zerothi commented Jan 5, 2022

zerothi commented Jan 10, 2022

Artemy-Mellanox commented Jan 11, 2022

zerothi commented Jan 11, 2022

sb22bs commented Jan 13, 2022

zerothi commented Feb 2, 2022

sb22bs commented Feb 16, 2022 •

edited

Loading

Artemy-Mellanox commented Feb 22, 2022

sb22bs commented Feb 23, 2022 •

edited

Loading

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 28, 2022

sb22bs commented Feb 28, 2022

Error in mlx5dv_create_qp in the DC transport #5749

Error in mlx5dv_create_qp in the DC transport #5749

Comments

lyu commented Sep 29, 2020

Describe the bug

Steps to Reproduce

Setup and versions

Additional information (depending on the issue)

yosefe commented Sep 29, 2020

lyu commented Sep 29, 2020

tonycurtis commented Sep 30, 2020

alex--m commented Aug 21, 2021 • edited Loading

yosefe commented Aug 21, 2021

alex--m commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

yosefe commented Aug 22, 2021

alex--m commented Aug 22, 2021

zerothi commented Jan 5, 2022 • edited Loading

Artemy-Mellanox commented Jan 5, 2022

zerothi commented Jan 5, 2022

zerothi commented Jan 10, 2022

Artemy-Mellanox commented Jan 11, 2022

zerothi commented Jan 11, 2022

sb22bs commented Jan 13, 2022

zerothi commented Feb 2, 2022

sb22bs commented Feb 16, 2022 • edited Loading

Artemy-Mellanox commented Feb 22, 2022

sb22bs commented Feb 23, 2022 • edited Loading

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 24, 2022

sb22bs commented Feb 24, 2022

Artemy-Mellanox commented Feb 28, 2022

sb22bs commented Feb 28, 2022

alex--m commented Aug 21, 2021 •

edited

Loading

zerothi commented Jan 5, 2022 •

edited

Loading

sb22bs commented Feb 16, 2022 •

edited

Loading

sb22bs commented Feb 23, 2022 •

edited

Loading