numactl.8

.\" t
.\" Copyright 2003,2004 Andi Kleen, SuSE Labs.
.\"
.\" Permission is granted to make and distribute verbatim copies of this
.\" manual provided the copyright notice and this permission notice are
.\" preserved on all copies.
.\"
.\" Permission is granted to copy and distribute modified versions of this
.\" manual under the conditions for verbatim copying, provided that the
.\" entire resulting derived work is distributed under the terms of a
.\" permission notice identical to this one.
.\" 
.\" Since the Linux kernel and libraries are constantly changing, this
.\" manual page may be incorrect or out-of-date.  The author(s) assume no
.\" responsibility for errors or omissions, or for damages resulting from
.\" the use of the information contained herein.  
.\" 
.\" Formatted or processed versions of this manual, if unaccompanied by
.\" the source, must acknowledge the copyright and authors of this work.
.TH NUMACTL 8 "Mar 2004" "SuSE Labs" "Linux Administrator's Manual"
.SH NAME
numactl \- Control NUMA policy for processes or shared memory 
.SH SYNOPSIS
.B numactl
[
.B \-\-all
] [
.B \-\-balancing
] [
.B \-\-interleave nodes
] [
.B \-\-interleave-weight weight
] [
.B \-\-preferred node 
] [
.B \-\-preferred-many nodes
] [
.B \-\-membind nodes
] [ 
.B \-\-cpunodebind nodes
] [
.B \-\-physcpubind cpus
] [
.B \-\-localalloc
] [\-\-] command {arguments ...}
.br
.B numactl \-\-show
.br
.B numactl \-\-hardware
.br
.B numactl 
[
.B \-\-huge
] [
.B \-\-offset offset
] [
.B \-\-shmmode shmmode
] [
.B \-\-length length
] [
.B \-\-strict
]
.br
[
.B \-\-shmid id
]
.B \-\-shm shmkeyfile
|
.B \-\-file tmpfsfile
.br
[
.B \-\-touch
] [
.B \-\-dump
] [
.B \-\-dump-nodes
]
memory policy
.SH DESCRIPTION
.B numactl
runs processes with a specific NUMA scheduling or memory placement policy.
The policy is set for command and inherited by all of its children.
In addition it can set persistent policy for shared memory segments or files.
.PP
Use -- before command if using command options that could be confused
with numactl options.
.PP
.I nodes
may be specified as N,N,N or  N-N or N,N-N or  N-N,N-N and so forth.
Relative
.I nodes
may be specified as +N,N,N or  +N-N or +N,N-N and so forth. The + indicates that
the node numbers are relative to the process' set of allowed nodes in its
current cpuset.
A !N-N notation indicates the inverse of N-N, in other words all nodes
except N-N.  If used with + notation, specify !+N-N. When
.I same
is specified the previous nodemask specified on the command line is used.
all means all nodes in the current cpuset.
.PP
Instead of a number a node can also be:
.TS
tab(|);
l l.
netdev:DEV|The node connected to network device DEV.
file:PATH |The node the block device of PATH.
ip:HOST   |The node of the network device of HOST
block:PATH|The node of block device PATH
pci:[seg:]bus:dev[:func]|The node of a PCI device.
.TE

Note that block resolves the kernel block device names only
for udev names in /dev use
.I file:
.PP
.I weight
is either a string "bwa" or a comma delimited list of node and its weight
value, represented in node_id*weight format (e.g., "N*a,M*b").
.TP
Policy settings are:
.TP
.B \-\-all, \-a
Unset default cpuset awareness, so user can use all possible CPUs/nodes
for following policy settings.
.TP
.B \-\-interleave=nodes, \-i nodes
Set a memory interleave policy. Memory will be allocated using round robin
on
.I nodes.
When memory cannot be allocated on the current interleave target fall back
to other nodes.
Multiple nodes may be specified on --interleave, --membind and --cpunodebind.
.TP
.B \-\-interleave-weight=weight, \-w weight
Set a weighted memory interleave policy. Memory will be allocated according to the
ratio represented in
.I weight.
When
.I weight
is a string "bwa", bandwidth-aware interleaving will be applied. Read
.B FILES
section for more information. Otherwise, users can set their own interleave
weight as described above. When memory cannot be allocated on the current
interleave target fall back to other nodes.
Multiple nodes and their weights can be specified at
.I weight.
.TP
.B \-\-membind=nodes, \-m nodes
Only allocate memory from nodes.  Allocation will fail when there
is not enough memory available on these nodes.
.I nodes
may be specified as noted above.
.TP
.B \-\-cpunodebind=nodes, \-N nodes
Only execute
.I command
on the CPUs of
.I nodes. 
Note that nodes may consist of multiple CPUs.
.I nodes
may be specified as noted above.
.TP
.B \-\-physcpubind=cpus, \-C cpus
Only execute
.I process
on
.I cpus.
This accepts cpu numbers as shown in the
.I processor
fields of 
.I /proc/cpuinfo,
or relative cpus as in relative to the current cpuset.
You may specify "all", which means all cpus in the current cpuset.
Physical
.I cpus
may be specified as N,N,N or  N-N or N,N-N or  N-N,N-N and so forth.
Relative
.I cpus
may be specified as +N,N,N or  +N-N or +N,N-N and so forth. The + indicates that
the cpu numbers are relative to the process' set of allowed cpus in its
current cpuset.
A !N-N notation indicates the inverse of N-N, in other words all cpus
except N-N.  If used with + notation, specify !+N-N.
.TP
.B \-\-localalloc, \-l 
Try to allocate on the current node of the process, but if memory cannot be allocated there fall back to other nodes.
.TP
.B \-\-preferred=node
Preferably allocate memory on 
.I node,
but if memory cannot be allocated there fall back to other nodes.
This option takes only a single node number.
Relative notation may be used.
.TP
.B \-\-balancing, \-b
Enable Linux kernel NUMA balancing for the process if it is supported by kernel.
This should only be used with
.I \-\-membind, \-m
only, otherwise ignored.
.TP
.B \-\-preferred-many=node
Preferably allocate memory on
.I nodes,
but if memory cannot be allocated there fall back to other nodes.
This option takes a mask of preferred nodes where the closest node to local is
considered most preferred.
Relative notation may be used.
.TP
.B \-\-show, \-s
Show NUMA policy settings of the current process. 
.TP
.B \-\-hardware, \-H
Show inventory of available nodes on the system.
.TP 0
Numactl can set up policy for a SYSV shared memory segment or a file in shmfs/hugetlbfs.
 
This policy is persistent and will be used by
all mappings from that shared memory. The order of options matters here.
The specification must at least include either of 
.I \-\-shm, 
.I \-\-shmid, 
.I \-\-file
to specify the shared memory segment or file and a memory policy like described 
above (
.I \-\-interleave, 
.I \-\-interleave-weight,
.I \-\-localalloc, 
.I \-\-preferred,
.I \-\-preferred-many,
.I \-\-membind
).
.TP
.B \-\-huge
When creating a SYSV shared memory segment use huge pages.
Only valid before \-\-shmid or \-\-shm
.TP 
.B \-\-offset
Specify offset into the shared memory segment. Default 0. 
Valid units are 
.I m
(for MB), 
.I g 
(for GB), 
.I k 
(for KB),
otherwise it specifies bytes.
.TP
.B \-\-strict
Give an error when a page in the policied area in the shared memory
segment already was faulted in with a conflicting policy. Default
is to silently ignore this.
.TP
.B \-\-shmmode shmmode
Only valid before \-\-shmid or \-\-shm
When creating a shared memory segment set it to numeric mode 
.I shmmode.
.TP
.B \-\-length length
Apply policy to 
.I length 
range in the shared memory segment or make 
the segment length long
Default is to use the remaining length 
Required when a shared memory segment is created and specifies the length
of the new segment then. Valid units are 
.I m
(for MB), 
.I g 
(for GB), 
.I k 
(for KB),
otherwise it specifies bytes.
.TP
.B \-\-shmid id
Create or use a shared memory segment with numeric ID
.I id
.TP 
.B \-\-shm shmkeyfile
Create or use a shared memory segment, with the ID generated
using 
.I ftok(3) 
from shmkeyfile
.TP
.B \-\-file tmpfsfile
Set policy for a file in tmpfs or hugetlbfs
.TP
.B \-\-touch
Touch pages to enforce policy early. Default is to not touch them, the policy
is applied when an applications maps and accesses a page.
.TP
.B \-\-dump
Dump policy in the specified range.
.TP
.B \-\-dump-nodes
Dump all nodes of the specific range (very verbose!)
.TP
Valid node specifiers
.TS
tab(:);
l l. 
all:All nodes
number:Node number
number1{,number2}:Node number1 and Node number2
number1-number2:Nodes from number1 to number2
! nodes:Invert selection of the following specification.
.TE
.SH EXAMPLES
numactl \-\-physcpubind=+0-4,8-12 myapplic arguments
Run myapplic on cpus 0-4 and 8-12 of the current cpuset.

numactl \-\-interleave=all bigdatabase arguments
Run big database with its memory interleaved on all CPUs.

numactl \-\-cpunodebind=0 \-\-membind=0,1 process
Run process on node 0 with memory allocated on node 0 and 1.

numactl \-\-cpunodebind=0 \-\-membind=0,1 -- process -l
Run process as above, but with an option (-l) that would be confused with
a numactl option.

numactl \-\-cpunodebind=0 \-\-balancing \-\-membind=0,1 process
Run process on node 0 with memory allocated on node 0 and 1.  Optimize the
page placement with Linux kernel NUMA balancing mechanism if possible.

numactl \-\-cpunodebind=netdev:eth0 \-\-membind=netdev:eth0 network-server
Run network-server on the node of network device eth0 with its memory
also in the same node.

numactl \-\-preferred=1 numactl \-\-show
Set preferred node 1 and show the resulting state.

numactl \-\-preferred-many=0x3 numactl \-\-show
Set preferred nodes 1 and 2, and show the resulting state.

numactl --interleave=all --shm /tmp/shmkey 
Interleave all of the sysv shared memory region specified by
/tmp/shmkey over all nodes.

numactl --interleave-weight=bwa process Run process with its memory applied by
bandwidth-aware interleaving.

numactl --interleave-weight="0*2,1*1" process Run process with its memory using
node0 and node1 with 2:1 ratio.

Place a tmpfs file on 2 nodes:
  numactl --membind=2 dd if=/dev/zero of=/dev/shm/A bs=1M count=1024
  numactl --membind=3 dd if=/dev/zero of=/dev/shm/A seek=1024 bs=1M count=1024


numactl --localalloc /dev/shm/file
Reset the policy for the shared memory file 
.I file
to the default localalloc policy.
.SH NOTES
Requires a NUMA policy aware kernel.

Command is not executed using a shell. If you want to use shell metacharacters
in the child use sh -c as wrapper.

Setting policy for a hugetlbfs file does currently not work because
it cannot be extended by truncate.

Shared memory segments larger than numactl's address space cannot 
be completely policied. This could be a problem on 32bit architectures.
Changing it piece by piece may work.

The old
.I --cpubind
which accepts node numbers, not cpu numbers, is deprecated
and replaced with the new 
.I --cpunodebind
and 
.I --physcpubind
options.

.SH FILES
.I /proc/cpuinfo
for the listing of active CPUs. See 
.I proc(5)
for details.

.I /sys/devices/system/node/node*/numastat
for NUMA memory hit statistics.

.I /sys/kernel/mm/interleave_weight/enabled
should be set as true to use --interleave-weight=bwa.

.I /sys/kernel/mm/interleave_weight/node/node*/interleave_weight
to set bandwidth-aware interleaving.

.SH COPYRIGHT
Copyright 2002,2004 Andi Kleen, SuSE Labs.
numactl and the demo programs are under the GNU General Public License, v.2

.SH SEE ALSO
.I set_mempolicy(2)
,
.I get_mempolicy(2)
,
.I mbind(2)
,
.I sched_setaffinity(2)
, 
.I sched_getaffinity(2)
,
.I proc(5)
, 
.I ftok(3)
,
.I shmat(2)
,
.I migratepages(8)