Use memory nodes.

Understand what a memory node is.

A machine with one processor package has one pool of memory. Every processor reaches that memory at the same cost, and where an object lives does not matter.

A machine with several packages usually attaches memory to each one. A package and the memory attached to it form a memory node. A thread reaches memory on its own node faster than memory on another node, because the second request crosses the link between packages.

The host firmware declares a distance for each pair of nodes. Ten names a node's distance to itself. A larger value orders one pair against another. It is not a latency, and firmware often declares it poorly.

Placement is worth doing when a program's working set is large enough for this difference to show in its running time. A program that fits in cache, or one that allocates little, does not gain from it.

This crate answers three separate questions. The root package reports which nodes the host has. One child package places memory on a chosen node. Another supplies a storage pool that allocates from one.

Add the standalone crate.

Add the Flyology organization index before the community index. Then add flyology_numa as a normal Alire dependency.

alr index --reset-community
alr index --add=git+https://github.com/flyology-ada/alire-index.git \
  --name=flyology --before=community
alr with flyology_numa
alr build

The crate accepts GNAT 13 through 16. Its Alire manifest makes it available only on Linux and macOS.

The dependency does not prepare a custom runtime. It also does not depend on the flyology crate.

Read the node structure.

The crate reads the host description once, while the package elaborates. Later queries return that result. The crate does not observe a node that appears or disappears afterwards.

Online_Nodes reports the nodes the host has. Processors_Of reports the processors attached to one node, and Node_Of reports the node a processor belongs to.

with Ada.Text_IO;
with Flyology_NUMA;

procedure Show_Nodes is
   package NUMA renames Flyology_NUMA;
begin
   for Node of NUMA.Online_Nodes loop
      Ada.Text_IO.Put_Line
        ("node"   & NUMA.Node_Id'Image (Node)
         & " has" & Natural'Image (NUMA.Count (NUMA.Processors_Of (Node)))
         & " processors");
   end loop;
end Show_Nodes;

A Node_Id is the host's own node number. It matches the number in a sysfs path and in numactl output.

Use the permitted nodes.

A process can be permitted fewer nodes than the host has. A container shows the whole machine's node list, and a control group restricts which of those nodes the process may allocate on. The host description alone therefore does not establish permission.

Allowed_Nodes reports the nodes this process may use. Use that set to choose a node. Online_Nodes answers what the machine has, which is a different question.

Support returns a Support_Report. Its Restricted component is true when the two sets differ. Its Complete component is false when the crate could not read or represent part of the description.

Handle one memory domain.

Every query answers on every supported host. A host without memory-node structure reports one node that holds every processor. That report is accurate, because such a host has one memory domain.

The Source component of Support_Report is a Discovery_Source. It separates a description read from the host from that single-node report.

macOS is always the single-node case. It describes no memory-node structure. Its performance and efficiency processor clusters are not memory nodes, because those clusters share one memory domain. The crate does not report them.

Place memory on a node.

Flyology_NUMA.Placement acts on the reported structure. Reading a description always succeeds, and acting on one often does not, so the two are separate packages.

Check Support before you place anything. It returns a Support_Level with three values.

Keep Denied separate from Unsupported_Host. A container sandbox commonly refuses these calls while the machine underneath has memory nodes that other processes use. Reporting that refusal as a missing facility would be wrong.

Apply_To places one address range. Placement acts on whole pages, so the range must begin on a Page_Size boundary.

Flyology_NUMA.Placement.Apply_To
  (Base   => Region_Base,
   Length => Region_Length,
   Policy => Flyology_NUMA.Placement.Interleaved,
   Nodes  => Flyology_NUMA.Allowed_Nodes,
   Result => Outcome);

The Policy_Kind value selects how pages are drawn. Bound draws only from the named nodes and fails instead of drawing elsewhere. Interleaved draws from the nodes in rotation, which trades a near node's latency for the combined transfer rate of several. Preferred draws from elsewhere instead of failing, and the host uses only the lowest-numbered node in the set.

Every operation returns a Placement_Outcome. No operation in this package raises an exception.

Apply_To_Thread places the memory the calling thread acquires from then on. Permitted_Nodes asks the host again, which matters after the process moves between control groups.

Allocate from a node.

Ada already names where an object's memory comes from: the storage pool of its access type. Flyology_NUMA.Pools supplies a pool whose subpools are memory nodes.

A pool alone would fix the node when the access type is declared. A subpool is chosen for each allocation, so the node is named where the object is created.

with Flyology_NUMA.Placement;
with Flyology_NUMA.Pools;

procedure Allocate_Near is
   package NUMA renames Flyology_NUMA;

   Arena : NUMA.Pools.Node_Pool
     (Policy => NUMA.Placement.Bound,
      Extent => 1024 * 1024);

   type Sample_Access is access Sample with Storage_Pool => Arena;

   Item : constant Sample_Access :=
     new (NUMA.Pools.On_Node (Arena, Near)) Sample'(...);
begin
   null;
end Allocate_Near;

On_Node returns the subpool that draws from one node. The Node_Pool discriminants set the policy and how much memory the pool obtains at a time. Its policy must be a Binding_Policy, because the two policies that name no node would place nothing on the node a subpool stands for.

The pool obtains whole pages from the host and places them before it hands out any storage. Memory from the ordinary heap cannot serve this. Such memory can begin partway into a page whose remainder belongs to something else, and placing that page would move memory the crate does not own.

Allocation advances through pages the pool already holds. Freeing one object does nothing. Memory returns to the host when the subpool is deallocated or the pool is finalized. This suits memory that an application builds up and discards together.

Placement_Reached reports whether the host accepted the placement of every page. It is false on a host that cannot place memory, and allocation still succeeds there. Reserved_Bytes reports how much memory the pool holds for one node.

Interpret distances and packages.

Node_Distance reports the distance the host firmware declares between two nodes. Ten names a node's distance to itself. The values order node pairs. They are not latencies, and firmware often declares them poorly.

A host can divide one processor package into several memory nodes. It then declares a small non-local distance, such as 11, between two nodes of one package. A value above ten therefore does not mean a different package. Compare Package_Of when package identity is what matters.

A node can carry memory and have no processor attached. A memory expander produces this, and so does a high-bandwidth memory tier. Has_Processors reports false for such a node. The node is a legal target for memory, and it is never local to any thread.

Memory_Bytes reports the memory one node carries. It returns a Byte_Query. Check Available before you read Bytes. The Value_Or functions return a caller's own fallback for an unanswered query. A fallback does not become a detected result.

Ada numbers processors from one, and a host numbers them from zero. To_CPU applies that mapping. It reports Not_A_Specific_CPU when the host's processor numbering has a gap, because the arithmetic would then name a processor the caller did not intend.

Test the crate.

The crate test script checks the description reported for the host that runs it. It also reads recorded host descriptions, which cover node numbering, distance rows, control-group restriction, and the descriptions the reader must refuse.

./flyology_numa/scripts/test.sh
./flyology_numa/scripts/docs.sh

Recorded descriptions cover reading. They cannot cover placing, which needs a kernel that has several nodes. A separate script boots guests with two and four memory nodes and runs the host suite inside them.

./flyology_numa/scripts/multinode-check.sh

That script needs a Linux host, a kernel image, and qemu. It is separate from the crate test script for that reason.

Use the generated numa API reference for exact declarations. The crate's README and source contain the recorded descriptions and the guest topologies.