RDMA Controller¶
1. Overview¶
1-1. What is RDMA controller?¶
RDMA controller allows user to limit RDMA/IB specific resources that a given set of processes can use. These processes are grouped using RDMA controller.
RDMA controller defines two resources which can be limited for processes of a cgroup.
1-2. Why RDMA controller needed?¶
Currently user space applications can easily take away all the rdma verb specific resources such as AH, CQ, QP, MR etc. Due to which other applications in other cgroup or kernel space ULPs may not even get chance to allocate any rdma resources. This can lead to service unavailability.
Therefore RDMA controller is needed through which resource consumption of processes can be limited. Through this controller different rdma resources can be accounted.
1-3. How is RDMA controller implemented?¶
RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains resource accounting per cgroup, per device using resource pool structure. Each such resource pool is limited up to 64 resources in given resource pool by rdma cgroup, which can be extended later if required.
This resource pool object is linked to the cgroup css. Typically there are 0 to 4 resource pool instances per cgroup, per device in most use cases. But nothing limits to have it more. At present hundreds of RDMA devices per single cgroup may not be handled optimally, however there is no known use case or requirement for such configuration either.
Since RDMA resources can be allocated from any process and can be freed by any of the child processes which shares the address space, rdma resources are always owned by the creator cgroup css. This allows process migration from one to other cgroup without major complexity of transferring resource ownership; because such ownership is not really present due to shared nature of rdma resources. Linking resources around css also ensures that cgroups can be deleted after processes migrated. This allow progress migration as well with active resources, even though that is not a primary use case.
Whenever RDMA resource charging occurs, owner rdma cgroup is returned to the caller. Same rdma cgroup should be passed while uncharging the resource. This also allows process migrated with active RDMA resource to charge to new owner cgroup for new resource. It also allows to uncharge resource of a process from previously charged cgroup which is migrated to new cgroup, even though that is not a primary use case.
Resource pool object is created in following situations. (a) User sets the limit and no previous resource pool exist for the device of interest for the cgroup. (b) No resource limits were configured, but IB/RDMA stack tries to charge the resource. So that it correctly uncharge them when applications are running without limits and later on when limits are enforced during uncharging, otherwise usage count will drop to negative.
Resource pool is destroyed if all the resource limits are set to max and it is the last resource getting deallocated.
User should set all the limit to max value if it intents to remove/unconfigure the resource pool for a particular device.
IB stack honors limits enforced by the rdma controller. When application query about maximum resource limits of IB device, it returns minimum of what is configured by user for a given cgroup and what is supported by IB device.
Following resources can be accounted by rdma controller.
hca_handle
Maximum number of HCA Handles
hca_object
Maximum number of HCA Objects
2. Usage Examples¶
Configure resource limit:
echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max
Query resource limit:
cat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max
Query current usage:
cat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23
Delete resource limit:
echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max
3. RDMA Interface Files¶
The following interface files are available in each non-root RDMA cgroup.
- rdma.max
A read-write file which describes the configured resource limit for an RDMA/IB device. See the Usage Examples above.
- rdma.current
A read-only file which describes the current resource usage.
- rdma.peak
A read-only nested-keyed file which shows the historical high watermark of resource usage per device since the cgroup was created.
An example for mlx4 and ocrdma device follows:
mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=0 hca_object=23- rdma.events
A read-only nested-keyed file which exists on non-root cgroups and contains the following keys:
- max
The number of times a process in this cgroup or its descendants attempted an RDMA resource allocation that was rejected because a rdma.max limit in the subtree was reached. This is a hierarchical counter propagated upward to all ancestor cgroups. A value change in this file generates a file modified event.
- alloc_fail
The number of RDMA resource allocation attempts that originated in this cgroup or its descendants and failed due to a rdma.max limit being reached. This is a hierarchical counter propagated upward.
An example for mlx4 device follows:
mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0- rdma.events.local
Similar to rdma.events but the fields are local to the cgroup, i.e. not hierarchical. The file modified event generated on this file reflects only the local events.
The following nested keys are defined.
- max
The number of times a process in this cgroup or its descendants attempted an RDMA resource allocation that was rejected because this cgroup’s own rdma.max limit was reached.
- alloc_fail
The number of RDMA resource allocation attempts originating from this cgroup that failed due to this cgroup’s or an ancestor’s rdma.max limit.
An example for mlx4 device follows:
mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0