BPF Iterators

Motivation

There are a few existing ways to dump kernel data into user space. The most popular one is the /proc system. For example, cat /proc/net/tcp6 dumps all tcp6 sockets in the system, and cat /proc/net/netlink dumps all netlink sockets in the system. However, their output format tends to be fixed, and if users want more information about these sockets, they have to patch the kernel, which often takes time to publish upstream and release. The same is true for popular tools like ss where any additional information needs a kernel patch.

To solve this problem, the drgn tool is often used to dig out the kernel data with no kernel change. However, the main drawback for drgn is performance, as it cannot do pointer tracing inside the kernel. In addition, drgn cannot validate a pointer value and may read invalid data if the pointer becomes invalid inside the kernel.

The BPF iterator solves the above problem by providing flexibility on what data (e.g., tasks, bpf_maps, etc.) to collect by calling BPF programs for each kernel data object.

How BPF Iterators Work

A BPF iterator is a type of BPF program that allows users to iterate over specific types of kernel objects. Unlike traditional BPF tracing programs that allow users to define callbacks that are invoked at particular points of execution in the kernel, BPF iterators allow users to define callbacks that should be executed for every entry in a variety of kernel data structures.

For example, users can define a BPF iterator that iterates over every task on the system and dumps the total amount of CPU runtime currently used by each of them. Another BPF task iterator may instead dump the cgroup information for each task. Such flexibility is the core value of BPF iterators.

A BPF program is always loaded into the kernel at the behest of a user space process. A user space process loads a BPF program by opening and initializing the program skeleton as required and then invoking a syscall to have the BPF program verified and loaded by the kernel.

In traditional tracing programs, a program is activated by having user space obtain a bpf_link to the program with bpf_program__attach(). Once activated, the program callback will be invoked whenever the tracepoint is triggered in the main kernel. For BPF iterator programs, a bpf_link to the program is obtained using bpf_link_create(), and the program callback is invoked by issuing system calls from user space.

Next, let us see how you can use the iterators to iterate on kernel objects and read data.

How to Use BPF iterators

BPF selftests are a great resource to illustrate how to use the iterators. In this section, we’ll walk through a BPF selftest which shows how to load and use a BPF iterator program. To begin, we’ll look at bpf_iter.c, which illustrates how to load and trigger BPF iterators on the user space side. Later, we’ll look at a BPF program that runs in kernel space.

Loading a BPF iterator in the kernel from user space typically involves the following steps:

• The BPF program is loaded into the kernel through libbpf. Once the kernel has verified and loaded the program, it returns a file descriptor (fd) to user space.

• Obtain a link_fd to the BPF program by calling the bpf_link_create() specified with the BPF program file descriptor received from the kernel.

• Next, obtain a BPF iterator file descriptor (bpf_iter_fd) by calling the bpf_iter_create() specified with the bpf_link received from Step 2.

• Trigger the iteration by calling read(bpf_iter_fd) until no data is available.

• Close the iterator fd using close(bpf_iter_fd).

• If needed to reread the data, get a new bpf_iter_fd and do the read again.

The following are a few examples of selftest BPF iterator programs:

• bpf_iter_tcp4.c

• bpf_iter_task_vmas.c

• bpf_iter_task_file.c

Let us look at bpf_iter_task_file.c, which runs in kernel space:

Here is the definition of bpf_iter__task_file in vmlinux.h. Any struct name in vmlinux.h in the format bpf_iter__<iter_name> represents a BPF iterator. The suffix <iter_name> represents the type of iterator.

struct bpf_iter__task_file {
        union {
            struct bpf_iter_meta *meta;
        };
        union {
            struct task_struct *task;
        };
        u32 fd;
        union {
            struct file *file;
        };
};

In the above code, the field ‘meta’ contains the metadata, which is the same for all BPF iterator programs. The rest of the fields are specific to different iterators. For example, for task_file iterators, the kernel layer provides the ‘task’, ‘fd’ and ‘file’ field values. The ‘task’ and ‘file’ are reference counted, so they won’t go away when the BPF program runs.

Here is a snippet from the bpf_iter_task_file.c file:

SEC("iter/task_file")
int dump_task_file(struct bpf_iter__task_file *ctx)
{
  struct seq_file *seq = ctx->meta->seq;
  struct task_struct *task = ctx->task;
  struct file *file = ctx->file;
  __u32 fd = ctx->fd;

  if (task == NULL || file == NULL)
    return 0;

  if (ctx->meta->seq_num == 0) {
    count = 0;
    BPF_SEQ_PRINTF(seq, "    tgid      gid       fd      file\n");
  }

  if (tgid == task->tgid && task->tgid != task->pid)
    count++;

  if (last_tgid != task->tgid) {
    last_tgid = task->tgid;
    unique_tgid_count++;
  }

  BPF_SEQ_PRINTF(seq, "%8d %8d %8d %lx\n", task->tgid, task->pid, fd,
          (long)file->f_op);
  return 0;
}

In the above example, the section name SEC(iter/task_file), indicates that the program is a BPF iterator program to iterate all files from all tasks. The context of the program is bpf_iter__task_file struct.

The user space program invokes the BPF iterator program running in the kernel by issuing a read() syscall. Once invoked, the BPF program can export data to user space using a variety of BPF helper functions. You can use either bpf_seq_printf() (and BPF_SEQ_PRINTF helper macro) or bpf_seq_write() function based on whether you need formatted output or just binary data, respectively. For binary-encoded data, the user space applications can process the data from bpf_seq_write() as needed. For the formatted data, you can use cat <path> to print the results similar to cat /proc/net/netlink after pinning the BPF iterator to the bpffs mount. Later, use rm -f <path> to remove the pinned iterator.

For example, you can use the following command to create a BPF iterator from the bpf_iter_ipv6_route.o object file and pin it to the /sys/fs/bpf/my_route path:

$ bpftool iter pin ./bpf_iter_ipv6_route.o  /sys/fs/bpf/my_route

And then print out the results using the following command:

$ cat /sys/fs/bpf/my_route

Implement Kernel Support for BPF Iterator Program Types

To implement a BPF iterator in the kernel, the developer must make a one-time change to the following key data structure defined in the bpf.h file.

struct bpf_iter_reg {
          const char *target;
          bpf_iter_attach_target_t attach_target;
          bpf_iter_detach_target_t detach_target;
          bpf_iter_show_fdinfo_t show_fdinfo;
          bpf_iter_fill_link_info_t fill_link_info;
          bpf_iter_get_func_proto_t get_func_proto;
          u32 ctx_arg_info_size;
          u32 feature;
          struct bpf_ctx_arg_aux ctx_arg_info[BPF_ITER_CTX_ARG_MAX];
          const struct bpf_iter_seq_info *seq_info;
};

After filling the data structure fields, call bpf_iter_reg_target() to register the iterator to the main BPF iterator subsystem.

The following is the breakdown for each field in struct bpf_iter_reg.

Fields	Description
target	Specifies the name of the BPF iterator. For example: bpf_map, bpf_map_elem. The name should be different from other bpf_iter target names in the kernel.
attach_target and detach_target	Allows for target specific link_create action since some targets may need special processing. Called during the user space link_create stage.
show_fdinfo and fill_link_info	Called to fill target specific information when user tries to get link info associated with the iterator.
get_func_proto	Permits a BPF iterator to access BPF helpers specific to the iterator.
ctx_arg_info_size and ctx_arg_info	Specifies the verifier states for BPF program arguments associated with the bpf iterator.
feature	Specifies certain action requests in the kernel BPF iterator infrastructure. Currently, only BPF_ITER_RESCHED is supported. This means that the kernel function cond_resched() is called to avoid other kernel subsystem (e.g., rcu) misbehaving.
seq_info	Specifies the set of seq operations for the BPF iterator and helpers to initialize/free the private data for the corresponding seq_file.

Click here to see an implementation of the task_vma BPF iterator in the kernel.

Parameterizing BPF Task Iterators

By default, BPF iterators walk through all the objects of the specified types (processes, cgroups, maps, etc.) across the entire system to read relevant kernel data. But often, there are cases where we only care about a much smaller subset of iterable kernel objects, such as only iterating tasks within a specific process. Therefore, BPF iterator programs support filtering out objects from iteration by allowing user space to configure the iterator program when it is attached.

BPF Task Iterator Program

The following code is a BPF iterator program to print files and task information through the seq_file of the iterator. It is a standard BPF iterator program that visits every file of an iterator. We will use this BPF program in our example later.

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>

char _license[] SEC("license") = "GPL";

SEC("iter/task_file")
int dump_task_file(struct bpf_iter__task_file *ctx)
{
      struct seq_file *seq = ctx->meta->seq;
      struct task_struct *task = ctx->task;
      struct file *file = ctx->file;
      __u32 fd = ctx->fd;
      if (task == NULL || file == NULL)
              return 0;
      if (ctx->meta->seq_num == 0) {
              BPF_SEQ_PRINTF(seq, "    tgid      pid       fd      file\n");
      }
      BPF_SEQ_PRINTF(seq, "%8d %8d %8d %lx\n", task->tgid, task->pid, fd,
                      (long)file->f_op);
      return 0;
}

Creating a File Iterator with Parameters

Now, let us look at how to create an iterator that includes only files of a process.

First, fill the bpf_iter_attach_opts struct as shown below:

LIBBPF_OPTS(bpf_iter_attach_opts, opts);
union bpf_iter_link_info linfo;
memset(&linfo, 0, sizeof(linfo));
linfo.task.pid = getpid();
opts.link_info = &linfo;
opts.link_info_len = sizeof(linfo);

linfo.task.pid, if it is non-zero, directs the kernel to create an iterator that only includes opened files for the process with the specified pid. In this example, we will only be iterating files for our process. If linfo.task.pid is zero, the iterator will visit every opened file of every process. Similarly, linfo.task.tid directs the kernel to create an iterator that visits opened files of a specific thread, not a process. In this example, linfo.task.tid is different from linfo.task.pid only if the thread has a separate file descriptor table. In most circumstances, all process threads share a single file descriptor table.

Now, in the userspace program, pass the pointer of struct to the bpf_program__attach_iter().

link = bpf_program__attach_iter(prog, &opts);
iter_fd = bpf_iter_create(bpf_link__fd(link));

If both tid and pid are zero, an iterator created from this struct bpf_iter_attach_opts will include every opened file of every task in the system (in the namespace, actually.) It is the same as passing a NULL as the second argument to bpf_program__attach_iter().

The whole program looks like the following code:

#include <stdio.h>
#include <unistd.h>
#include <bpf/bpf.h>
#include <bpf/libbpf.h>
#include "bpf_iter_task_ex.skel.h"

static int do_read_opts(struct bpf_program *prog, struct bpf_iter_attach_opts *opts)
{
      struct bpf_link *link;
      char buf[16] = {};
      int iter_fd = -1, len;
      int ret = 0;

      link = bpf_program__attach_iter(prog, opts);
      if (!link) {
              fprintf(stderr, "bpf_program__attach_iter() fails\n");
              return -1;
      }
      iter_fd = bpf_iter_create(bpf_link__fd(link));
      if (iter_fd < 0) {
              fprintf(stderr, "bpf_iter_create() fails\n");
              ret = -1;
              goto free_link;
      }
      /* not check contents, but ensure read() ends without error */
      while ((len = read(iter_fd, buf, sizeof(buf) - 1)) > 0) {
              buf[len] = 0;
              printf("%s", buf);
      }
      printf("\n");
free_link:
      if (iter_fd >= 0)
              close(iter_fd);
      bpf_link__destroy(link);
      return 0;
}

static void test_task_file(void)
{
      LIBBPF_OPTS(bpf_iter_attach_opts, opts);
      struct bpf_iter_task_ex *skel;
      union bpf_iter_link_info linfo;
      skel = bpf_iter_task_ex__open_and_load();
      if (skel == NULL)
              return;
      memset(&linfo, 0, sizeof(linfo));
      linfo.task.pid = getpid();
      opts.link_info = &linfo;
      opts.link_info_len = sizeof(linfo);
      printf("PID %d\n", getpid());
      do_read_opts(skel->progs.dump_task_file, &opts);
      bpf_iter_task_ex__destroy(skel);
}

int main(int argc, const char * const * argv)
{
      test_task_file();
      return 0;
}

The following lines are the output of the program.

PID 1859

   tgid      pid       fd      file
   1859     1859        0 ffffffff82270aa0
   1859     1859        1 ffffffff82270aa0
   1859     1859        2 ffffffff82270aa0
   1859     1859        3 ffffffff82272980
   1859     1859        4 ffffffff8225e120
   1859     1859        5 ffffffff82255120
   1859     1859        6 ffffffff82254f00
   1859     1859        7 ffffffff82254d80
   1859     1859        8 ffffffff8225abe0

Without Parameters

Let us look at how a BPF iterator without parameters skips files of other processes in the system. In this case, the BPF program has to check the pid or the tid of tasks, or it will receive every opened file in the system (in the current pid namespace, actually). So, we usually add a global variable in the BPF program to pass a pid to the BPF program.

The BPF program would look like the following block.

......
int target_pid = 0;

SEC("iter/task_file")
int dump_task_file(struct bpf_iter__task_file *ctx)
{
      ......
      if (task->tgid != target_pid) /* Check task->pid instead to check thread IDs */
              return 0;
      BPF_SEQ_PRINTF(seq, "%8d %8d %8d %lx\n", task->tgid, task->pid, fd,
                      (long)file->f_op);
      return 0;
}

The user space program would look like the following block:

......
static void test_task_file(void)
{
      ......
      skel = bpf_iter_task_ex__open_and_load();
      if (skel == NULL)
              return;
      skel->bss->target_pid = getpid(); /* process ID.  For thread id, use gettid() */
      memset(&linfo, 0, sizeof(linfo));
      linfo.task.pid = getpid();
      opts.link_info = &linfo;
      opts.link_info_len = sizeof(linfo);
      ......
}

target_pid is a global variable in the BPF program. The user space program should initialize the variable with a process ID to skip opened files of other processes in the BPF program. When you parametrize a BPF iterator, the iterator calls the BPF program fewer times which can save significant resources.

Parametrizing VMA Iterators

By default, a BPF VMA iterator includes every VMA in every process. However, you can still specify a process or a thread to include only its VMAs. Unlike files, a thread can not have a separate address space (since Linux 2.6.0-test6). Here, using tid makes no difference from using pid.

Parametrizing Task Iterators

A BPF task iterator with pid includes all tasks (threads) of a process. The BPF program receives these tasks one after another. You can specify a BPF task iterator with tid parameter to include only the tasks that match the given tid.