13. PCI Support for Live Update

The PCI subsystem participates in the Live Update process to enable drivers to preserve their PCI devices across kexec.

Preserving a device requires preserving two independent sets of state: the driver’s own state, which the driver preserves with no involvement from the PCI core, and the PCI core’s state about the device, which the next kernel needs so that enumeration does not disturb a device that is still running. This file implements the latter.

13.1. FLB Data

Userspace decides which devices are preserved, using LUO file preservation: a driver exposes a file that represents a single PCI device, and userspace preserves the device with ioctl(LIVEUPDATE_SESSION_PRESERVE_FD) on that file. Binding preservation to a file gives it proper lifecycle management, e.g. the preservation is undone if userspace cancels it or goes away. How a driver exposes that file is up to the driver and invisible to the PCI core (vfio-pci variant drivers, the first intended use-case, use their per-device cdev).

LUO only knows that a file was preserved; it does not know that the file represents a PCI device. Drivers therefore register their struct liveupdate_file_handler with the PCI core:

  • pci_liveupdate_register_flb(driver_file_handler)

  • pci_liveupdate_unregister_flb(driver_file_handler)

LUO then refcounts the PCI core’s FLB against the files preserved by that handler, and that refcount drives the lifetime of struct pci_ser: pci_flb_preserve() allocates and preserves it when the first file is preserved, and pci_flb_unpreserve() frees it when the last file is unpreserved. In the next kernel, pci_flb_retrieve() hands the PCI core the struct pci_ser built by the previous kernel, whenever the PCI core asks for it (e.g. during enumeration), and pci_flb_finish() frees it once the PCI core is done with it.

State handed over by the previous kernel is trusted. The PCI core validates it only far enough to detect an incompatible or corrupt hand over, and makes no attempt to defend against deliberate modification, since a previous kernel able to corrupt preserved state is able to corrupt arbitrary memory anyway.

13.1.1. Call Flow

# Driver initialization
pci_liveupdate_register_flb(fh)

# Userspace: ioctl(LIVEUPDATE_SESSION_PRESERVE_FD, devfd)
luo_preserve_file()
  luo_flb_file_preserve()
    luo_flb_file_preserve_one()      # first preserved file only
      pci_flb_preserve()             # alloc and preserve struct pci_ser
  fh->ops->preserve()                # driver callback
    pci_liveupdate_preserve(dev)     # record this device in struct pci_ser

# Userspace: preservation cancelled or session torn down
luo_file_unpreserve_files()
  luo_flb_file_unpreserve()
    liveupdate_flb_put_outgoing()    # last unpreserved file only
      pci_flb_unpreserve()           # free struct pci_ser

# ---------------- kexec ----------------

# New kernel: PCI enumeration
pci_setup_device()
  pci_liveupdate_setup_device()
    liveupdate_flb_get_incoming()
      luo_flb_retrieve_one()         # first request only
        pci_flb_retrieve()           # previous kernel's struct pci_ser

# Userspace: ioctl(LIVEUPDATE_SESSION_FINISH)
luo_file_finish_one()
  fh->ops->finish()                  # driver callback
    pci_liveupdate_finish(dev)       # release this device's pci_dev_ser
  luo_flb_file_finish()
    liveupdate_flb_put_incoming()    # last incoming file only
      pci_flb_finish()               # free struct pci_ser

13.2. Device Tracking

Drivers must notify the PCI core when specific devices are preserved or unpreserved with the following APIs:

  • pci_liveupdate_preserve(pci_dev)

  • pci_liveupdate_unpreserve(pci_dev)

This allows the PCI core to keep its FLB data (struct pci_ser) up to date with the list of outgoing preserved devices for the next kernel.

After kexec, whenever a device is enumerated, the PCI core will check if it is an incoming preserved device (i.e. preserved by the previous kernel) by checking the incoming FLB data (struct pci_ser).

Drivers must notify the PCI core when an incoming device is done participating in the incoming Live Update with the following API:

  • pci_liveupdate_finish(pci_dev)

The PCI core does not enforce any ordering of pci_liveupdate_finish() and pci_liveupdate_preserve(), i.e., a PCI device can be outgoing (preserved for next kernel) and incoming (preserved by previous kernel) at the same time.

13.3. Restrictions

The PCI core enforces the following restrictions on which devices can be preserved. These may be relaxed in the future:

  • The device cannot be a Virtual Function (VF).

  • The device cannot require device-specific quirks to enable Access Control Services (ACS).

13.4. Driver Binding

In the outgoing kernel, the driver must ensure that it does not release a device between pci_liveupdate_preserve() and pci_liveupdate_unpreserve().

In the incoming kernel, the driver must ensure that it does not release a preserved device between probe() and pci_liveupdate_finish().

It is the user’s responsibility to ensure that incoming preserved devices are bound to the correct driver. The PCI core does not protect against a device getting preserved by driver A in the outgoing kernel and then getting bound to driver B in the incoming kernel.

13.5. PCI-to-PCI Bridges

Any PCI-to-PCI bridges upstream of a preserved device are automatically preserved when the device is preserved. The PCI core keeps track of the number of downstream devices that are preserved under a bridge so that the bridge is only unpreserved once all downstream devices are unpreserved.

This enables the PCI core and any drivers bound to the bridge to participate in the Live Update so that preserved endpoints can continue issuing memory transactions during the Live Update.

13.6. BDF Stability

The PCI core guarantees that preserved devices can be identified by the same bus, device, and function numbers for as long as they are preserved (including across kexec). To accomplish this, the PCI core keeps the secondary and subordinate bus numbers that the previous kernel programmed into bridges, if the previous kernel preserved any device. This is true even on architectures that always assign new bus numbers during scanning. The kernel assumes the previous kernel established a sane bus topology across kexec.

Bridges that do not have bus numbers are assigned new ones as usual, so hot-adding a bridge keeps working, both during and after a Live Update. The two-pass bridge scan ensures such bridges are only assigned bus numbers above those already claimed by preserved bridges.

If a preserved bridge comes up without a valid bus number configuration, e.g. because it was reset during kexec, the PCI core refuses to assign it new bus numbers and does not enumerate anything below it. Assigning new bus numbers would silently change the BDF of every preserved device in its hierarchy. The PCI core also stops assigning bus numbers to the other bridges on the same bus, since the bus numbers of the failed bridge can no longer be read from hardware and handing them to another bridge would let an unrelated device inherit the BDF of a preserved device.

13.7. Handling Preserved Devices

The PCI core treats preserved devices differently than non-preserved devices. This section enumerates those differences.

  • The PCI core adopts all ACS controls enabled on incoming preserved devices rather than assigning new ones. This ensures that TLPs are routed the same way after Live Update and ensures that IOMMU groups do not change. Note that a device will use its adopted ACS controls for the lifetime of its struct pci_dev (i.e. even after pci_liveupdate_finish()).

  • The PCI core adopts ARI Forwarding Enable on all bridges with downstream preserved devices to ensure that all preserved devices on the bridge’s secondary bus are addressable after the Live Update.

  • The PCI core does not disable bus mastering on outgoing preserved devices during kexec. This allows preserved devices to issue memory transactions throughout the Live Update.

13.8. Driver API

int pci_liveupdate_preserve(struct pci_dev *dev)

Preserve a PCI device across Live Update

Parameters

struct pci_dev *dev

The PCI device to preserve.

Description

pci_liveupdate_preserve() notifies the PCI core that a PCI device should be preserved across the next Live Update. Drivers are expected to call pci_liveupdate_preserve() from their struct liveupdate_file_handler preserve() callback to ensure the outgoing struct pci_ser is already set up.

pci_liveupdate_preserve() automatically preserves all bridges upstream of dev.

Return

0 on success, <0 on failure.

void pci_liveupdate_unpreserve(struct pci_dev *dev)

Cancel preservation of a PCI device

Parameters

struct pci_dev *dev

The PCI device to unpreserve.

Description

pci_liveupdate_unpreserve() notifies the PCI core that a PCI device should no longer be preserved across the next Live Update. Drivers are expected to call pci_liveupdate_unpreserve() from their struct liveupdate_file_handler unpreserve() callback to ensure the outgoing struct pci_ser is already set up.

pci_liveupdate_unpreserve() automatically unpreserves all bridges upstream of dev.

void pci_liveupdate_finish(struct pci_dev *dev)

Finish the preservation of a PCI device

Parameters

struct pci_dev *dev

The PCI device

Description

pci_liveupdate_finish() notifies the PCI core that a PCI device that was preserved across the previous Live Update has finished participating in Live Update. Drivers must call pci_liveupdate_finish() from their struct liveupdate_file_handler finish() callback to ensure the incoming struct pci_ser is allocated.

pci_liveupdate_finish() automatically finishes all bridges upstream of dev.

bool pci_liveupdate_is_incoming(struct pci_dev *dev)

Check if a device is incoming-preserved

Parameters

struct pci_dev *dev

The PCI device to check

Description

Check if a device was preserved across Live Update by the previous kernel, i.e. the device is incoming-preserved. Note that a device is only considered incoming-preserved prior to pci_liveupdate_finish(). It is up to drivers to synchronize usage of pci_liveupdate_is_incoming() with their own call to pci_liveupdate_finish() to avoid acting on stale data.

Return

True if the device is incoming-preserved, false otherwise.

int pci_liveupdate_register_flb(struct liveupdate_file_handler *fh)

Register a file handler with the PCI core

Parameters

struct liveupdate_file_handler *fh

The file handler to register.

Description

Drivers that support preserving PCI devices across Live Update must call pci_liveupdate_register_flb() to register their struct liveupdate_file_handler with the PCI core, typically at module init, and always before any file managed by fh can be preserved.

Registering links the PCI core’s FLB to fh, so that LUO allocates the PCI core’s outgoing struct pci_ser (via pci_flb_preserve()) when the first file managed by any registered handler is preserved, and frees it (via pci_flb_unpreserve()) when the last such file is unpreserved.

Return

0 on success, <0 on failure.

void pci_liveupdate_unregister_flb(struct liveupdate_file_handler *fh)

Unregister a file handler with the PCI core

Parameters

struct liveupdate_file_handler *fh

The file handler to unregister.

13.9. Internal API

struct pci_liveupdate_global

Global state for PCI Live Update support

Definition:

struct pci_liveupdate_global {
    struct rw_semaphore rwsem;
    bool had_incoming;
};

Members

rwsem

Reader/writer semaphore used to protect the incoming and outgoing FLBs, and the references to them in struct pci_dev.

had_incoming

True if the previous kernel preserved at least one PCI device. Set when the incoming FLB is retrieved and never cleared, so it stays true after Live Update finishes.

struct pci_flb_outgoing

Outgoing PCI FLB object

Definition:

struct pci_flb_outgoing {
    struct pci_ser *ser;
    struct kho_block_set block_set;
};

Members

ser

Pointer to the preserved struct pci_ser.

block_set

The KHO block set holding the outgoing devices.

Description

This structure holds the runtime state for the outgoing PCI Live Update state. It wraps the serialized pci_ser and the block_set used to manage the serialized entries.

struct pci_flb_incoming

Incoming PCI FLB object

Definition:

struct pci_flb_incoming {
    struct pci_ser *ser;
    struct xarray xa;
    struct kho_block_set block_set;
};

Members

ser

The incoming struct pci_ser from the previous kernel.

xa

Xarray used to quickly lookup devices in ser.

block_set

The KHO block set holding the incoming devices.

Description

This structure holds the runtime state for the incoming PCI Live Update state. It wraps the serialized pci_ser, the block_set used to restore the serialized entries, and an xarray for fast lookups.

bool pci_liveupdate_preserve_bus_numbers(void)

Determine if the PCI core should preserve bus numbers when scanning bridges.

Parameters

void

no arguments

Description

This function is called by the PCI core when it is scanning a bridge. It determines whether the PCI core should preserve the secondary and subordinate bus numbers that the previous kernel programmed into that bridge, rather than assigning new ones. This is necessary to keep RequesterIDs constant for preserved devices issuing memory transactions.

Bus numbers are preserved everywhere, and for the lifetime of the kernel, if the previous kernel preserved any device. Bus numbers have to be preserved above a preserved device anyway, since an upstream bridge cannot expand its window. Applying the same policy everywhere matches the scope of pcibios_assign_all_busses(), and gives an answer that cannot change part way through the two passes of a bridge scan.

The incoming FLB is retrieved while setting up the first device, which always happens before any bridge is scanned, so this returns the same answer for the entire enumeration.

Note that this does not prevent the PCI core from assigning bus numbers to bridges that do not have any, e.g. bridges that are hot-added after the Live Update. See pci_liveupdate_refuse_bus_numbers() for the one case where the PCI core must refuse to do so.

Return

True if bus numbers should be preserved, false otherwise.

bool pci_liveupdate_refuse_bus_numbers(struct pci_bus *bus, struct pci_dev *dev)

Determine if the PCI core must refuse to assign bus numbers to the provided bridge.

Parameters

struct pci_bus *bus

The PCI bus the bus numbers would be assigned from.

struct pci_dev *dev

The PCI bridge device the bus numbers would be assigned to.

Description

This function is called by the PCI core before it assigns bus numbers to a bridge that does not have any.

A bridge that was preserved by the previous kernel but came up without a valid bus number configuration, e.g. because it was reset during kexec, is left alone by the PCI core and therefore has no child bus once the first pass of the bridge scan is done.

The PCI core must not assign bus numbers from bus while such a bridge is on it, including to the failed bridge itself. Assigning new bus numbers to the failed bridge would silently change the BDF of every preserved device in its hierarchy. Its bus numbers cannot be read from hardware anymore either, so they cannot be excluded from assignment, and handing them to another bridge would let an unrelated device inherit the BDF of a preserved device.

Return

True if dev must not be assigned bus numbers, false otherwise.

int pci_liveupdate_adopt_acs(struct pci_dev *dev)

Adopt ACS controls

Parameters

struct pci_dev *dev

The PCI device to adopt ACS controls for

Description

For devices preserved across a Live Update, leave the ACS controls established by the previous kernel alone instead of programming new ones. The adopted controls are recorded by the pci_save_state() call in pci_bus_add_device(), so they are reapplied by pci_restore_state() if the device is subsequently reset.

Return

0 on success, or -EINVAL if the device was not preserved, requires device-specific quirks, or has nowhere to record the adopted controls.

int pci_liveupdate_adopt_ari(struct pci_dev *dev)

Adopt ARI configuration

Parameters

struct pci_dev *dev

The PCI device to adopt ARI configuration for

Description

For devices preserved across a Live Update, read the ARI state from hardware and adopt it. This ensures the device continues to use the ARI configuration established by the previous kernel.

Return

0 on success, or -EINVAL if the device was not preserved.

13.10. Live Update ABI

This header defines the ABI for preserving core PCI state across kexec using Live Update File-Lifecycle Bound (FLB) data.

This interface is a contract. Any modification to any of the serialization structs defined here constitutes a breaking change. Such changes require incrementing the version number in the PCI_LUO_FLB_VERSION number.

struct pci_dev_ser

Serialized state about a single PCI device.

Definition:

struct pci_dev_ser {
    u32 domain;
    u16 bdf;
    u16 refcount;
};

Members

domain

The device’s PCI domain number (segment).

bdf

The device’s PCI bus, device, and function number.

refcount

Reference count used by the PCI core to keep track of whether it is done using a device’s struct pci_dev_ser. The value of the refcount is equal to the number of preserved devices at or below it in the PCI hierarchy when the struct pci_dev_ser is in use, and 0 otherwise.

struct pci_ser

PCI Subsystem Live Update State

Definition:

struct pci_ser {
    u32 version;
    u32 nr_devices;
    u64 devices;
};

Members

version

The version of the “pci” FLB struct. This field must never be deleted, moved, or resized, as the kernel depends on always being able to check the struct pci_ser version number.

nr_devices

The number of devices that were preserved.

devices

Physical address of the first KHO block containing pci_dev_ser.

Description

This struct tracks state about all devices that are being preserved across a Live Update for the next kernel. It contains only state owned by the PCI core; the state a driver needs to resume its device is preserved separately by that driver.

13.11. See Also