[PATCH v2] staging: fsl-mc: add DPAA2 overview readme
Stuart Yoder
stuart.yoder at freescale.com
Wed Aug 12 01:03:04 UTC 2015
> -----Original Message-----
> From: Tillmann Heidsieck [mailto:theidsieck at leenox.de]
> Sent: Tuesday, August 11, 2015 6:01 AM
> To: Yoder Stuart-B08248; gregkh at linuxfoundation.org; Rivera Jose-B46482; katz Itai-RM05202
> Cc: devel at driverdev.osuosl.org; linux-kernel at vger.kernel.org; agraf at suse.de; arnd at arndb.de
> Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme
>
> Hi Stuart,
>
> I am by no means a native speaker, but I have proof-read my fair share of articles and theses, so here are a
> bunch of suggestions which might or might not help improve you document.
>
> On 07.08.2015 03:09, Stuart Yoder wrote:
> > add README file providing an overview of the DPAA2 architecture
> > and how it is integrated in Linux
> >
> > Signed-off-by: Stuart Yoder <stuart.yoder at freescale.com>
> > ---
> > -v2: added changelog text
> >
> > drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++
> > drivers/staging/fsl-mc/TODO | 4 -
> > 2 files changed, 364 insertions(+), 4 deletions(-)
> > create mode 100644 drivers/staging/fsl-mc/README.txt
> >
> > diff --git a/drivers/staging/fsl-mc/README.txt b/drivers/staging/fsl-mc/README.txt
> > new file mode 100644
> > index 0000000..8214102
> > --- /dev/null
> > +++ b/drivers/staging/fsl-mc/README.txt
> > @@ -0,0 +1,364 @@
> > +Copyright (C) 2015 Freescale Semiconductor Inc.
> > +
> > +DPAA2 (Data Path Acceleration Architecture Gen2)
> > +------------------------------------------------
> > +
> > +This document provides an overview of the Freescale DPAA2 architecture
> > +and how it is integrated into the Linux kernel.
> > +
> > +Contents summary
> > + -DPAA2 overview
> > + -Overview of DPAA2 objects
> > + -DPAA2 Linux driver architecture overview
> > + -bus driver
> > + -dprc driver
>
> - DPRC driver
>
> > + -allocator
> > + -dpio driver
>
> - DPIO driver
>
> > + -Ethernet
> > + -mac
> mac -> MAC
> > +
> > +DPAA2 Overview
> > +--------------
> > +
> > +DPAA2 is a hardware architecture designed for high-speeed network
> > +packet processing. DPAA2 consists of sophisticated mechanisms for
> > +processing Ethernet packets, queue management, buffer management,
> > +autonomous L2 switching, virtual Ethernet bridging, and accelerator
> > +(e.g. crypto) sharing.
> > +
> > +A DPAA2 hardware component called the Management Complex (or MC) manages the
> > +DPAA2 hardware resources. The MC provides an object-based abstraction for
> > +software drivers to use the DPAA2 hardware.
> > +
> > +The MC uses DPAA2 hardware resources such as queues, buffer pools, and
> > +network ports to create functional objects/devices such as network
> > +interfaces, an L2 switch, or accelerator instances.
> > +
> > +The MC provides memory-mapped I/O command interfaces (MC portals)
> > +which DPAA2 software drivers use to operate on DPAA2 objects:
> > +
> > + +--------------------------------------+
> > + | OS |
> > + | DPAA2 drivers |
> > + | | |
> > + +-----------------------------|--------+
> > + |
> > + | (create,discover,connect
> > + | config,use,destroy)
> > + |
> > + DPAA2 |
> > + +------------------------| mc portal |-+
> > + | | |
> > + | +- - - - - - - - - - - - -V- - -+ |
> > + | | | |
> > + | | Management Complex (MC) | |
> > + | | | |
> > + | +- - - - - - - - - - - - - - - -+ |
> > + | |
> > + | Hardware Hardware |
> > + | Resources Objects |
> > + | --------- ------- |
> > + | -queues -DPRC |
> > + | -buffer pools -DPMCP |
> > + | -Eth MACs/ports -DPIO |
> > + | -network interface -DPNI |
> > + | profiles -DPMAC |
> > + | -queue portals -DPBP |
> > + | -MC portals ... |
> > + | ... |
> > + | |
> > + +--------------------------------------+
> > +
> > +The MC mediates operations such as create, discover,
> > +connect, configuration, and destroy. Fast-path operations
> > +on data, such as packet transmit/receive, are not mediated by
> > +the MC and are done directly using memory mapped regions in
> > +DPIO objects.
> > +
> > +Overview of DPAA2 Objects
> > +-------------------------
> > +The section provides a brief overview of some key objects
> > +in the DPAA2 hardware. A simple scenario is described illustrating
> > +the objects involved in creating a network interfaces.
> > +
> > +-DPRC (Datapath Resource Container)
> > +
> > + A DPRC is an container object that holds all the other
>
> ... is a container ...
>
> > + types of DPAA2 objects. In the example diagram below there
> > + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
> > + in the container.
> > +
> > + +---------------------------------------------------------+
> > + | DPRC |
> > + | |
> > + | +-------+ +-------+ +-------+ +-------+ +-------+ |
> > + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |
> > + | +-------+ +-------+ +-------+ +---+---+ +---+---+ |
> > + | | DPMCP | | DPIO | |
> > + | +-------+ +-------+ |
> > + | | DPMCP | |
> > + | +-------+ |
> > + | |
> > + +---------------------------------------------------------+
> > +
> > + From the point of view of an OS, a DPRC is bus-like. Like
>
> maybe replace "is bus-like" with "behaves similar to a bus" or "can be viewed as a bus"
>
> > + a plug-and-play bus, such as PCI, DPRC commands can be used to
>
> maybe replace "Like a pnp bus..." with "Similar to a plug-and-play bus, such as PCI, DPRC ..."
>
> > + enumerate the contents of the DPRC, discover the hardware
> > + objects present (including mappable regions and interrupts).
> > +
> > + dprc.1 (bus)
> > + |
> > + +--+--------+-------+-------+-------+
> > + | | | | |
> > + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1
> > + dpmcp.2 dpio.2
> > + dpmcp.3
> > +
> > + Hardware objects can be created and destroyed dynamically, providing
> > + the ability to hot plug/unplug objects in and out of the DPRC.
> > +
> > + A DPRC has a mappable mmio region (an MC portal) that can be used
>
> - mmio -> MMIO (not sure whether mappable MMIO is redundant or not)
> - a MC portal
>
> > + to send MC commands. It has an interrupt for status events (like
> > + hotplug).
> > +
> > + All objects in a container share the same hardware "isolation context".
> > + This means that with respect to an IOMMU the isolation granularity
> > + is at the DPRC (container) level, not at the individual object
> > + level.
> > +
> > + DPRCs can be defined statically and populated with objects
> > + via a config file passed to the MC when firmware starts
> > + it. There is also a Linux user space tool called "restool"
> > + that can be used to create/destroy containers and objects
> > + dynamically.
> > +
> > +-DPAA2 Objects for an Ethernet Network Interface
> > +
> > + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
> > + queuing mechanisms, configuration mechanisms, buffer management,
> > + physical ports, and interrupts. DPAA2 uses a more granular approach
> > + utilizing multiple hardware objects. Each object has specialized
> > + functions, and are used together by software to provide Ethernet network
>
> Each object provides specialized functions. Groups of these objects are used by the software to provide
> Ethernet network interface functionality.
>
> > + interface functionality. This approach provides efficient use of finite
> > + hardware resources, flexibility, and performance advantages.
> > +
> > + The diagram below shows the objects needed for a simple
> > + network interface configuration on a system with 2 CPUs.
> > +
> > + +---+---+ +---+---+
> > + CPU0 CPU1
> > + +---+---+ +---+---+
> > + | |
> > + +---+---+ +---+---+
> > + DPIO DPIO
> > + +---+---+ +---+---+
> > + \ /
> > + \ /
> > + \ /
> > + +---+---+
> > + DPNI --- DPBP,DPMCP
> > + +---+---+
> > + |
> > + |
> > + +---+---+
> > + DPMAC
> > + +---+---+
> > + |
> > + port/PHY
> > +
> > + Below the objects are described. For each object a brief description
>
> The objects depicted in this figure are described below.
>
> > + is provided along with a summary of the kinds of operations the object
> > + supports and a summary of key resources of the object (mmio regions
> > + and irqs).
>
> mmio -> MMIO
> irqs -> IRQs
>
> > +
> > + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
> > + hardware device that connects to an Ethernet PHY and allows
> > + physical transmission and reception of Ethernet frames.
> > + -mmio regions: none
>
> mmio -> MMIO
>
> > + -irqs: dpni link change
>
> irqs -> IRQs
> dpni -> DPNI
>
> > + -commands: set link up/down, link config, get stats,
> > + irq config, enable, reset
>
> irq -> IRQ
>
> > +
> > + -DPNI (Datapath Network Interface): contains TX/RX queues,
> > + network interface configuration, and rx buffer pool configuration
>
> rx -> RX
>
> > + mechanisms.
> > + -mmio regions: none
>
> mmio -> MMIO
>
> > + -irqs: link state
>
> irqs -> IRQs
>
> > + -commands: port config, offload config, queue config,
> > + parse/classify config, irq config, enable, reset
>
> irq -> IRQ
>
> > +
> > + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
> > + packets and do hardware buffer pool management operations. For
> > + optimum performance there is typically DPIO per CPU. This allows
>
> For optimum performance there is typically one DPIO assigned to each CPU
>
> > + each CPU to perform simultaneous enqueue/dequeue operations.
>
> This allows different CPUs to simultaneously perform enqueue/dequeue operations.
>
> > + -mmio regions: queue operations, buffer mgmt
>
> mmio -> MMIO
> mgmt -> management
>
> > + -irqs: data availability, congestion notification, buffer
> > + pool depletion
>
> irqs -> IRQs
>
> > + -commands: irq config, enable, reset
>
> irq -> IRQ
>
> > +
> > + -DPBP (Datapath Buffer Pool): represents a hardware buffer
> > + pool.
> > + -mmio regions: none
> > + -irqs: none
> > + -commands: enable, reset
>
> mmio -> MMIO
> irqs -> IRQs
>
> > +
> > + -DPMCP (Datapath MC Portal): provides an MC command portal.
> > + Used by drivers to send commands to the MC to manage
> > + objects.
> > + -mmio regions: MC command portal
> > + -irqs: command completion
> > + -commands: irq config, enable, reset
>
> mmio -> MMIO
> irqs -> IRQs
> irq -> IRQ
>
> > +
> > + Object Connections
> > + ------------------
> > + Some objects have explicit relationships that must
> > + be configured:
> > +
> > + -DPNI <--> DPMAC
> > + -DPNI <--> DPNI
> > + -DPNI <--> L2-switch-port
> > + A DPNI must be connected to something such as a DPMAC,
> > + another DPNI, or L2 switch port. The DPNI connection
> > + is made via a DPRC command.
> > +
> > + +-------+ +-------+
> > + | DPNI | | DPMAC |
> > + +---+---+ +---+---+
> > + | |
> > + +==========+
> > +
> > + -DPNI <--> DPBP
> > + A network interface requires a 'buffer pool' (DPBP
> > + object) which provides a list of pointers to memory
> > + where received Ethernet data is to be copied. The
> > + Ethernet driver configures the DPBPs associated with
> > + the network interface.
> > +
> > + Interrupts
> > + ----------
> > + All interrupts generated by DPAA2 objects are message
> > + interrupts. At the hardware level message interrupts
> > + generated by devices will normally have 3 components--
> > + 1) a non-spoofable 'device-id' expressed on the hardware
> > + bus, 2) an address, 3) a data value.
> > +
> > + In the case of DPAA2 devices/objects, all objects in the
> > + same container/DPRC share the same 'device-id'.
> > + For ARM-based SoC this is the same as the stream ID.
> > +
> > +
> > +DPAA2 Linux Driver Overview
> > +---------------------------
> > +
> > +This section provides an overview of the Linux kernel drivers for
> > +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
> > +drivers and 2) functional object drivers (such as Ethernet).
> > +
> > +As described previously, a DPRC is a container that holds the other
> > +types of DPAA2 objects. It is functionally similar to a plug-and-play
> > +bus controller.
> > +
> > +Each object in the DPRC is a Linux "device" and is bound to a driver.
> > +The diagram below shows the Linux drivers involved in a networking
> > +scenario and the objects bound to each driver. A brief description
> > +of each driver follows.
> > +
> > + +------------+
> > + | OS Network |
> > + | Stack |
> > + +------------+ +------------+
> > + | Allocator |. . . . . . . | Ethernet |
> > + |(dpmcp,dpbp)| | (dpni) |
> > + +-.----------+ +---+---+----+
> > + . . ^ |
> > + . . <data avail, | |<enqueue,
> > + . . tx confirm> | | dequeue>
> > + +-------------+ . | |
> > + | DPRC driver | . +---+---V----+ +---------+
> > + | (dprc) | . . . . . .| DPIO driver| | MAC |
> > + +----------+--+ | (dpio) | | (dpmac) |
> > + | +------+-----+ +-----+---+
> > + |<dev add/remove> | |
> > + | | |
> > + +----+--------------+ | +--+---+
> > + | mc-bus driver | | | PHY |
> > + | | | |driver|
> > + | /fsl-mc at 80c000000 | | +--+---+
> > + +-------------------+ | |
> > + | |
> > + ================================ HARDWARE =========|=================|======
> > + DPIO |
> > + | |
> > + DPNI---DPBP |
> > + | |
> > + DPMAC |
> > + | |
> > + PHY ---------------+
> > + ===================================================|========================
> > +
> > +A brief description of each driver is provided below.
> > +
> > + mc-bus driver
> > + -------------
> mc-bus -> MC-Bus or MC-bus
> > + The mc-bus driver is a platform driver and is probed from an
> > + "/fsl-mc at xxxx" node in the device tree passed in by boot firmware.
> > + It is responsible for bootstrapping the DPAA2 kernel infrastructure.
> > + Key functions include:
> > + -registering a new bus type named "fsl-mc" with the kernel,
> > + and implementing bus call-backs (e.g. match/uevent/dev_groups)
> > + -implemeting APIs for DPAA2 driver registration and for device
> > + add/remove
> > + -creates an MSI irq domain
> irq -> IRQ
> > + -do a device add of the 'root' DPRC device, which is needed
> > + to bootstrap things
> > +
> > + DPRC driver
> > + -----------
> > + The dprc-driver is bound DPRC objects and does runtime management
> > + of a bus instance. It performs the initial bus scan of the DPRC
> > + and handles interrupts for container events such as hot plug.
> > +
> > + Allocator
> > + ----------
> > + Certain objects such as DPMCP and DPBP are generic and fungible,
> > + and are intended to be used by other drivers. For example,
> > + the DPAA2 Ethernet driver needs:
> > + -DPMCPs to send MC commands, to configure network interfaces
> > + -DPBPs for network buffer pools
> > +
> > + The allocator driver registers for these allocatable object types
> > + and those objects are bound to the allocator when the bus is probed.
> > + The allocator maintains a pool of objects that are available for
> > + allocation by other DPAA2 drivers.
> > +
> > + DPIO driver
> > + -----------
> > + The DPIO driver is bound to DPIO objects and provides services that allow
> > + other drivers such as the Ethernet driver to receive and transmit data.
> > + Key services include:
> > + -data availability notifications
> > + -hardware queuing operations (enqueue and dequeue of data)
> > + -hardware buffer pool management
> > +
> > + There is typically one DPIO object per physical CPU for optimum
> > + performance, allowing each CPU to simultaneously enqueue
> > + and dequeue data.
>
> allowing different CPUs to simultaneously ...
>
> > +
> > + The DPIO driver operates on behalf of all DPAA2 drivers
> > + active in the kernel-- Ethernet, crypto, compression,
> > + etc.
> > +
> > + Ethernet
> > + --------
> > + The Ethernet driver is bound to a DPNI and implements the kernel
> > + interfaces needed to connect the DPAA2 network interface to
> > + the network stack.
> > +
> > + Each DPNI corresponds to a Linux network interface.
> > +
> > + MAC driver
> > + ----------
> > + An Ethernet PHY is an off-chip, board specific component and is managed
> > + by the appropriate PHY driver via an mdio bus. The MAC driver
> > + plays a role of being a proxy between the PHY driver and the
> > + MC. It does this proxy via the MC commands to a DPMAC object.
> > diff --git a/drivers/staging/fsl-mc/TODO b/drivers/staging/fsl-mc/TODO
> > index c29516b..3894368 100644
> > --- a/drivers/staging/fsl-mc/TODO
> > +++ b/drivers/staging/fsl-mc/TODO
> > @@ -1,7 +1,3 @@
> > -* Add README file (with ASCII art) describing relationships between
> > - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc.
> > - Also, define all acronyms used.
> > -
> > * Decide if multiple root fsl-mc buses will be supported per Linux instance,
> > and if so add support for this.
> >
Appreciate the detailed review. Like most of your suggestions and
will incorporate on the next update to this document.
Stuart
More information about the devel
mailing list