ENEA LINUX NETWORKING PROFILE

Technology trends show that Linux-based Operating Systems have increased their presence in the area of high-performance networking applications. While there are no standardized ways of programatically accessing hardware offload capabilities, several paradigms co-exist in Linux ecosystem to address this specific need (e.g. USDPAA, DPDK, ODP etc.) Networking Profile in Enea Linux is a framework for anyone attempting to implement high-performance networking applications on various hardware platforms. It aims to bring in place all necessary building blocks which facilitate efficient development of Linux-based solutions on top of network accelerated hardware platforms. As different hardware platforms have distinct data-path acceleration solutions, Networking Profile implementation is very dependent on underlying hardware capabilities.

This document tries to describe the implementation details, changes, additions, kernel configurations and tunings Enea made in order to achieve highly optimized Linux distributions for networking applications.

The following paragraphs focus on Enea Linux Networking Profile on DPAA-based QorIQ platforms, illustrating the implementation and changes on NXP P2041rdb target.

Table of Content
-------------------------------------------
1. Supported Targets
2. Real-Time Performance
------- 2.1 Kernel Modifications
------- 2.2 CPU-Isolation with partrt
------- 2.3 Latency Benchmarks
3. USDPAA Usage
------- 3.1 Packages
------- 3.2 Prepare Target
------- 3.3 Device Trees
------- 3.4 Boot Parameters
------- 3.5 Boot Instructions P2041RDB
------- 3.6 Run Reflector
------- 3.7 Run SRA
------- 3.8 Throughput using USDPAA

1. Supported Targets

Enea Linux Networking Profile has initially been tested on p2041rdb.

2. Improving Real-Time Performance

2.1 Kernel Modifications

When modifying a kernel for high-performance and low-latency applications there are several aspects to take into consideration. In the Enea Linux Real-Time Guide a thorough investigation and explanation of how to optimize Linux for low latency is given. Below is a short description of kernel features added specifically to Enea Linux Networking Profile in order to enhance real-time performance.

        Table 2.1 Added kernel features and their properties.
Change Reason
RCU priority boosting -> cfg/rcu_boost.cfg Give low priority readers a higher priority to keep them from blocking tasks of higher prority. [RCU]
Offload RCU callback Processing -> cfg/rcu_nocb To reduce OS jitter, enable offloading of RCU callback processing to kernel threads. The rcu_nocbs boot parameter is used to define the set of CPUs to be offloaded.
Hotplug CPU -> cfg/hotplug_cpu.cfg Allows CPUs to be added to/removed from a live kernel. [HOTPLUG]

2.1.1 Boot Parameters

From [KERN-PARA]:


rcu_nocbs=      [KNL]
                In kernels built with CONFIG_RCU_NOCB_CPU=y, set
                the specified list of CPUs to be no-callback CPUs.
                Invocation of these CPUs' RCU callbacks will
                be offloaded to "rcuox/N" kthreads created for
                that purpose, where "x" is "b" for RCU-bh, "p"
                for RCU-preempt, and "s" for RCU-sched, and "N"
                is the CPU number.  This reduces OS jitter on the
                offloaded CPUs, which can be useful for HPC and
                real-time workloads.  It can also improve energy
                efficiency for asymmetric multiprocessors.

isolcpus=       [KNL,SMP] Isolate CPUs from the general scheduler.
                Format:
                <cpu number>,...,<cpu number>
                or
                <cpu number>-<cpu number>
                (must be a positive range in ascending order)
                or a mixture
                <cpu number>,...,<cpu number>-<cpu number>

                This option can be used to specify one or more CPUs
                to isolate from the general SMP balancing and scheduling
                algorithms. You can move a process onto or off an
                "isolated" CPU via the CPU affinity syscalls or cpuset.
                <cpu number> begins at 0 and the maximum value is
                "number of CPUs in system - 1".

                This option is the preferred way to isolate CPUs. The
                alternative -- manually setting the CPU mask of all
                tasks in the system -- can cause problems and
                suboptimal load balancer performance.

2.2 Cpu-isolation with partrt

A tool called partrt is included in the networking profile to divide an SMP Linux system into partitions. A description of the tool can be found in the Linux Real-Time Guide.

2.3 Latency Benchmarks

The cyclictest suite [CYCLIC] is a measurement of system latency used in many projects. As a comparison the measurement was applied to Enea Linux 6.0 Standard, and to Enea Linux Networking profile to investigate the impact of the changes to the system.

Below cyclictest is tested on the two different systems, average and maximum latency are presented in the tables below, first the test on the standard profile and after the results on the networking profile are shown. It is also combined with stress [STRESS] to show system performance under different type of loads.

    Command: Cyclictest with no stress
CPU # P I C_std Avg_std (us) Max_std (us) C_net Avg_net (us) Max_net (us)
0 99 1000 100000 9 24 100000 6 11
1 99 1500 66817 9 21 66812 6 12
2 99 2000 50208 9 23 50106 6 18
3 99 2500 40083 9 14 40082 6 10 
    Command: cyclictest with hdd stress:
    # stress -d 4 --hdd-bytes 1M &
CPU # P I C_std Avg_std (us) Max_std (us) C_net Avg_net (us) Max_net (us)
0 99 1000 100000 14 223 100000 11 77
1 99 1500 66820 14 231 66745 11 100
2 99 2000 50109 14 186 50055 11 76
3 99 2500 40083 14 176 40041 11 81 
    Command: cyclictest with vm stress:
    # stress -m 4 --vm-bytes 4096 &
CPU # P I C_std Avg_std (us) Max_std (us) C_net Avg_net (us) Max_net (us)
0 99 1000 100000 5 20 100000 3 15
1 99 1500 66818 6 14 66739 3 7
2 99 1500 50109 6 14 50103 3 9
3 99 1500 40079 6 14 40081 3 6 
    Command: cyclictest with full stress trial 1:
    # stress -c 4 -i 4 -m 4 --vm-bytes 4096 -d 4 --hdd-bytes 4096 &
CPU # P I C_std Avg_std (us) Max_std (us) C_net Avg_net (us) Max_net (us)
0 99 1000 99808 7 93 99815 6 58
1 99 1500 66733 9 105 66739 6 54
2 99 2000 50039 9 79 50039 7 61
3 99 2500 40016 10 83 40032 6 57 
    Command: cyclictest with full stress trial 2:
    # stress -c 4 -i 4 -m 4 --vm-bytes 4096 -d 4 --hdd-bytes 1M &
CPU # P I C_std Avg_std (us) Max_std (us) C_net Avg_net (us) Max_net (us)
0 99 1000 100000 13 201 100000 10 87
1 99 1500 66646 11 186 66685 9 85
2 99 2000 49969 10 195 49998 10 70
3 99 2500 39960 11 112 39992 10 90

3. USDPAA Usage

The need for predictive and good performance for networking systems is critical. One way of achieving greater performance is for user-space to avoid interactions with the kernel. The kernel is responsible for hardware acceleration allocation and scheduling. By using frameworks such as USDPAA[NXP-SDK] and DPDK[DPDK] control over certain hardware can be given to user-space. USDPAA is specific to NXP/Freescale's QoriQ platforms, for more information please see their guide to USDPAA [NXP-SDK].

In the guides below, an example of how to prepare a p2041rdb target with SRIO and ethernet through USDPAA is given.

3.1 Packages

The networking profile supports and includes all packages necessary for software support of USDPAA. If another image is created the below listed packages are relevant to include (all available in meta-fsl-ppc and meta-freescale) in order to add support for USDPAA and some example applications.

    * usdpaa
    * usdpaa-apps
    * fmc
    * fmlib
    * flib
    * eth-config

3.2 Prepare Target

The SRIO application needs us to boot with a RCW and board configuration that allows usage of the PCI extender port. The below examples are specific to p2041rdb, but similar steps can be taken for other targets where SRIO is not enabled by default.

3.2.1 Reset Control Word (RCW)

The reset control word must configure the serial-deserializer (SERDES) bus for SRIO. This can be done by either a predefined binary/setting, or can be created in Code Warrior [CW].

The RCW used in this example was given in the NXP/Freescale SRA User Guide of [NXP-SDK].

To program the RCW to target from u-boot follow the steps below:

=> tftp 1000000 <path-2-file>/RR_RS_0x02.bin

=> md 0xec000000
ec000000: aa55aa55 010e0100 12600000 00000000 .U.U.....`......
ec000010: 241c0000 00000000 248e40c0 c3c02000 $.......$.@... .
ec000020: de800000 40000000 00000000 00000000 ....@...........
ec000030: 00000000 d0030f07 00000000 00000000 ................
ec000040: 00000000 00000000 091380c0 000009c4 ................
ec000050: 09000010 00000000 091380c0 000009c4 ................
ec000060: 09000014 00000000 091380c0 000009c4 ................
ec000070: 09000018 81d00000 091380c0 000009c4 ................
ec000080: 890b0050 00000002 091380c0 000009c4 ...P............
ec000090: 890b0054 00000002 091380c0 000009c4 ...T............
ec0000a0: 890b0058 00000002 091380c0 000009c4 ...X............
ec0000b0: 890b005c 00000002 091380c0 000009c4 ...\............
ec0000c0: 890b0090 00000002 091380c0 000009c4 ................
ec0000d0: 890b0094 00000002 091380c0 000009c4 ................
ec0000e0: 890b0098 00000002 091380c0 000009c4 ................
ec0000f0: 890b009c 00000002 091380c0 000009c4 ................

=> protect off 0xec000000 +$filesize
Un-Protected 1 sectors
=> erase 0xec000000 +$filesize

. done
Erased 1 sectors

=> cp.b 1000000 0xec000000 $filesize
Copy to Flash... 9done

=> protect on 0xec000000 +$filesize
Protected 1 sectors

=> md 0xec000000
ec000000: aa55aa55 010e0100 12600000 00000000 .U.U.....`......
ec000010: 241c0000 00000000 088040c0 c3c02000 $.........@... .
ec000020: de800000 40000000 00000000 00000000 ....@...........
ec000030: 00000000 d0030f07 00000000 00000000 ................
ec000040: 00000000 00000000 091380c0 000009c4 ................
ec000050: 09000010 00000000 091380c0 000009c4 ................
ec000060: 09000014 00000000 091380c0 000009c4 ................
ec000070: 09000018 81d00000 091380c0 000009c4 ................
ec000080: 890b0050 00000002 091380c0 000009c4 ...P............
ec000090: 890b0054 00000002 091380c0 000009c4 ...T............
ec0000a0: 890b0058 00000002 091380c0 000009c4 ...X............
ec0000b0: 890b005c 00000002 091380c0 000009c4 ...\............
ec0000c0: 890b0090 00000002 091380c0 000009c4 ................
ec0000d0: 890b0094 00000002 091380c0 000009c4 ................
ec0000e0: 890b0098 00000002 091380c0 000009c4 ................
ec0000f0: 890b009c 00000002 091380c0 000009c4 ................
=>

In order to obtain specific hardware settings, some multiplexers need to be set. Descriptions of these can be obtained by typing cpld -h in u-boot, information about the peripherals are also available in the board's respective user guide. For SRIO on the p2041rdb target the following settings are necessary.

cpld_cmd lane_mux 6 0
cpld_cmd lane_mux a 0
cpld_cmd lane_mux c 0
cpld_cmd lane_mux d 0

3.3 Device Trees

USDPAA enabled targets have very specific device trees, this is because instead of handing over hardware to the linux kernel, it is managed by the DPAA framework.

The networking profile includes a custom device-tree (-usdpaa-enea.dtb), currently defined for p2041rdb. It gives one ethernet interface to the kernel while the rest belongs to DPAA.

Available and tested device-trees for p2041rdb:

3.4 Boot Parameters

USDPAA demands some custom boot arguments. If not given, or if given improperly the USDPAA applications will not be usable. The NXP/Freescale manual covers these arguments, however might be misdirecting since the documentation in several places are, rather than target agnostic, specific instructions that are only applicable to certain targets. If unsure, one can consult the benchmarking chapter of the NXP/Freescale SDK documentation that include more exact steps per tested targets.

For our purposes of testing SRIO and the reflector application we only need the 'usdpaa_mem' boot argument. If the reserved memory is too large, it will cause segmentation faults.

    Table 4.1 'usdpaa_mem=?'
TARGET usdpaa_mem
p2041rdb =< 64M

3.5 Boot instructions P2041RDB

Below are instructions on how to boot a p2041rdb board with usdpaa enabled.

3.5.1 Boot over NFS server

tftp 1000000 uImage-p2041rdb.bin
tftp c00000 uImage-p2041rdb-usdpaa-enea.dtb

setenv bootargs root=/dev/nfs
nfsroot=172.21.3.8:/unix/enea_linux_rootfs/<folder-path> rw ip=dhcp
console=ttyS0,115200 memmap=16M$0xf7000000 mem=4080M max_addr=f6ffffff
usdpaa_mem=64M

bootm 1000000 - c00000

3.5.2 RAM Boot

tftp 1000000 uImage-p2041rdb.bin
tftp 2000000 uImage-p2041rdb-usdpaa.dtb
tftp 5000000 enea-image-networking-p2041rdb.ext2.gz.u-boot

setenv bootargs root=/dev/ram rw console=ttyS0,115200 ramdisk_size=10000000
log_buf_len=128K usdpaa_mem=64M

bootm 0x1000000 0x5000000 0x2000000

3.6 Run Reflector

Reflector is a demo application from NXP/Freescale that through ethernet recieves a package and sends it back to with switched source-destination IP addresses switched.

In order to test reflector an ethernet cable must be connected between either two p2041rdb targets or between a work-PC and the p2041rdb target.

Connect two targets by ethernet and do the following steps to test connection with usdpaa:

  1. Boot Board A with uImage-p2041rdb-uspdaa-enea.dtb
  2. Boot Board B with uImage-p2041rdb.dtb
  3. On board A do the following:
        # Configure what ethernet ports that should be used by dpaa
        $ vi config-p2041rdb.xml
        <cfgdata>
          <config>
            <engine name="fm0">
              <port type="MAC" number="2"
policy="hash_ipsec_src_dst_spi_policy1"/>
              <port type="MAC" number="3"
policy="hash_ipsec_src_dst_spi_policy2"/>
              <port type="MAC" number="4"
policy="hash_ipsec_src_dst_spi_policy3"/>
              <port type="MAC" number="5"
policy="hash_ipsec_src_dst_spi_policy4"/>
            </engine>
          </config>
        </cfgdata>

        $ fmc -c config-p2041rdb.xml -p /usr/etc/usdpaa_policy_hash_ipv4.xml -a

        # start reflector and check the ethernet ports:
        $ reflector
        reflector> ifconfig
  1. On board B do the following:
        # Configure the network interface to which you connected the ethernet
          cable, by choosing an ip- and MAC address.
        $ ifconfig -a
        $ ifconfig <eth-x> 192.168.0.10 netmask 255.255.255.0 up

        # Configure to connect to the hw address
        $ arp -s 192.168.0.11 <hw-address> -i <eth-x>

        # Ping Board A
        $ ping 192.168.0.11

3.7 Run SRA

The user space drivers from NXP/Freescale support usage of SRIO from linux user space. The SRA application is a demo application from NXP/Freescale that can implement writing from one SRIO interface to another, avioding kernel interaction by using DMA (direct memory access) memory management. More information on the drivers can be found in the NXP/freescale SDK [NXP-SDK].

In the test run below two boards are prepared with SRIO interfaces. Both boards are initialized with a memory setting that sets the different SRIO memory spaces to different values. Then data from Board A's Write-prepapration space is written to Board B's Map space.

        # start the srio application
        $ sra

        # setup board B (receiver)
        sra> sra -attr port1 win_attr 1 nwrite nread

        # set local memory to data predefined in 0x100000
        sra> sra -op port1 1 0 0 s 0x100000

        # read to view what is written to port 1
        sra> sra -op port1 1 0 0 p 0x100000


        # setup board A (transmitter)
        sra> sra -attr port1 win_attr 1 nwrite nread

        # set local memory and then read to confirm
        sra> sra -op port1 1 0 0 s 0x100000
        sra> sra -op port1 1 0 0 p 0x100000

        # write what is in 'write preparing space' to outbound
        sra> sra -op port1 1 0 0 w 0x100000

        # confirm on board B that 'write preparation space' from board A is written in
        # 'map space'
        sra> sra -op port1 1 0 0 p 0x100000

          Board A                                 Board B
        +-------------------+                   +-------------------+
        | Map space         |           +------>| Map space         |
        +-------------------+           |       +-------------------+
        | Read data space   |           |       | Read data space   |
        +-------------------+  outbound |       +-------------------+
        | write preparation |-----------+       | write preparation |
        | space             |                   | space             |
        +-------------------+                   +-------------------+
        | reserved space    |                   | reserved space    |
        +-------------------+                   +-------------------+

Image 3.1 SRIO memory space

3.8 Throughput using USDPAA

To show some of the power of using the USDPAA framework a test of the throughput over a 10G ethernet link was tested and the results are presented below.

The tool used for performance measuring was Spirent Test Center which is used as a packet generator along with the “Spirent Test Center” application, version 4.33. The test targets are connected to the Spirent Test Center through 10G Ethernet ports(XAUI-RISER card). On Enea Linux 6.0 a the USDPAA reflector was used as a packet forwarding application. The resulting Throughput performance measured with spirent can be seen in image 3.2 below.

Drawing 
Image 3.2 Throughput on p2041rdb over a 10G ethernet port using USDPAA.
x-axis shows the frame size in bytes, and y-axis the aggregated throughput in
megabits per second.

[NXP-SDK] QorIQ SDK 1.9 Documentation - https://freescale.sdlproducts.com/LiveContent/web/pub.xql?c=t&action=home&pub=QorIQ_SDK_1.9&lang=en-US#addHistory=true&filename=GUID-81837065-81AD-449B-8572-E96C3EED636F.xml&docid=GUID-81837065-81AD-449B-8572-E96C3EED636F&inner_id=&tid=&query=&scope=&resource=&toc=false&eventType=lcContent.loadHome

[KERN-PARA] Kernel Parameters, https://www.kernel.org/doc/Documentation/kernel-parameters.txt

[RCU] Paul McKenney, Priority-Boosting RCU Read-Side Critical Section, https://lwn.net/Articles/220677/

[HOTPLUG] https://www.kernel.org/doc/Documentation/cpu-hotplug.txt

[DPDK] http://dpdk.org/

[CYCLIC] https://rt.wiki.kernel.org/index.php/Cyclictest

[STRESS] http://linux.die.net/man/1/stresshttp://linux.die.net/man/1/stress

[CW] http://www.nxp.com/products/software-and-tools/software-development-tools/codewarrior-development-tools:CW_HOME