linux-next network throughput performance regression

[Simon Xiao]

I compared the network throughput performance on SLES12 bare metal servers, between SLES12 default kernel and latest linux-next (2015-11-05) kernel, based on the test results, I suspect there is a network regression exists on Linux-Next over the 40G Ethernet network:
a) iperf3 reports 50% performance drop with single TCP stream on latest linux-next;
b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams on latest linux-next;
Another throughput benchmarking tool (ntttcp-for-linux) test result is also listed at the end of the email for reference.


Server configuration:
------------------------------
Two servers (one client and one server, cross linked by 40G Ethernet), which have:
a) CPU: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 2 sockets, 16 CPUs, cache size : 20480 KB
b) Memory: 64 GB
c) Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro], 40G Ethernet, default driver


Test with iperf3:
------------------------------
iperf3: https://github.com/esnet/iperf

a) SLES12 default kernel, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128	
Throughput (G bps)	36.7	37.3	37.6	37.7	37.7	37.7	37.7	25.7		

b) SLES12 + Linux-Next 20151105, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128
Throughput (G bps)	18.2	32.2	34.6	32.8	27.6	32.0	27.0	21.3
Percentage dropped	-50%	-14%	-8%	-13%	-27%	-15%	-28%	-17%


Test with ntttcp-for-linux:
------------------------------
ntttcp-for-linux: https://github.com/Microsoft/ntttcp-for-linux 

a) SLES12 default kernel, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512
Throughput (Gbps)	36.19	37.29	37.67	37.68	37.7	37.72	37.74	37.76	37.81	37.9	

b) SLES12 + Linux-Next 20151105, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512	
Throughput (Gbps)	28.12	34.01	37.6	36.53	32.94	33.07	33.63	33.44	33.83	34.42	
Percentage dropped	-22%	-9%	0%	-3%	-13%	-12%	-11%	-11%	-11%	-9%