NIC performance with iperf
FYI. Windows networking sucks, across all versions of Windows. (At least, at Gigabit speeds, it does.) Between my Windows 7 PC (3.06Ghz HT Pentium 4, Intel PRO 1000/MT Desktop Adapter [82540EM]) and Gentoo Linux server (2 x 2.2GHz Xeon [P4 core], PCI-X [64-bit,66Mhz] Intel PRO 1000/MT Dual-Port Server Adapter [82546EB]) Using "iperf -s" and "iperf -c 192.168.x.x", I get - consistently - between 238Mbps and 250Mbps **in both directions**. Interesting to note is that the Windows 7 PC is CPU-limited on both sending and receiving, despite having all the offload and acceleration features turned on. Simultaneously, the Gentoo box sees a spike on one CPU... to 1%. In UDP traffic, however, I see a startling asymmetry: PC-to-Linux can only send UDP at ~50Mbps, Linux-to-PC sends UDP at ~475Mbps. WTF? I also tried the Parallel option, which runs multiple streams in parallel. Results are consistent, aggregate bandwidth is still only about 260Mbps. Two streams pegs both logical (HT) CPUs on the client at ~95% utilization. I tried enabling and disabling both the Windows Firewall and my AV software, which made essentially no difference at all. Then I also tried running the same set of tests from a Windows 2003 R2 server (2 x 2.13GHz Athlon MP, PCI-X [64-bit,66Mhz] Broadcom NetXtreme [BCM57xx]) N.B. this is the same model of motherboard Kevin is using in his Linux server. and observed slightly better throughput to the Linux server: about 380Mbps. Still better was Linux-to-W2K3R2, at about 440Mbps. UDP performance was similar to Kevin's results, at 103Mbps W2K3-to-Linux & 950Mbps Linux-to-W2K3. CPU utilization on the W2K3R2 box exhibited similar properties as the Win7 client, although none of the tests actually managed to max out either CPU, instead peaking at about 70% utilization. Just for kicks, I ran a test between a RHEL5 VM (under VMWare ESXi 4.0U1, VMXNET adapter) and my Linux server... 920+ Mbps every time! Of course, that number may be suspect since the UDP test reports speeds of 45.3Gbps :-). Conclusion: the culprit is the Windows networking stack. Note that W2K3R2 is supposed to support zero-copy TCP offload, theoretically allowing network communications at wirespeed with extremely low latency. So much for that idea... -Adam Thompson <athompso@athompso.net> (204) 291-7950
Hmm. Now I'll have to do some tests at work next week using various combinations of OS and VMware ESX/vSphere. I'll get back to you. :) On Fri, Apr 16, 2010 at 3:36 PM, Adam Thompson <athompso@athompso.net>wrote:
FYI. Windows networking sucks, across all versions of Windows. (At least, at Gigabit speeds, it does.)
Between my Windows 7 PC (3.06Ghz HT Pentium 4, Intel PRO 1000/MT Desktop Adapter [82540EM]) and Gentoo Linux server (2 x 2.2GHz Xeon [P4 core], PCI-X [64-bit,66Mhz] Intel PRO 1000/MT Dual-Port Server Adapter [82546EB])
Using "iperf -s" and "iperf -c 192.168.x.x", I get - consistently - between 238Mbps and 250Mbps **in both directions**.
Interesting to note is that the Windows 7 PC is CPU-limited on both sending and receiving, despite having all the offload and acceleration features turned on. Simultaneously, the Gentoo box sees a spike on one CPU... to 1%.
In UDP traffic, however, I see a startling asymmetry: PC-to-Linux can only send UDP at ~50Mbps, Linux-to-PC sends UDP at ~475Mbps. WTF?
I also tried the Parallel option, which runs multiple streams in parallel. Results are consistent, aggregate bandwidth is still only about 260Mbps. Two streams pegs both logical (HT) CPUs on the client at ~95% utilization.
I tried enabling and disabling both the Windows Firewall and my AV software, which made essentially no difference at all.
Then I also tried running the same set of tests from a Windows 2003 R2 server (2 x 2.13GHz Athlon MP, PCI-X [64-bit,66Mhz] Broadcom NetXtreme [BCM57xx]) N.B. this is the same model of motherboard Kevin is using in his Linux server. and observed slightly better throughput to the Linux server: about 380Mbps. Still better was Linux-to-W2K3R2, at about 440Mbps. UDP performance was similar to Kevin's results, at 103Mbps W2K3-to-Linux & 950Mbps Linux-to-W2K3.
CPU utilization on the W2K3R2 box exhibited similar properties as the Win7 client, although none of the tests actually managed to max out either CPU, instead peaking at about 70% utilization.
Just for kicks, I ran a test between a RHEL5 VM (under VMWare ESXi 4.0U1, VMXNET adapter) and my Linux server... 920+ Mbps every time! Of course, that number may be suspect since the UDP test reports speeds of 45.3Gbps :-).
Conclusion: the culprit is the Windows networking stack. Note that W2K3R2 is supposed to support zero-copy TCP offload, theoretically allowing network communications at wirespeed with extremely low latency. So much for that idea...
-Adam Thompson <athompso@athompso.net> (204) 291-7950
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
I installed a cheap GigE switch for a customer using 2006-era P4 Dell desktops, running XP with cheap Broadcom GigE NICs. Constant 600-700mbit for file transfers. CPU I think was ~30%, but that may have included the AV scanning the file. I assumed Iperf is developed/optimized natively under *NIX and the Windows version is compiled under Cygwin, do you have any indication that it's optimized?
I installed a cheap GigE switch for a customer using 2006-era P4 Dell desktops, running XP with cheap Broadcom GigE NICs. Constant 600-700mbit for file transfers. CPU I think was ~30%, but that may have included the AV scanning the file.
I assumed Iperf is developed/optimized natively under *NIX and the Windows version is compiled under Cygwin, do you have any indication that it's optimized?
I'm certain iperf is NOT optimized for Windows. However, empirically I can determine using application-level tests (i.e. FTP, SAMBA, etc.) that the numbers I'm seeing are consistent. Worth mentioning is that Microsoft shipped "ttcp.exe" on most versions of Windows XP install media, under VALUEADD\NET\something, IIRC. Presumably that binary is reasonably-well optimized. I recall using it to test between Windows XP and Solaris, and seeing half-decent speeds but nothing wonderful (~400Mbps). Equally, I know that I too, have seen lesser Windows clients talking to Windows servers at higher speeds. It's also worth noting that SMB traffic to my Linux box is noticeably slower than it was to the Sun Ultra Enterprise 450 it replaced. While there's obviously something wrong with the networking stack on my Windows 7 PC (100% CPU utilization just to do TCP?!) it's also possible that there's some specific interaction between the Linux and Windows stacks. As I demonstrated, Linux-to-linux works great. Kevin demonstrated that MacOSX-Linux works great. Many of us have anecdotal evidence that Windows-Windows works well, and several people have recently demonstrated that Windows-Linux does NOT work well. And I've personally demonstrated (although I haven't posted any numbers here) that Windows-Solaris (*not* OpenSolaris, if it makes any difference) works better than Windows-Linux. (OK, I engaged in a bit of Windows-bashing when I blamed the Windows networking stack *exclusively*.) I don't have any desire to set up a test lab in my living room just to try out the various combinations... how much effect can the GigE switch have on performance? I'm using dirt-cheap D-Link 5-port switches if that makes any difference. -Adam Thompson <athompso@athompso.net> (204) 291-7950
Isn't Windows still running a BSD decendent IP stack? Just thought I'd troll a little ;) John Sent from my BlackBerry device on the Rogers Wireless Network -----Original Message----- From: "Adam Thompson" <athompso@athompso.net> Date: Sat, 17 Apr 2010 14:34:39 To: 'Continuation of Round Table discussion'<roundtable@muug.mb.ca> Subject: Re: [RndTbl] NIC performance with iperf
I installed a cheap GigE switch for a customer using 2006-era P4 Dell desktops, running XP with cheap Broadcom GigE NICs. Constant 600-700mbit for file transfers. CPU I think was ~30%, but that may have included the AV scanning the file.
I assumed Iperf is developed/optimized natively under *NIX and the Windows version is compiled under Cygwin, do you have any indication that it's optimized?
I'm certain iperf is NOT optimized for Windows. However, empirically I can determine using application-level tests (i.e. FTP, SAMBA, etc.) that the numbers I'm seeing are consistent. Worth mentioning is that Microsoft shipped "ttcp.exe" on most versions of Windows XP install media, under VALUEADD\NET\something, IIRC. Presumably that binary is reasonably-well optimized. I recall using it to test between Windows XP and Solaris, and seeing half-decent speeds but nothing wonderful (~400Mbps). Equally, I know that I too, have seen lesser Windows clients talking to Windows servers at higher speeds. It's also worth noting that SMB traffic to my Linux box is noticeably slower than it was to the Sun Ultra Enterprise 450 it replaced. While there's obviously something wrong with the networking stack on my Windows 7 PC (100% CPU utilization just to do TCP?!) it's also possible that there's some specific interaction between the Linux and Windows stacks. As I demonstrated, Linux-to-linux works great. Kevin demonstrated that MacOSX-Linux works great. Many of us have anecdotal evidence that Windows-Windows works well, and several people have recently demonstrated that Windows-Linux does NOT work well. And I've personally demonstrated (although I haven't posted any numbers here) that Windows-Solaris (*not* OpenSolaris, if it makes any difference) works better than Windows-Linux. (OK, I engaged in a bit of Windows-bashing when I blamed the Windows networking stack *exclusively*.) I don't have any desire to set up a test lab in my living room just to try out the various combinations... how much effect can the GigE switch have on performance? I'm using dirt-cheap D-Link 5-port switches if that makes any difference. -Adam Thompson <athompso@athompso.net> (204) 291-7950 _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (4)
-
Adam Thompson -
Colin Stanners -
john＠johnlange.ca -
Kevin McGregor