\section{10Gbit/s path tests}
During SuperComputing 2003\footnote{SC2003: \url{http://www.sc-conference.org/sc2003/}}, we made some tentative TCP performance measurements on 10Gbps links between hosts at the SLAC/FNAL booth at the Phoenix convention center and a host at the SLAC/Stanford Point of Presence at the Palo Alto Internet eXchange (PAIX) and hosts at StarLight in Chicago. Due to the short amount of time we had access to these links ($\sim$ 3 days) and the emphasis on demonstrating the maximum throughput for the SC03 Bandwidth Challenge these measurement are necessarily incomplete, however some of the tentative results are felt to be worth reporting. 

\subsection{Setup}
All the hosts at Phoenix and PAIX were Dell 2650s with dual Xeon CPUs, a 533MHz front side bus, each with an Intel PRO/10GbE LR Network Interface Card (NIC) plugged into the 133MHz 64 bit PCI-X bus slot. There were 3 hosts at the SLAC/FNAL booth at SC03, two with 3.06GHz CPUS, and the third with 2.04GHz CPUs. The host at PAIX had dual Xeon 3.06GHz CPUS, a 533MHz front side bus, and an Intel PRO 10GbE Ethernet connection (level 2) to a Cisco HPR router in LA. From the HPR router the signal was transmitted via an OC192/POS circuit to a Juniper router managed by SCInet at the Phoenix Convention Center. From the Juniper the path went via a Force 10 E1200 router to the Cisco 6509 in the SLAC/FNAL booth.

The StarLight/Chicago path from the booth went via the Force 10 router through a second Juniper router at 10Gbps to the Abeline core and thence via Abilene routers at Los Angeles, Sunnyvale, Kansas City, and Indianapolis to Chicago.

\subsection{Methodology}
We set up the sending hosts at SC03 with the Caltech Fast TCP stack, and the DataTAG altAIMD stack  \footnote{Available at: \url{http://www.hep.ucl.ac.uk/~ytl/tcpip/linux/altaimd/}} that allowed dynamic (without reboot) selection of the standard Linux TCP stack (New Reno with Fast re-transmit), the Manchester University implementation of the High Speed TCP (HS-TCP) and the Cambridge University Scalable TCP stack (S-TCP). By default we set the Maximum Transfer Unit (MTU) to 9000Bytes and the transmit queue length (txqueuelen) to 2000 packets.

We started the measurements just before midnight on Wednesday 19th November. This was a reasonably controlled set of measurements with little cross-traffic. Each test was for 1200 seconds, with a given stack, and fixed maximum window size. 


\subsubsection{Tests to PAIX (17ms RTT)}
On the Phoenix to PAIX link we used maximum window sizes of 8MBytes, 16MBytes and 32MBytes\footnote{More explanations are available on: \url{http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2003/hiperf.html} and \url{http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2003/stack-cf.html}}. This bracketed the nominal optimum window size calculated from the BDP of 17ms * 10Gbps  $\sim$ 20MBytes. For the PAIX link, all the tests were made with a single TCP stream. For Reno, HS-TCP and S-TCP there was little observable differences between the four stacks in the achievable bandwidth behavior.

\begin{itemize}
\item For an MTU of 9000Bytes, a window size of 8MBytes was inadequate and resulted in throughput being limited to about 2.9Gbps for Reno single stream, HS-TCP and S-TCP.
\item For an MTU of 9000Bytes with window sizes of 16MBytes or 32MBytes for Reno single stream, HS-TCP and S-TCP the throughput increased to about 4.3Gbps. 
\item The throughputs with an MTU of 9000Bytes were very stable. Typical values of the Stability Index (standard deviation / average throughput) were $<$ 0.4\%. The larger values for Reno with a single stream and an 8MByte window and HS-TCP with a 16MByte window were each caused by a single sudden drop of throughput for one 5 second period.
\item Reducing the MTU from 9000Bytes to 1500Bytes reduced the throughput for a 16MByte window from 4.3Gbps to about 700Mbps, and for an 8MByte window from 2.9Gbps to $\sim$ 360Mbps. Also the throughputs were less stable with the 1500Byte MTU (Stability Index $>$ 10\%).
\item With an MTU of only 1500Bytes, Fast TCP gave similar performance to HS-TCP and S-TCP when they ran with 1500Byte MTUs.  
\end{itemize}

The 4.3Gbps limit was slightly less than the $\sim$ 5.0Gbps achieved with UDP transfers in the lab at SLAC between back to back 3.06GHz Dell PowerEdge 2650 hosts. On the other hand it is less than that calculated from the expected data transfer rate for a 10GE NIC with a 64 bit 133 MHz PCI-X bus\footnote{See Figure 3.1 of Perfrormance of Gigabit and 10 Gigabit Ethernet NICs with Server Quality Motherboards by R Hughes-Jones, P. Clarke, S. Dallisom and G. Fairey, to be published}. The limitation in throughput is believed to be due to CPU factors (CPU speed, memory/bus speed or the I/O chipset). The relative decrease in throughput going from 9000Byte MTU to a 1500Byte MTU was roughly proportional to the reduction in MTU size. This maybe related to the extra cpu spped / memory bandwidth required to process the 6 times as many, but 6 times as small MTUs. Back to back UDP transfers in the lab at SLAC between 3.06GHz Dell PowerEdge 2650 hosts achieved about 1.5Gbps or about twice the 700Mbps achieved with the SC03 long distance TCP transfers. Further work is required to understand this discrepancy.



