* The 125Mbits/s limitation was caused by the TCP windows/streams setting being
non optimal for this path (see below).
The hostnames are anonymized for privacy reasons.
It can be seen that the throughput to only CESnet and IN2P3 appear to be
affected. In some cases (e.g. UK, and NL) this is since the route did not
change, in other cases (e.g. IT) maybe the throughput is not large enough to
notice a change in bottleneck.
The large variation in the SWITCH.CH case are due to large variations (due
in turn to cron jobs) in the load on the 1.switch.ch host.
After the change, the iperf throughput from
SLAC to CESnet in Prague, Czech Republic
dropped from about
250 Mbits/s to about 150 Mbits/s, but the ping RTT stayed constant.
route from SLAC to CESNET reduced from 17 to 13 hops (that is good),
the last 6 hops were
identical in both cases, the changes were for
hops 3-11 for the Internet2 route and hops 3-7 for the ESnet route.
The route from
CESNET to SLAC after the change
are pretty symmetric with the route from SLAC to CESNET.
In the case of SLAC to IN2P3 in Lyon France, the
iperf throughput dropped from about 125
Mbits/s to about 60 Mbits/s. Also the ping RTT increased from about
150ms to about 160ms.
The traceroutes before and after show
the number of hops changing from 17 to 15 and the
first 2 and last
2 hops being identical. In the former case (Internet2) the peering from
Abilene/Internet2 was to GEANT, after the routing change ESnet handed
off the packets to opentransit. I would have exepected IN2P3 to have been
routed by GEANT and RENATER so perhaps opentransit is a backup route.
We sent email to email@example.com describing the problem, and also to
Anne Marie Lutz of IN2P3 identifying the possible use of a backup route.
Mike O'Connor of ESnet responded via Email6/12/03 at 11:07am PDT:
ESnet had been filtering a GEANT route to ccsvsn04.in2p3.fr. The
126.96.36.199/16 route from GEANT is now being accepted in NY.
Since the route was corrected the IPERF times have recovered to the
previous performance levels for node1.in2p3.fr.
The previous path traversed an opentransit peering in Chicago with the
shaping set to:
vbr peak 135m sustained 67m burst 100
The proper path via GEANT peers at OC48 in NYC.
The changes are nicely illustrated by the
ABwE available bandwidth
tool results which are updated every minute.
The iperf configuration was 8 parallel streams with 1024KByte window.
It appears that though this worked well for the Internet2 path, it is
much less than optimal for the ESnet path. This is also confirmed
by the ABwE results
which indicated that available bandwidth increased when we moved the path
from Abilene to ESnet (note the step up on the blue line from about 200
Mbits/s to 280Mbits/s), at around 4pm. Noting that the single stream
iperf (measured by IEPM-BW with a 1024KByte window)
was about 7Mbits/s and we want to achieve about 280MBits/s, we tried
48 streams with a 256KByte window and achieved about 300Mbits/s.
We therefore used tcpload.pl to scan through various window sizes
and streams to find the
optimal configuration and used 30 streams
and a 512 kByte window.
When we switched over from Internet2 to ESnet, the ESnet route to IN2P3 was
less than optimal and had rate limiting on it. This was fixed by ESnet
peering directly with GEANT for the IN2P3 path.
The ESnet route to CESnet was very good, however the measurement tool (iperf)
needed re-configuring (change the window and streams) to work well on the
new path. ABwE was very valuable in identifying this since it showed
immediately that the new path via ESnet had similar available
bandwdith to the Internet2 path.
Page owner: Les Cottrell