SLAC logo

Drop in Throughput between SLAC and UMich, NCSA, UFL and Internet2

Warren Matthews. Page created: November 11, 2002. Last updated November 21, 2002.

Central Computer Access | Computer Networking | Network Group | ICFA-NTF Monitoring 		SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook
Still under construction. Completion RSN

A drop in performance was reported between SLAC and the IEPM-BW target nodes at the University of Michigan (Umich), the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana Champaign and the University of Florida (UFL).

The following graph shows the throughput measured by iperf between SLAC and NCSA, SLAC and UFL and between SLAC and the IEPM-BW target node at the Internet2 office (Internet2) in Ann Arbor, MI. Measurements to Umich are not shown because the variation is large and the effect is not clearly seen. Routing to both Umich and Internet2 is via MichNet, the Michigan Regional Network.

It appears throughput between SLAC and UFL dropped significantly around midnight (Pacific time) on November 1. Throughput between SLAC and NCSA dropped around 7pm. Throughput between SLAC and Internet2 also dropped around 7pm, but recovered early the next morning. Measurements to UFL and NCSA followed a pattern characteristic of a congested network for several days until the previous performance was restored on November 8.

The throughput was compared with traceroutes to these sites. A clear correlation between a route change and the poorer performance could be seen.

Each distinguishable route is allocated a route number. The following graph shows the route number and the throughput achieved with iperf between SLAC and NCSA. The dominant route before the change is route number 19, after is route number 23. Route 19 was restored along with the better performance on November 7.

The full routes before and after along with the minor routes are shown in the following table. Routes 21 is incomplete but appears to be the same as Route 19. Routes 24 and 29 are incomplete but appear to be the same as Route 23.

192321242629
rtr-core1-iepm.slac.stanford.edu rtr-core1-iepm.slac.stanford.edu rtr-core1-iepm.slac.stanford.edu
rtr-dmz1-ger.slac.stanford.edu rtr-dmz1-ger.slac.stanford.edu rtr-dmz1-ger.slac.stanford.edu
i2-gateway.stanford.edu i2-gateway.stanford.edu 192.68.191.146
stan.pos.calren2.net stan.pos.calren2.net snv-pos-slac.es.net
sunv--stan.pos.calren2.net sunv--stan.pos.calren2.net snva-esnet.abilene.ucaid.edu
abilene--qsv.pos.calren2.net c2-qanh-gsr-qsv-gsr.atm.calren2.net dnvr-snva.abilene.ucaid.edu
snvang-snva.abilene.ucaid.edu usc--qanh.pos.calren2.net kscy-dnvr.abilene.ucaid.edu
kscyng-snvang.abilene.ucaid.edu abilene--usc.atm.calren2.net ipls-kscy.abilene.ucaid.edu
iplsng-kscyng.abilene.ucaid.edu losang-losa.abilene.ucaid.edu clev-ipls.abilene.ucaid.edu
abilene-charlie.gw.ncsa.edu snvang-losang.abilene.ucaid.edu 192.122.183.9
acb-1-vlan100.gw.ncsa.edu kscyng-snvang.abilene.ucaid.edu pos0-0-0.ucaid.mich.net
niobe.ncsa.uiuc.edu iplsng-kscyng.abilene.ucaid.edu thunderbird.internet2.edu
  abilene-charlie.gw.ncsa.edu  
  acb-1-vlan100.gw.ncsa.edu  
  niobe.ncsa.uiuc.edu
 

Where:
snva = sunnyvale, snvang = sunnyvale next generation, kscyng = kansas city next generation, iplsng = indianapolis next generation, losa = los angeles, losang = los angeles next generation

It appears the detour via Los Angeles (in bold) is responsible for the hit. 


> > Hello Paola,
> > My name is Caroline Carver and I have been assigned to your
> > inquiry into the routing change regarding UMICH, NCSA and
> > UFL.  Can you send me a trace route from your site to each of
> > these and let me do some investigating on my end.
> > Thank you,
> > Caroline

> We just checked our routes now and they have bounced back to
> Kansas City as of this morning.
>
> Our issue is: was there an announced changed in route
> policies we missed?
> Do you have any idea what made bounce back as of this
> morning? And why for a week it was going via LA?
Also just as performance to the above recovered, a similar detour afflicted the node at Internet2 

rtr-core1-pub6                      rtr-core1-iepm.slac.stanford.edu
rtr-dmz1-ger                        rtr-dmz1-ger.slac.stanford.edu
i2-gateway.stanford.edu             i2-gateway.stanford.edu
STAN.POS.calren2.NET                stan.pos.calren2.net
SUNV--STAN.POS.calren2.net          sunv--stan.pos.calren2.net
C2-QANH-GSR-QSV-GSR.ATM.calren2.net abilene--qsv.pos.calren2.net
USC--QAnh.POS.calren2.net           snvang-snva.abilene.ucaid.edu
Abilene--USC.ATM.calren2.net        kscyng-snvang.abilene.ucaid.edu
losang-losa.abilene.ucaid.edu       iplsng-kscyng.abilene.ucaid.edu
snvang-losang.abilene.ucaid.edu     ipls-iplsng.abilene.ucaid.edu
kscyng-snvang.abilene.ucaid.edu     clev-ipls.abilene.ucaid.edu
iplsng-kscyng.abilene.ucaid.edu     so-0-2-0x1.aa1.mich.net
so-0-2-0x1.aa1.mich.net             pos0-0-0.ucaid.mich.net
pos0-0-0.ucaid.mich.net             thunderbird.internet2.edu
thunderbird.internet2.edu   



On Tue, 12 Nov 2002, Laurence Kirchmeier wrote:

> Merit didn't make a change, Abilene did. The Cleveland Abilene node is
> being
> decommissioned so our Ann Arbor I2 connection has been moved to the new
> IPLSng node ( as of 11/9/02 ).
the route to thunderbird.internet2.edu is via Stanford (=32), Calren (north=11422, south=11422) abilene (=11537) and Michnet (=237) 
> rtr-dmz1>sho ip bgp 207.75.164.95
> BGP routing table entry for 207.75.164.0/23, version 2888391
> Paths: (1 available, best #1, table Default-IP-Routing-Table)
>   Not advertised to any peer
>   32 11423 11422 11537 237, (received & used)
>     192.68.191.83 from 192.68.191.83 (171.64.0.71)
>       Origin IGP, localpref 100, valid, external, best
The route to NCSA is the restored one, NCSA has it's own AS (=1224) and is directly peered to Abilene. 
rtr-dmz1>sho ip bgp 141.142.36.38
BGP routing table entry for 141.142.0.0/16, version 2829435  
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  32 11423 11537 1224, (received & used)
    192.68.191.83 from 192.68.191.83 (171.64.0.71)
      Origin IGP, localpref 100, valid, external, best
However, U of Michigan is advertised via Michnet, so you'd think it would have the same route as the internet2 node, however, it routes directly from calren-north to Abilene. 
rtr-dmz1>sho ip bgp 141.211.101.122
BGP routing table entry for 141.211.0.0/16, version 2888401
Paths: (1 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  32 11423 11537 237, (received & used)
    192.68.191.83 from 192.68.191.83 (171.64.0.71)
      Origin IGP, localpref 100, valid, external, best