SLAC logo

The influence of strong LAN traffic on target hosts on monitoring results Network logo

Jiri Navratil and Les Cottrell. Page created: May 2 2003.

Central Computer Access | Computer Networking | Network Group | ICFA-NTF Monitoring
SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook

Introduction

This case shows the possibility to discover network problems via monitoring of available bandwidth. Our experimental monitor ABwE is running 24 hours a day. It is using Packet-pair dispersion techniques to analyze ABW (available Bandwidth) on selected paths. We are sending only 20 packet-pairs probes. So it means that ABwE doesn't generate large amounts of network traffic and it is totally non-intrusive. The current mode is set to run 1 measurent to each host per 120 seconds. More details about ABwE can be found in the following publication.

Abilene problems (May 6th - May 7th 2003)

The following paragraphs describe the situations when we discovered some bandwidth problems (probably caused by routing chages in Abilene network) We detect problems on the path to Florida University (increased crros-traffic) in time between 23.00-24.00 (May 6th) and once again at about 6.15 am (May 7th). Corresponding to this ABW dropped. From other US sites monitoring from SLAC the University of Michigen has been heavily involved in this case.
Also all connetions to Europe: IN2p3,Italy, Czech (via Abilene) has been hit by this. As this situation has been detected by ABwE is on following pictures:

ABwE results for SLAC-UFL path (night May-6/7)

ABwE results for SLAC-UMICH path (night May-6/7)

ABwE results for SLAC-France-In2p3 path (night May-6/7)

ABwE results for SLAC-Italy-Padova path (night May-6/7)

ABwE results for SLAC-Italy-Milano path (night May-6/7)

ABwE results for SLAC-CzechRep-Cesnet path (night May-6/7)


Also IEPM tools recognized this problem IEPM-BW. Traceroute protocols shows the new routes to these sites: via backup connection ALTER.NET to CESNET (Prague) or GLBX.NET to Italy (Padova)The traceroute information for these changes are in following paragraphs.
NORMAL PATH to UMICH:
traceroute to xxx.physics.lsa.umich.edu (141.211.101.122), 30 hops max, 38 byte packets
 1  rtr-core1-pub6 (134.79.27.2)  0.784 ms  0.649 ms  0.642 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.674 ms  0.646 ms  0.634 ms
 3  i2-gateway.stanford.edu (192.68.191.83)  0.642 ms  0.823 ms  0.612 ms
 4  STAN.POS.calren2.NET (171.64.1.213)  0.692 ms  0.698 ms  0.636 ms
 5  SUNV--STAN.POS.calren2.net (198.32.249.73)  1.083 ms  0.706 ms  0.792 ms
 6  Abilene--QSV.POS.calren2.net (198.32.249.162)  1.062 ms  1.084 ms  1.024 ms
 7  kscyng-snvang.abilene.ucaid.edu (198.32.8.103)  36.298 ms  36.235 ms  36.235 ms
 8  iplsng-kscyng.abilene.ucaid.edu (198.32.8.80)  45.755 ms  49.931 ms  54.410 ms
 9  so-0-2-0x1.aa1.mich.net (192.122.183.9)  55.928 ms  56.082 ms  56.099 ms
10  atm1-0x24.michnet8.mich.net (198.108.23.82)  56.223 ms  56.360 ms  56.071 ms
11  * * *
12  A-ARB3-LSA-NG.r-LSA.umnet.umich.edu (141.211.5.22)  56.975 ms  56.740 ms  56.751 ms

BACKUP to UMICH (from IEPM log):
traceroute to 141.211.43.109 (141.211.43.109), 10 hops max, 38 byte packets
  1  rtr-gsr-test (134.79.243.1)  0.324 ms
  2  rtr-dmz1-ger (134.79.135.15)  0.323 ms
  3  slac-rt4.es.net (192.68.191.146)  0.394 ms
  4  snv-pos-slac.es.net (134.55.209.1)  0.751 ms
  5  chicr1-oc192-snvcr1.es.net (134.55.209.54)  48.941 ms
  6  chirt1-ge0-chicr1.es.net (134.55.209.190)  48.943 ms
  7  198.125.140.146 (198.125.140.146)  49.536 ms
  8  ge5-0x24.michnet8.mich.net (198.108.23.245)  55.629 ms
  9  a-arb3-merit1.c-arb3.umnet.umich.edu (192.122.183.82)  56.309 ms
 10  A-ARB3-SEB-NG.r-SEB.umnet.umich.edu (141.211.7.169)  56.916 ms


NORMAL PATH to Padova:
traceroute to xxx.pd.infn.it (193.206.210.5), 30 hops max, 38 byte packets
 1  rtr-core1-pub6 (134.79.27.2)  0.754 ms  0.648 ms  0.624 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.661 ms  0.625 ms  0.623 ms
 3  i2-gateway.stanford.edu (192.68.191.83)  0.657 ms  0.616 ms  0.610 ms
 4  STAN.POS.calren2.NET (171.64.1.213)  0.691 ms  0.637 ms  0.653 ms
 5  SUNV--STAN.POS.calren2.net (198.32.249.73)  1.102 ms  1.014 ms  1.070 ms
 6  Abilene--QSV.POS.calren2.net (198.32.249.162)  1.408 ms  1.264 ms  1.415 ms
 7  kscyng-snvang.abilene.ucaid.edu (198.32.8.103)  36.646 ms  36.412 ms  36.228 ms
 8  iplsng-kscyng.abilene.ucaid.edu (198.32.8.80)  45.475 ms  51.931 ms  45.498 ms
 9  chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  49.365 ms  54.587 ms  56.081 ms
10  nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  69.665 ms  69.749 ms  69.492 ms
11  abilene.uk1.uk.geant.net (62.40.103.25)  137.355 ms  137.221 ms  137.216 ms
12  uk.fr1.fr.geant.net (62.40.96.89)  144.294 ms  144.217 ms  144.265 ms
13  fr.ch1.ch.geant.net (62.40.96.29)  152.574 ms  152.644 ms  152.421 ms
14  ch.it1.it.geant.net (62.40.96.34)  165.743 ms  165.683 ms  165.821 ms
15  garr-prim-gw.it1.it.geant.net (62.40.103.190)  165.622 ms  165.969 ms  166.279 ms
16  bo-mi-g.garr.net (193.206.134.22)  169.832 ms  169.476 ms  169.456 ms
17  infnpd-rtg.bo.garr.net (193.206.128.34)  179.082 ms  179.024 ms  179.225 ms
18  172.16.32.9 (172.16.32.9)  178.552 ms  178.443 ms  178.604 ms
19  xxx.pd.infn.it (193.206.210.5)  178.566 ms  178.447 ms  178.522 ms


BACKUP to PADOVA (from IEPM log):
##(05/07/2003 00:10:39 1052291439)TRACE - CMD - /usr/bin/time -p /usr/sbin/traceroute -m 10 -q 1 -w 3 XXX.PD.INFN.IT 2>&1
##(05/07/2003 00:10:39 1052291439) traceroute to xxxx.PD.INFN.IT (193.206.210.5), 10 hops max, 38 byte packets
 1  rtr-gsr-test (134.79.243.1)  0.194 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.357 ms
 3  slac-rt4.es.net (192.68.191.146)  0.282 ms
 4  snv-pos-slac.es.net (134.55.209.1)  0.750 ms
 5  snvrt1-ge0-snvcr1.es.net (134.55.209.90)  0.805 ms
 6  paix-pa-snv.es.net (134.55.208.205)  1.603 ms
 7  atm3-0-207-5M.cr1.SJC.gblx.net (208.50.13.53)  2.277 ms
 8  so-6-0-0-2488M.cr1.PAO2.gblx.net (208.51.224.141)  2.633 ms
 9  pos2-0-2488M.cr1.lin1.gblx.net (62.16.32.94)  157.313 ms
 10  so0-0-0-622M.ar2.lin1.gblx.net (208.51.236.58)  159.061 ms
 ...

NORMAL PATH to CESNET
traceroute to xxx.cesnet.cz (195.113.187.2), 30 hops max, 38 byte packets
 1  rtr-core1-pub6 (134.79.27.2)  0.943 ms  0.446 ms  0.362 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.856 ms  0.431 ms  0.716 ms
 3  i2-gateway.stanford.edu (192.68.191.83)  0.838 ms  0.616 ms  0.608 ms
 4  STAN.POS.calren2.NET (171.64.1.213)  0.691 ms  0.683 ms  0.644 ms
 5  SUNV--STAN.POS.calren2.net (198.32.249.73)  1.065 ms  1.041 ms  1.230 ms
 6  Abilene--QSV.POS.calren2.net (198.32.249.162)  1.358 ms  1.243 ms  1.346 ms
 7  kscyng-snvang.abilene.ucaid.edu (198.32.8.103)  36.661 ms  36.713 ms  36.794 ms
 8  iplsng-kscyng.abilene.ucaid.edu (198.32.8.80)  45.684 ms  61.853 ms  47.860 ms
 9  chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  49.138 ms  49.402 ms  49.612 ms
10  nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  69.571 ms  69.517 ms  69.601 ms
11  abilene.uk1.uk.geant.net (62.40.103.25)  137.345 ms  137.299 ms  137.264 ms
12  uk.fr1.fr.geant.net (62.40.96.89)  144.419 ms  144.394 ms  144.255 ms
13  fr.de1.de.geant.net (62.40.96.49)  152.831 ms  152.487 ms  152.608 ms
14  de.cz1.cz.geant.net (62.40.96.37)  160.960 ms  161.163 ms  161.100 ms
15  cesnet-gw.cz1.cz.geant.net (62.40.103.30)  161.649 ms  161.372 ms  161.559 ms
16  r41prg-pos13-0-stm16.cesnet.cz (195.113.156.110)  161.350 ms  161.788 ms  161.530 ms

BACKUP PATH to CESNET (from IEPM log):
##(05/06/2003 23:26:59 1052288819)TRACE - CMD - /usr/bin/time -p /usr/sbin/traceroute -m 10 -q 1 -w 3 iepm-bw.cesnet.cz 2>&1
##(05/06/2003 23:26:59 1052288819) traceroute to xxx.cesnet.cz (195.113.187.2), 10 hops max, 38 byte packets
 1  rtr-gsr-test (134.79.243.1)  0.179 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.251 ms
 3  slac-rt4.es.net (192.68.191.146)  1.389 ms
 4  snv-pos-slac.es.net (134.55.209.1)  0.788 ms
 5  snvrt1-ge0-snvcr1.es.net (134.55.209.90)  0.713 ms
 6  188.ATM1-0.BR2.SJC1.ALTER.NET (204.255.174.49)  40.029 ms
 7  154.ATM3-0.XR1.SJC1.ALTER.NET (152.63.51.174)  18.231 ms
 8  0.so-0-0-0.XL1.SJC1.ALTER.NET (152.63.55.114)  35.076 ms
 9  0.so-1-0-0.XL1.PAO1.ALTER.NET (152.63.50.154)  32.665 ms
 10  POS6-0.GW6.PAO1.ALTER.NET (152.63.55.9)  35.721 ms
...

This is from traceroute monitor:

France: IN2P3

normal path
1052286194 ccsvsn04.in2p3.fr rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,0 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 i2-gateway.stanford.edu,192.68.191.83,0,0 STAN.POS.calren2.NET,171.64.1.213,0,0 SUNV--STAN.POS.calren2.net,198.32.249.73,0,0 Abilene--QSV.POS.calren2.net,198.32.249.162,0,1 kscyng-snvang.abilene.ucaid.edu,198.32.8.103,0,37 iplsng-kscyng.abilene.ucaid.edu,198.32.8.80,0,52 chinng-iplsng.abilene.ucaid.edu,198.32.8.76,0,54 nycmng-chinng.abilene.ucaid.edu,198.32.8.83,0,74 abilene.uk1.uk.geant.net,62.40.103.25,0,141 uk.fr1.fr.geant.net,62.40.96.89,0,149 renater-gw.fr1.fr.geant.net,62.40.103.54,0,144 lyon-pos9-0.cssi.renater.fr,193.51.179.130,0,158 in2p3-lyon.cssi.renater.fr,193.51.181.6,0,157 in2p3-lyon.cssi.renater.fr,193.51.181.6,0,158

backup path
1052287397 ccsvsn04.in2p3.fr rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,0 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 192.68.191.146,192.68.191.146,0,0 snv-pos-slac.es.net,134.55.209.1,0,1 chicr1-oc192-snvcr1.es.net,134.55.209.54,0,51 chirt1-ge0-chicr1.es.net,134.55.209.190,0,51 A5-3-2.CHIBB1.Chicago.opentransit.net,198.125.140.6,0,49 P1-0.CHICR1.Chicago.opentransit.net,193.251.128.202,0,49 P9-0.NYKCR2.New-york.opentransit.net,193.251.242.249,0,82 P4-0.PASCR1.Pastourelle.opentransit.net,193.251.241.133,0,159 P4-0.BAGCR3.Bagnolet.opentransit.net,193.251.241.118,0,163 P1-0.BAGBB1.Bagnolet.opentransit.net,193.251.241.145,0,161 193.51.185.30,193.51.185.30,0,159 in2p3-lyon.cssi.renater.fr,193.51.181.6,0,158 in2p3-lyon.cssi.renater.fr,193.51.181.6,0,158

Italy: Padova

normal path
1052286244 ipm03.pd.infn.it rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,2 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 i2-gateway.stanford.edu,192.68.191.83,0,0 STAN.POS.calren2.NET,171.64.1.213,0,0 SUNV--STAN.POS.calren2.net,198.32.249.73,0,0 Abilene--QSV.POS.calren2.net,198.32.249.162,0,1 kscyng-snvang.abilene.ucaid.edu,198.32.8.103,0,37 iplsng-kscyng.abilene.ucaid.edu,198.32.8.80,0,56 chinng-iplsng.abilene.ucaid.edu,198.32.8.76,0,51 nycmng-chinng.abilene.ucaid.edu,198.32.8.83,0,70 abilene.uk1.uk.geant.net,62.40.103.25,0,139 uk.fr1.fr.geant.net,62.40.96.89,0,146 fr.ch1.ch.geant.net,62.40.96.29,0,154 ch.it1.it.geant.net,62.40.96.34,0,166 garr-prim-gw.it1.it.geant.net,62.40.103.190,0,165 bo-mi-g.garr.net,193.206.134.22,0,169 infnpd-rtg.bo.garr.net,193.206.128.34,0,180 172.16.32.9,172.16.32.9,100,180 ipm03.pd.infn.it,193.206.210.5,0,178

backup path
1052289845 ipm03.pd.infn.it rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,0 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 192.68.191.146,192.68.191.146,0,0 snv-pos-slac.es.net,134.55.209.1,0,1 snvrt1-ge0-snvcr1.es.net,134.55.209.90,0,1 paix-pa-snv.es.net,134.55.208.205,0,98 atm3-0-207-5M.cr1.SJC.gblx.net,208.50.13.53,0,3 so-6-0-0-2488M.cr1.PAO2.gblx.net,208.51.224.141,0,3 pos2-0-2488M.cr1.lin1.gblx.net,62.16.32.94,0,159 so0-0-0-622M.ar2.lin1.gblx.net,208.51.236.58,0,159 Dante-Milan-2.so-6-0-0.ar2.LIN1.gblx.net,64.214.196.242,0,162 bo-mi-g.garr.net,193.206.134.22,0,165 infnpd-rtg.bo.garr.net,193.206.128.34,0,174 172.16.32.9,172.16.32.9,100,174 ipm03.pd.infn.it,193.206.210.5,10,174

Czech: CESNET

normal path
1052286214 iepm-bw.cesnet.cz rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,0 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 i2-gateway.stanford.edu,192.68.191.83,0,0 STAN.POS.calren2.NET,171.64.1.213,0,0 SUNV--STAN.POS.calren2.net,198.32.249.73,0,0 Abilene--QSV.POS.calren2.net,198.32.249.162,0,1 kscyng-snvang.abilene.ucaid.edu,198.32.8.103,0,37 iplsng-kscyng.abilene.ucaid.edu,198.32.8.80,0,51 chinng-iplsng.abilene.ucaid.edu,198.32.8.76,0,54 nycmng-chinng.abilene.ucaid.edu,198.32.8.83,0,72 abilene.uk1.uk.geant.net,62.40.103.25,0,139 uk.fr1.fr.geant.net,62.40.96.89,0,145 fr.de1.de.geant.net,62.40.96.49,0,154 de.cz1.cz.geant.net,62.40.96.37,0,162 cesnet-gw.cz1.cz.geant.net,62.40.103.30,0,161 r41prg-pos13-0-stm16.cesnet.cz,195.113.156.110,0,176

backup path
1052287417 iepm-bw.cesnet.cz rtr-core1-iepm.slac.stanford.edu,134.79.143.2,0,0 rtr-dmz1-ger.slac.stanford.edu,134.79.135.15,0,0 192.68.191.146,192.68.191.146,0,2 snv-pos-slac.es.net,134.55.209.1,0,3 snvrt1-ge0-snvcr1.es.net,134.55.209.90,0,5 188.ATM1-0.BR2.SJC1.ALTER.NET,204.255.174.49,0,15 154.ATM3-0.XR2.SJC1.ALTER.NET,152.63.51.178,0,19 0.so-0-0-0.XL2.SJC1.ALTER.NET,152.63.55.126,0,21 0.so-7-0-0.XL2.PAO1.ALTER.NET,152.63.113.21,0,17 POS7-0.GW6.PAO1.ALTER.NET,152.63.55.14,0,17 208.214.140.58,208.214.140.58,0,23 sca-bb1-pos0-0-0.telia.net,213.248.86.57,0,24 chi-bb1-pos1-0-0.telia.net,213.248.80.33,0,64 nyk-bb1-pos0-1-0.telia.net,213.248.80.5,0,98 kbn-bb1-pos0-1-1.telia.net,213.248.66.125,0,185 hbg-bb1-pos3-0-0.telia.net,213.248.65.78,0,188 prag-b2-pos1-2.telia.net,213.248.65.86,0,191 cesnet-01804-prag-b2.c.telia.net,213.248.79.102,0,190
Page owner: Jiri Navratil