> -----Original Message----- > From: Cottrell, Les > Sent: Wednesday, May 19, 2004 4:09 PM > To: 'Jerome Bernier' > Cc: iepm-l; Buhrmaster, Gary > Subject: Traceroute behavior from SLAC to IN2P3 > > I have been analyzing traceroutes from SLAC to many of our > collaborator sites. The one to IN2P3 is intriguing since it exhibits > what I call "stutters" (i.e. the same router interface responds to > more than one TTL) at the last hop (see below). Also the last hop has > an !X annotation (communication Administratively prohibited). > > If we try and use ICMP probes with the traceroute to get round some > blocking then that does not help (see below) > > It is not a problem for us, in fact it is interesting since it helps > debug our analysis programs. However, I was wondering if you know what > the cause of the stutters might be > > We have another route (to BNL) which also shows stutters but in that > case there are no annotations (possibly since in that case we have to > use ICMP probes to get the traceroute to work). > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > ##(05/19/2004 00:10:33 1084950633)CTRACE - START > ##(05/19/2004 00:10:43 1084950643)TRACE - CMD - /usr/bin/time -p > /usr/sbin/traceroute -q 1 -w 2 CCSVSN04.IN2P3.FR 2>&1 > ##(05/19/2004 00:10:43 1084950643) traceroute to CCSVSN04.IN2P3.FR > (134.158.104.199), 30 hops max, 38 byte packets > ##(05/19/2004 00:10:43 1084950643) 1 rtr-gsr-test > (134.79.243.1) 0.125 ms > ##(05/19/2004 00:10:43 1084950643) 2 rtr-dmz1-ger > (134.79.135.15) 0.251 ms > ##(05/19/2004 00:10:43 1084950643) 3 slac-rt4.es.net > (192.68.191.146) 0.265 ms > ##(05/19/2004 00:10:43 1084950643) 4 snv-pos-slac.es.net > (134.55.209.1) 0.779 ms > ##(05/19/2004 00:10:43 1084950643) 5 > chicr1-oc192-snvcr1.es.net (134.55.209.54) 53.545 ms > ##(05/19/2004 00:10:43 1084950643) 6 > aoacr1-oc192-chicr1.es.net (134.55.209.58) 69.086 ms > ##(05/19/2004 00:10:43 1084950643) 7 > esnet.ny1.ny.geant.net (62.40.103.213) 69.802 ms > ##(05/19/2004 00:10:43 1084950643) 8 ny.uk1.uk.geant.net > (62.40.96.170) 142.196 ms > ##(05/19/2004 00:10:43 1084950643) 9 uk.fr1.fr.geant.net > (62.40.96.89) 148.989 ms > ##(05/19/2004 00:10:43 1084950643) 10 renater-10G-gw.fr1.fr.geant.net > (62.40.103.162) 149.487 ms > ##(05/19/2004 00:10:43 1084950643) 11 lyon-pos9-0.cssi.renater.fr > (193.51.179.130) 157.204 ms > ##(05/19/2004 00:10:43 1084950643) 12 in2p3-lyon.cssi.renater.fr > (193.51.181.6) 157.262 ms > ##(05/19/2004 00:10:43 1084950643) 13 in2p3-lyon.cssi.renater.fr > (193.51.181.6) 157.600 ms !X > ##(05/19/2004 00:10:43 1084950643) real 1.11 > ##(05/19/2004 00:10:43 1084950643) user 0.00 > ##(05/19/2004 00:10:43 1084950643) sys 0.00 > ##(05/19/2004 00:10:43 1084950643) SUMMARY: numhops=13; > ##(05/19/2004 00:10:43 1084950643) TIMESUMMARY: real=1.11; user=0.00; > sys=0.00; testtime=05/19/2004 00:10:43 1084950643; > ##(05/19/2004 00:10:44 1084950644)CTRACE - FINISHED (1) > ##(05/19/2004 00:25:51 1084951551)CTRACE - START > ##(05/19/2004 00:26:00 1084951560)TRACE - CMD - /usr/bin/time -p > /usr/sbin/traceroute -q 1 -w 2 CCSVSN04.IN2P3.FR 2>&1 > ##(05/19/2004 00:26:00 1084951560) traceroute to CCSVSN04.IN2P3.FR > (134.158.104.199), 30 hops max, 38 byte packets > ##(05/19/2004 00:26:00 1084951560) 1 rtr-gsr-test > (134.79.243.1) 0.132 ms > ##(05/19/2004 00:26:00 1084951560) 2 rtr-dmz1-ger > (134.79.135.15) 0.239 ms > ##(05/19/2004 00:26:00 1084951560) 3 slac-rt4.es.net > (192.68.191.146) 0.235 ms > ##(05/19/2004 00:26:00 1084951560) 4 snv-pos-slac.es.net > (134.55.209.1) 0.794 ms > ##(05/19/2004 00:26:00 1084951560) 5 > chicr1-oc192-snvcr1.es.net (134.55.209.54) 48.680 ms > ##(05/19/2004 00:26:00 1084951560) 6 > aoacr1-oc192-chicr1.es.net (134.55.209.58) 68.983 ms > ##(05/19/2004 00:26:00 1084951560) 7 > esnet.ny1.ny.geant.net (62.40.103.213) 69.129 ms > ##(05/19/2004 00:26:00 1084951560) 8 ny.uk1.uk.geant.net > (62.40.96.170) 142.107 ms > ##(05/19/2004 00:26:00 1084951560) 9 uk.fr1.fr.geant.net > (62.40.96.89) 148.979 ms > ##(05/19/2004 00:26:00 1084951560) 10 renater-10G-gw.fr1.fr.geant.net > (62.40.103.162) 149.297 ms > ##(05/19/2004 00:26:00 1084951560) 11 lyon-pos9-0.cssi.renater.fr > (193.51.179.130) 157.111 ms > ##(05/19/2004 00:26:00 1084951560) 12 in2p3-lyon.cssi.renater.fr > (193.51.181.6) 157.290 ms > ##(05/19/2004 00:26:00 1084951560) 13 * > ##(05/19/2004 00:26:00 1084951560) 14 in2p3-lyon.cssi.renater.fr > (193.51.181.6) 157.481 ms !X > ##(05/19/2004 00:26:00 1084951560) real 3.10 > ##(05/19/2004 00:26:00 1084951560) user 0.00 > ##(05/19/2004 00:26:00 1084951560) sys 0.00 > ##(05/19/2004 00:26:00 1084951560) SUMMARY: numhops=14; > ##(05/19/2004 00:26:00 1084951560) TIMESUMMARY: real=3.10; user=0.00; > sys=0.00; testtime=05/19/2004 00:26:00 1084951560; > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 26cottrell@hercules:~>traceroute -I CCASN07.IN2P3.FR traceroute to > CCASN07.IN2P3.FR (134.158.105.27), 30 hops max, > 38 byte packets > 1 rtr-gsr-test (134.79.243.1) 0.147 ms 0.074 ms 0.068 ms Icmp > checksum is wrong > 2 rtr-dmz1-ger (134.79.135.15) 139.682 ms 0.801 ms 0.381 ms > 3 slac-rt4.es.net (192.68.191.146) 0.283 ms 0.239 ms 0.238 ms > 4 snv-pos-slac.es.net (134.55.209.1) 0.675 ms 5.369 ms 0.608 ms > 5 chicr1-oc192-snvcr1.es.net (134.55.209.54) 48.696 ms > 48.679 ms 48.676 ms > 6 aoacr1-oc192-chicr1.es.net (134.55.209.58) 69.006 ms > 68.999 ms 69.078 ms > 7 esnet.ny1.ny.geant.net (62.40.103.213) 69.093 ms 69.056 ms > 69.094 ms > 8 ny.uk1.uk.geant.net (62.40.96.170) 142.116 msIcmp checksum is > wrong > 142.118 ms 142.184 ms > 9 uk.fr1.fr.geant.net (62.40.96.89) 148.988 ms 148.968 ms > 149.026 ms > 10 renater-10G-gw.fr1.fr.geant.net (62.40.103.162) 149.293 msIcmp > checksum is wrong Icmp checksum is wrong > 149.375 ms 149.321 ms > 11 lyon-pos9-0.cssi.renater.fr (193.51.179.130) 157.081 ms > 157.081 ms 157.101 ms > Icmp checksum is wrong > 12 in2p3-lyon.cssi.renater.fr (193.51.181.6) 157.184 msIcmp checksum > is wrong Icmp checksum is wrong Icmp checksum is wrong > 157.186 msIcmp checksum is wrong > 157.210 ms > Icmp checksum is wrong > Icmp checksum is wrong > Icmp checksum is wrong > Icmp checksum is wrong > Icmp checksum is wrong > 13 *Icmp checksum is wrong > Icmp checksum is wrong > Icmp checksum is wrong > From: Buhrmaster, Gary Sent: Wednesday, May 19, 2004 4:17 PM To: Cottrell, Les Cc: iepm-l; 'Jerome Bernier' Subject: RE: Traceroute behavior from SLAC to IN2P3 Les, I think this is to be expected with administratively prohibited paths (depending on implementation in either the hardware or software). First the router gets the packet on the incoming interface, decrements the TTL to zero, and responds. The next packet has a one higher TTL, which the router is about to forward, and then detects that the ACL prohibits it. Returned as administratively prohibited. And then there is the fact that many routers (to reduce the royal overhead) will not respond to every packet if they occur too often (rate limiting). Gary