Icmp checksums reported in Traceroutes We have observed this, see for example http://www.slac.stanford.edu/comp/net/bandwidth-tests/hercules/tracesummaries/today.html abd look at the orange dots, then click on Log* for that node, e.g. ##(05/19/2004 00:35:40 1084952140)TRACE - CMD - /usr/bin/time -p /usr/sbin/traceroute -q 1 -w 2 -I IPERF2-GIG.NSLABS.UFL.EDU 2>&1 ##(05/19/2004 00:35:40 1084952140) traceroute to IPERF2-GIG.NSLABS.UFL.EDU (65.118.160.5), 30 hops max, 38 byte packets ##(05/19/2004 00:35:40 1084952140) 1 rtr-gsr-test (134.79.243.1) 0.127 ms ##(05/19/2004 00:35:40 1084952140) 2 rtr-dmz1-ger (134.79.135.15) 0.221 ms ##(05/19/2004 00:35:40 1084952140) 3 i2-gateway.stanford.edu (192.68.191.83) 0.181 ms ##(05/19/2004 00:35:40 1084952140) 4 dc-oak-dc1--stan-ge.cenic.net (137.164.23.41) 1.781 ms ##(05/19/2004 00:35:40 1084952140) 5 dc-oak-dc2--oak-dc1-ge.cenic.net (137.164.22.37) 1.642 ms ##(05/19/2004 00:35:40 1084952140) 6 dc-csac-dc1--oak-dc2-ge.cenic.net (137.164.22.111) 3.340 ms ##(05/19/2004 00:35:40 1084952140) 7 dc-lax-dc1--sac-dc1-pos.cenic.net (137.164.22.126) 12.463 ms ##(05/19/2004 00:35:40 1084952140) 8 hpr-lax-hpr1--dc-lax-dc1-ge.cenic.net (137.164.22.13) 12.720 ms ##(05/19/2004 00:35:40 1084952140) 9 abilene-LA--hpr-lax-gsr1-10ge.cenic.net (137.164.25.3) 25.498 ms ##(05/19/2004 00:35:40 1084952140) 10 hstnng-losang.abilene.ucaid.edu (198.32.8.22) 44.314 ms ##(05/19/2004 00:35:40 1084952140) 11 atlang-hstnng.abilene.ucaid.edu (198.32.8.34) 64.498 ms ##(05/19/2004 00:35:40 1084952140) Icmp checksum is wrong ##(05/19/2004 00:35:40 1084952140) 12 a713.c12008.atla.abilene.ucaid.edu (192.80.53.46) 78.865 ms ##(05/19/2004 00:35:40 1084952140) 13 ssrb230a-ewan-msfc-2-v710-1.ns.ufl.edu (128.227.254.122) 79.085 ms ##(05/19/2004 00:35:40 1084952140) 14 ssrb-core-msfc-v2.ns.ufl.edu (128.227.254.176) 78.980 ms ##(05/19/2004 00:35:40 1084952140) 15 nslab-bpop-rsm-v222.nslabs.ufl.edu (128.227.74.130) 79.581 ms ##(05/19/2004 00:35:40 1084952140) 16 node2.nslabs.ufl.edu (65.118.xxx.xxx) 78.935 ms ##(05/19/2004 00:35:40 1084952140) real 0.56 ##(05/19/2004 00:35:40 1084952140) user 0.00 ##(05/19/2004 00:35:40 1084952140) sys 0.00 It has been observed by others, e.g. http://www.linuxsolved.com/forums/viewtopic.php?t=385 For more on ICMP and traceroute see: http://www.networksorcery.com/enp/protocol/icmp/msg30.htm According to Raj Jain in http://www.cse.ohio-state.edu/~jain/bnr/ftp/f19_icm.pdf "ICMP messages are not required to report checksum errors, some implementations still do." Looking at our data, e.g. grep "Icmp checksum" /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node*/2004_05/ctrace.2004_05_* | more Only appears when using -I (ICMP option), this is used with node1.cesnet.cz, node1.desy.de, node1.fzk.de, node1.man.ac.uk, node1.nslabbs.de, node2.jlab.org, node2.nslabs.ufl.edu, node2.rhic.bnl.gov It appears to occur on all the above routes at some time (not always) Typically occurs on one or two hops Using data for this year: 1. We want to a know a distribution of number of hops it occurs on for each route (i.e. for each route plot frequency vs number of hops (include zero in number of hops) reporting Icmp checksum) 2. For each route which hops are reprting Icmp checksum errors (i.e. x axis is hop name, y axis is the number of Icmp checksums for that hop) Also see if you can find out anything about the problem by searching the web. Also could post a question to some appropriate email lists. Occurs in UDP Probes also ------------------------- ICMP checksums can also occur on UDP probes. See for example: ##(05/26/2004 01:02:26 1085558546)TRACE - CMD - /usr/bin/time -p /usr/sbin/traceroute -q 1 -w 2 CCSVSN04.IN2P3.FR 2>&1 ##(05/26/2004 01:02:26 1085558546) traceroute to CCSVSN04.IN2P3.FR (134.158.104.199), 30 hops max, 38 byte packets ##(05/26/2004 01:02:26 1085558546) 1 rtr-gsr-test (134.79.243.1) 0.149 ms ##(05/26/2004 01:02:26 1085558546) 2 rtr-dmz1-ger (134.79.135.15) 0.308 ms ##(05/26/2004 01:02:26 1085558546) 3 slac-rt4.es.net (192.68.191.146) 0.247 ms ##(05/26/2004 01:02:26 1085558546) 4 snv-pos-slac.es.net (134.55.209.1) 0.717 ms ##(05/26/2004 01:02:26 1085558546) 5 chicr1-oc192-snvcr1.es.net (134.55.209.54) 48.720 ms ##(05/26/2004 01:02:26 1085558546) 6 aoacr1-oc192-chicr1.es.net (134.55.209.58) 69.100 ms ##(05/26/2004 01:02:26 1085558546) 7 esnet.ny1.ny.geant.net (62.40.103.213) 69.054 ms ##(05/26/2004 01:02:26 1085558546) 8 ny.uk1.uk.geant.net (62.40.96.170) 137.808 ms ##(05/26/2004 01:02:26 1085558546) 9 uk.fr1.fr.geant.net (62.40.96.89) 146.845 ms ##(05/26/2004 01:02:26 1085558546) 10 renater-10G-gw.fr1.fr.geant.net (62.40.103.162) 145.004 ms ##(05/26/2004 01:02:26 1085558546) 11 lyon-pos9-0.cssi.renater.fr (193.51.179.130) 152.921 ms ##(05/26/2004 01:02:26 1085558546) 12 in2p3-lyon.cssi.renater.fr (193.51.181.6) 152.799 ms ##(05/26/2004 01:02:26 1085558546) Icmp checksum is wrong ##(05/26/2004 01:02:26 1085558546) Icmp checksum is wrong ##(05/26/2004 01:02:26 1085558546) 13 * ##(05/26/2004 01:02:26 1085558546) 14 * ##(05/26/2004 01:02:26 1085558546) 15 in2p3-lyon.cssi.renater.fr (193.51.181.6) 153.453 ms !X ##(05/26/2004 01:02:26 1085558546) real 5.08 ##(05/26/2004 01:02:26 1085558546) user 0.00 ##(05/26/2004 01:02:26 1085558546) sys 0.01 ##(05/26/2004 01:02:26 1085558546) SUMMARY: numhops=15; ##(05/26/2004 01:02:26 1085558546) TIMESUMMARY: real=5.08; user=0.00; sys=0.01; testtime=05/26/2004 01:02:26 1085558546; ##(05/26/2004 01:02:31 1085558551)TRACE - FINISHED (5) 5142 "Icmp checksums" 112005 /user/sbin/traceroute 22846 /usr/sbin/traceroute -I 89159 /usr/sbin/traceroute (UDP probes) 4 "Icmp checksums" without -I (UDP probes) 22cottrell@flora03:~>grep "Icmp checksum" /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node*/2004_05/ctrace.2004_05_* | egrep -v nslabs | grep -v desy | grep -v cesnet | grep -v fzk | grep -v man | grep -v jlab | grep -v rhic | more /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node1.clrc.ac.uk/2004_05/ctrace.2004_05_29:##(05/29/2004 11:36:23 1085855783) I cmp checksum is wrong /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node1.nersc.gov/2004_05/ctrace.2004_05_17:##(05/17/2004 11:42:58 1084819378) Ic mp checksum is wrong /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node1.riken.go.jp/2004_05/ctrace.2004_05_26:##(05/26/2004 15:15:25 1085609725) Icmp checksum is wrong /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node1.sdsc.edu/2004_05/ctrace.2004_05_25:##(05/25/2004 09:15:40 1085501740) Icm p checksum is wrong grep "Icmp checksum" /afs/slac/www/comp/net/bandwidth-tests/hercules/slaconly/rawdata/node*/2004_05/ctrace.2004_05_* > /tmp/cksum 35cottrell@flora03:~>cut -f 3,3 -d" " /tmp/cksum | uniq -c | wc #number of traceroutes with one or more Icmp checksums 2693 5386 45781 # Cksums Freq 1 2022 2 435 3 37 4 22 5 8 6 11 7 7 8 4 9 9 10 95 11 1 12 3 13 4 15 4 16 2 17 2 18 3 19 1 20 7 21 0 22 3 23 5 24 1 25 0 26 1 27 0 28 1 29 1 30 2 31 0 32 0 33 1 49cottrell@flora03:~>cut -f 11,11 -d"/" /tmp/cksum | uniq -c | sort > ! /tmp/cksum-nodes ne /tmp/cksum-nodes 1 node1.clrc.ac.uk 1 node1.nersc.gov 1 node1.riken.go.jp 1 node1.sdsc.edu 249 node2.jlab.org 319 node1.man.ac.uk 389 node1.desy.de 421 node1.fzk.de 426 node2.rhic.bnl.gov 665 node1.nslabs.ufl.edu 706 node1.cesnet.cz 1963 node2.nslabs.ufl.edu