Problems with connectivity to site in S. Africa - August '06

While working on iepmbw.twaren.net, Connie observed that iepmbw.bnl.org is not accessible from taiwan and taiwan is not accessible from BNL. But both of the sites are accessible from other sites  e.g., SLAC

Initial Information fro Connie

From iepmbw.bnl.org

[cal@iepmbw ~]$ host iepmbw.twaren.net ;; connection timed out; no servers could be reached

[cal@iepmbw ~]$ host 211.79.61.163 ;; connection timed out; no servers could be reached

From iepmbw.twaren.net

[iepm@iepmbw ~/v3src]$ host 192.203.218.61 Host 61.218.203.192.in-addr.arpa not found: 5(REFUSED)

[iepm@iepmbw ~/v3src]$ host iepmbw.bnl.org iepmbw.bnl.org has address 192.203.218.61

Traceroutes

From SLAC to BNL

  1    <1 ms    <1 ms    <1 ms  rtr-core1-pub6.slac.stanford.edu [134.79.27.2]

  2    <1 ms    <1 ms    <1 ms  rtr-dmz1-ger.slac.stanford.edu [134.79.135.15]

  3    <1 ms    <1 ms    <1 ms  slac-rt4.es.net [192.68.191.146]

  4    <1 ms    <1 ms    <1 ms  slacmr1-slacrt4.es.net [134.55.209.93]

  5    <1 ms    <1 ms    <1 ms  snv2mr1-slacmr1.es.net [134.55.217.2]

  6    <1 ms    <1 ms    <1 ms  snv2sdn1-snv2mr1.es.net [134.55.207.37]

  7    49 ms    49 ms    49 ms  chicr1-oc192-snv2sdn1.es.net [134.55.209.54]

  8    69 ms    69 ms    68 ms  aoacr1-oc192-chicr1.es.net [134.55.209.58]

  9    71 ms    71 ms    71 ms  bnlmr1-aoacr1.es.net [134.55.218.45]

 10    71 ms    71 ms    71 ms  bnlsite-bnlmr1.es.net [198.124.216.178]

 11    71 ms    71 ms    71 ms  iepmbw.bnl.org [192.203.218.61]

From SLAC to Taiwan

  1    <1 ms    <1 ms    <1 ms  rtr-core1-pub6.slac.stanford.edu [134.79.27.2]

  2    <1 ms    <1 ms    <1 ms  rtr-dmz1-ger.slac.stanford.edu [134.79.135.15]

  3    <1 ms    <1 ms    <1 ms  i2-gateway.stanford.edu [192.68.191.83]

  4    <1 ms    <1 ms    <1 ms  hpr-svl-hpr--stan-ge.cenic.net [137.164.27.161]

  5     8 ms     8 ms     8 ms  lax-hpr--svl-hpr-10ge.cenic.net [137.164.25.12]

  6     8 ms     8 ms     8 ms  twaren-1-lo-jmb-702.lsanca.pacificwave.net [207.231.240.133]

  7   173 ms   173 ms   173 ms  s4-ge-chti-ir2.hcc-la.twaren.net [211.79.48.229]

  8   158 ms   158 ms   158 ms  211.79.59.194

  9   158 ms   158 ms   158 ms  iepmbw.twaren.net [211.79.61.163]

From Taiwan to BNL

 1  211.79.61.190 (211.79.61.190)  0.403 ms  0.204 ms  0.236 ms

 2  211.79.59.193 (211.79.59.193)  0.233 ms  0.109 ms  0.114 ms

 3  S16-POS-EBT-R1-TPC-HCC (211.79.59.182)  1.858 ms  1.722 ms  1.738 ms

 4  s4-ge-chti-ir1.chi-tpc.twaren.net (211.79.48.194)  192.259 ms  192.235 ms  192.252 ms

 5  10g-ge-peer.chi-esnet.twaren.net (211.79.48.173)  192.500 ms  192.489 ms  192.501 ms

 6  chicr1-10ge-chislmr1.es.net (134.55.217.53)  231.244 ms  231.204 ms  231.238 ms

 7  aoacr1-oc192-chicr1.es.net (134.55.209.58)  212.471 ms  212.476 ms  212.491ms

 8  bnlmr1-aoacr1.es.net (134.55.218.45)  253.314 ms  253.324 ms  253.338 ms

 9  * * *

10  * * bnlsite-bnlmr1.es.net (198.124.216.178)  214.763 ms !X

11  *Icmp checksum is wrong

Icmp checksum is wrong

Icmp checksum is wrong

Icmp checksum is wrong

Icmp checksum is wrong

Icmp checksum is wrong

 * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * bnlsite-bnlmr1.es.net (198.124.216.178)  214.682 ms !X

17  * * *

18  * * *

19  * bnlsite-bnlmr1.es.net (198.124.216.178)  214.578 ms !X *

20  bnlsite-bnlmr1.es.net (198.124.216.178)  214.718 ms !X * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * bnlsite-bnlmr1.es.net (198.124.216.178)  214.672 ms !X *

28  * bnlsite-bnlmr1.es.net (198.124.216.178)  282.394 ms !X *

29  * bnlsite-bnlmr1.es.net (198.124.216.178)  214.647 ms !X *

30  * * *

From BNL to Taiwan

traceroute: unknown host iepmbw.twaren.net

We were unable to get any results from BNL-Taiwan using host name. so we ran traceroute using IP address. This may be because of DNS problems

From BNL to Taiwan (211.79.61.163)

1 amon.nslsusers.bnl.gov (130.199.255.1) 0.630 ms 0.580 ms 0.529 ms
2 bnlmr1-bnlsite.es.net (198.124.216.177) 0.606 ms 0.566 ms *
3 aoacr1-bnlmr1.es.net (134.55.218.46) 2.780 ms 2.765 ms 2.709 ms
4 chicr1-oc192-aoacr1.es.net (134.55.209.57) 22.787 ms 22.794 ms *
5 chislmr1-10ge-chicr1.es.net (134.55.217.54) 22.786 ms 22.756 ms 22.702 ms
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
Icmp checksum is wrong
24 *Icmp checksum is wrong
* *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Analysis

if we look at trace results from Taiwan to BNL , we find that trace routes are very smooth until eighth hop when it enters the BNL site. This point onwards every thing is haphazard and we have a loop. This entry "bnlsite-bnlmr1.es.net (198.124.216.178) " appears many times with a delay of more than 200 ms and an X! sign which is indication of restricted policy.

BNL to Taiwan dns lookup failed so we used IP address. If we look at the traces we find that things look OK until 5th hop but then every thing is blank.

Our initial understanding was that there is some problem in DNS entry for Taiwan node at BNL. We received following response for that query.