Problems with Connection between SLAC and BINP/Novosibirsk - July '0 Network logo

Les Cottrell. Page created: July 25, 2006

Central Computer Access | Computer Networking | Network Group | More case studies
SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook

Problem

Andrey Yushkov of SLAC reported:
Sent: Wednesday, July 19, 2006 9:19 AM
To: help
Cc: net-admin
Subject: problem with SLAC-BINP(Novosibirsk, Russia) connection

 Hi,

 We have a problem with this connection, the tracerout output:

ayushkov@swan:~>traceroute star.inp.nsk.su traceroute to star.inp.nsk.su (193.124.167.6), 30 hops max, 38 byte packets
 1  rtr-core1-pub1a (134.79.23.2)  4.342 ms  0.360 ms  0.328 ms
 2  rtr-dmz1-ger (134.79.135.15)  0.429 ms  0.409 ms  0.406 ms
 3  slac-rt4.es.net (192.68.191.146)  0.516 ms  0.477 ms  0.455 ms
 4  slacmr1-slacrt4.es.net (134.55.209.93)  0.510 ms  0.435 ms  0.415 ms
 5  snv2mr1-slacmr1.es.net (134.55.217.2)  53.362 ms  0.882 ms  0.754 ms
 6  snv2sdn1-snv2mr1.es.net (134.55.207.37)  0.900 ms  0.868 ms  0.789 ms
 7  chicr1-oc192-snv2sdn1.es.net (134.55.209.54)  53.887 ms  54.292 ms
49.388 ms
 8  aoacr1-oc192-chicr1.es.net (134.55.209.58)  69.017 ms  80.040 ms
68.959 ms
 9  aoapr1-ge0-aoacr1.es.net (134.55.209.110)  69.110 ms  71.294 ms  69.073 ms 10  198.124.216.126 (198.124.216.126)  250.102 ms  251.509 ms  249.877 ms
11  keksw2-ns.kek.jp (130.87.4.35)  249.995 ms  250.298 ms  250.269 ms
12  kekcis7.kek.jp (130.87.43.7)  250.653 ms  250.286 ms  264.913 ms
13  * *

the ping output:

ayushkov@swan:~>ping star.inp.nsk.su
PING star.inp.nsk.su (193.124.167.6) 56(84) bytes of data.

--- star.inp.nsk.su ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4045ms

Results

Between 4:10 and 4:17am 7/19/06 PDT rainbow.inp.nsk.su stopped responding to pings, traceroutes, and thrulay probes to TCP port 5003. However it does appear to respond to pathchirp probes on UDP port 8365 (see
here). Further investigation showed that the following ports on rainbow.inp.nsk.su did not respond: TCP ports 22(ssh), 80(www), 23(telnet), 52 (ns), 25 (smtp), 21 (FTP), 37(ntp).

The problem was reported to ESnet and Sege Belov at BINP. Esnet did some follow up:


We checked our access to the hosts listed and we also can not get to the
host star.inp.nsk.su. failing at exactly the same point as what is listed
below.
It also appears the ping rainbow.inp.nsk.jp does not return a ipaddress host
lookup failure. ping: cannot resolve rainbow.inp.nsk.su: Host name lookup
failure.

The 198.124.216.126 address is our peer with kek so traffic is getting
through to the other side of the pond.

ESnet is not opening a ticket as this appears to be a host level problem,
however we have saved the information and will open a ticket if you would
like us too.

If you think that we can do something more please let us know.
 
Tracing route to 193.124.167.6 over a maximum of 30 hops

  1     1 ms    <1 ms    <1 ms  esnet-office1.es.net [198.128.1.5]
  2     1 ms    <1 ms    <1 ms  lbl3-esnet3.es.net [198.129.76.25]
  3    55 ms     6 ms     1 ms  lblmr1-ge-lblrt2.es.net [134.55.209.21]
  4     2 ms     1 ms     2 ms  slacmr1-lblmr1.es.net [134.55.219.10]
  5     2 ms     2 ms     2 ms  snv2mr1-slacmr1.es.net [134.55.217.2]
  6     2 ms     2 ms     2 ms  snv2sdn1-snv2mr1.es.net [134.55.207.37]
  7    50 ms    50 ms    50 ms  chicr1-oc192-snv2sdn1.es.net [134.55.209.54]
  8    70 ms    70 ms    71 ms  aoacr1-oc192-chicr1.es.net [134.55.209.58]
  9    70 ms    70 ms    70 ms  aoapr1-ge0-aoacr1.es.net [134.55.209.110]
 10   252 ms   251 ms   251 ms  198.124.216.126
 11   251 ms   251 ms   251 ms  keksw2-ns.kek.jp [130.87.4.35]
 12   252 ms   252 ms   252 ms  kekcis7.kek.jp [130.87.43.7]
 13     *        *     ^C 

Resolution

On 7/23/06 we received email from Serge Belov of BINP:

As I was informed today (Monday), that last week there were numerous 
problems with power lines in so called upper zone of Akademgorodok. 
Some switching equipment went mad for several (upto 10-12) hours. 
It happened on 19-20 July.
This time I was travelling from Irkutsk to Nsk by train without 
IP-connectivity and wasn't able even diagnose the problem.

Few technical notes:

- the only widely opened machine of BINP is rainbow, 
  star.inp.nsk.su is not suitable for network testing

- SLAC has routing to BINP via GEANT-Moscow-Samara, not via KEK

Page owner: Les Cottrell