Loss of connectivity from SLAC to multiple sites for about ten hours Network logo

Les Cottrell. Page created: September 2, 2004.

Central Computer Access | Computer Networking | Network Group | More case studies
SLAC Welcome
Highlighted Home
Detailed Home
Search
Phonebook

Problem

Richard Dubois of SLAC reported the following by email around 8:30am September 2, 2004:

We cannot connect from SLAC to the CERN www.vrvs.org web meeting site this morning. Is there a local reason why? Other members of our group (eg from Goddard) can connect to it. Our 08:30 web meeting is in danger...

Analysis and Cause

The node www.vrvs.org was pingable at 09:00. The
traceroute (and RTT) indicated that it was at Caltech. Testing with
telnet www.vrvs.org 80
indicated that the web port was responding. Also www.vrvs.org responded to a web browser.

Looking at the pings to CERN that are made each 10 minute interval for this period there was no break in connectivity. On the other hand pings from SLAC to Caltech failed from about 15 mins past midnight to a bit before 8:57am.

Looking at the traceroutes from SLAC to IEPM-BW sites SLAC's connectivity to several sites (mainly .edu, but not all .edu) were interrupted during that time. After recovering many of the sites had different routes than before the outage.

Looking at the CENIC maintenance page there was a scheduled outage from midnight to 5:00am. We contacted the CENIC NOC at about 11:15am and they said things were restored about 10:30am.

Looking at the topology of the routes to sites that lost connectivity it was apparent that the common factor wass CENIC.


Page owner: Les Cottrell