Network connectivity between SLAC and BINP, Novosibirsk

Les Cottrell. Page created: December 6, 2000.

Central Computer Access | Computer Networking | Network Group | ICFA-NTF Monitoring

Introduction

The Budker Institute of Nuclear Physics (BINP) in Novosibirsk, Russia is a major High Energy Physics (HEP) site and in particular has strong collaborations with SLAC. In order to facilitae physicsist at BINP collaborating with sites in the U.S. it is critically important for BINP to have good connectivity to ESnet and Internet 2.

Routes

The traceroute to BINP from SLAC shows the route goes via KEK in Tokyo, Japan. The pingroute of the route shows that, as expected, the delays kick in between Sunnyvale, California (snv-slac.es.net) and Japan, and between Japan and Novosibirsk. The pchar output indicates that the bottleneck bandwidth is 128kbps between Japan and Novosibirsk. The link from Sunnyvale (snv) to Japan is a 10Mbps ATM circuit which as of October 2000 was not saturated (Jim Leighton, ESnet).

According to email from Serge Belov of BINP sent December 8, 2000:


- The main channel is based on 128K link BINP-kek leased from
  Rostelecom and Japantelecom. This link works almost fine.
  Connectivity to SLAC and some other DOE sites is provided
  by our KEK colleagues as our tiny 128K could not overload
  their at least T1 connection between KEK and SLAC. Unfortunately
  that part of the whole route is working not so stable and we've
  observed some troubles even this November.

- Alternative route is based on 700K satyellite link BINP - DESY.
  The main problem here is that this link is used also by many
  organizations of Novosibirsk Scientific Center, so our share
  is not more than 10%. Of course only small part of these 10%
  is available to BINP-SLAC exchange.

RTT & loss

The round trip times (RTT) and losses from SLAC to BINP for August to December 2000 measured by the IEPM/PingER project are shown below. It can be seen that the RTT dropped from about 400msec. to under 300msec. in October 2000. However, around November 15t the the packet loss increased quite dramatically. Again the reduction in RTT and increase in loss to about 5% can be seen plainly. A 5% loss will make interactive work (e.g. telnet, ssh, Xwindows) difficult. For more on the impact of losses see: Tutorial on Internet Monitoring & PingER at SLAC.
RTT & Loss SLAC - BINP Aug-Dec 00

A longer term version of the RTT and loss going back to mid-1999 with one point per month (representing the median) is shown below where the measurements are made from the Novosibirsk PingER monitoring site to remote sites around the world (23 in N. America: US & Canada; 3 in Latin America; 16 in W. Europe; 5 in E. Europe; 2 in the Former Soviet Union; 8 in E. Asia; 1 in India; and 1 in the Mid. East). Again the decrease in RTT and increase in loss in October is evident.
RTT & Loss BINP to the world Mid 1999 thru Dec 2000

RTT & Loss BINP to the world Mid 1999 thru Dec 2000

Yukio karita of KEK sheds some light on these losses in email to cottrell 12/11/00:


There were at least three troubles which affected SLAC's connectivity to BINP
since this October.

1) As you know, the contractor for the NII(NACSIS)'s external links was
 changed from KDD to NTT on 1 October.   NII's POP in USA is located at NTT
 San Jose Office.   NII's USA link has a capacity of more than 300Mbps. 
 A 10Mbps ATM PVC connecting KEK and ESnet Sunnyvale POP was created within
 it.   (ESnet Sunnyvale POP to NII SanJose POP is made with a DS3 circuit
 prepared by Jim Leighton.) 

   The transition was made almost smoothly, however there were some breaks
 several times.     KEK's connectivity to ESnet was backed up by the APAN.   
 However this back up doesn't work for the BINP-ESnet connectivity.   

   Attached at the end of this mail is a mail which I sent to routing@es.net, 
 where I requested to create an IP tunnel for KEK and ESnet to peer over the 
 APAN link.   This IP tunnel is to provide a back up route for the ESnet-BINP 
 connectivity.   However I haven't received any response from routing@es.net 
 so far.   
     
2) We had some trouble at our Fore-ASX200BX, where the ESnet-KEK ATM PVC is
 terminated.   It took place at about 10:00 on 24 Novermber JST, and was 
 fixed at about 16:00 on 29 November JST.   The KEK-ESnet PVC was not 
 available then.   KEK's connectivity to ESnet was backed up by the APAN, 
 while BINP's connectivity to ESnet was not backed up.

3) Some miss configuration was made in NII's domestic ATM backbone at about
 8:00 on 21 November JST.   It introduced a high rate of ATM cell losses.   
 It affected the KEK-ESnet PVC too.   The miss configuration was fixed at 
 about 10:00 on 4 December JST.

Plans

According to email from Serge Belov dated December 8, 2000:


- The main direction of improving the connectivity between our sites
  should be either expanding that existing BINP-KEK channel upto,
  say, 512K - if our KEK colleagues are agree - or establishing
  another dedicated link Novosibirsk - SLAC. Currently both options
  are being considered and evaluated, but I can't provide you with
  any kind of time schedule.

Email from Yukio Karita, KEK on 12/11/00:


We are also very interested in getting higher speed/quality to BINP, and
have been discussing it with Japan Telecom.   The difficulty is in budgeting
the Russian half circuit.   BINP once said to me that they would try to
get the budget for 1.5Mbps from the Academy, but no good news has come yet.
So if USA side can help BINP for the Russian half circuit, it might be a good 
solution.  I believe KEK can pay for the Japanese half circuit.   An 1.5Mbps
leased line or an 1.5Mbps frame relay might be good for the next step. 
I don't worry about the impact of the 1.5Mbps on the KEK-ESnet link.
I believe it is not so difficult
to get more bandwidth (on the link from KEK to the US) if needed.   
NII(NACSIS)'s bandwidth to USA from 
where our 10Mbps is obtained has been and is going to be doubled each year.

Email from Serge Belov Dec 12, 2000 concerning upgrade costs (note these are the half circuit costs seen from the Russian end):


For the current scheme with transit through KEK 512K link will cost
10,350K$ monthly (Rostelecom prices) for international part and much 
less for local loop - about 500$ per month. 


One time payments are estimated to be similar to installation cost 
for a new channel, that is about 1K$ (this includes testing, wiring,
commutation etc).


This upgrade is possible with already installed E1 equipment where we
will thus consume 1,536Kbps and will have spare 512K that should be
reserved for our Moscow connectivity (which in turn also needs to be
upgraded by at least 512K).


More serious upgrade (upto mentioned T1) will require installation
of additional equipment - WAN Interface Card for this Cisco (800$)
and the pair of HDSL modems - total about 2K$ plus some time for 
delivering this things and connecting all this together. 


In this case monthly payments could be about 20K$ for intnl part
plus something around 500$ for local loop.


The variant with another dedicated link BINP-SLAC seems actually
not necessary provided KEK agreed with first scheme.


As I was informed Rostelecom already was requested by Japan Telecom
to consider this upgrade and the only problem will be to find 
funding for this project. This is the most uncertain topic.

[ Feedback | Reporting Problems ]
Page owner: Les Cottrell