Visit to England

October 11-25, 1999

IEPM home page | Tutorial | PingER Help | PingER Tools | PingER Summary Reports| PingER Detail Reports

Trip Report

Traveler Roger L. A. Cottrell, Assistant Director SLAC Computing Services, SLAC, POB 4349, Stanford University, California 94309
Dates of Trip October 19-October 25, 1999
Purpose of Visit To discuss with networking providers UKERNA/JANet and DANTE the current status of international networking in Europe and to learn of future plans and challenges. Also to learn of progress with the traceping project at Oxford. While at RAL I took the opportunity to discuss security at RAL.

Introduction

Most of this trip was funded and sponsored by NATO in order to attend the NATO Advanced Networking Workshop in Tbilisi, Georgia, where I also gave a talk on Internet Quality of Service. In this document I am only reporting on the DoE sponsored part of the trip, i.e. the visits to Daresbury, RAL and DANTE.

Visit to DANTE

I visited DANTE (Delivering Academic Networks To Europe) HQ in Cambridge, England on October 22nd, 1999. I met with Roberto Sabatino head of Network Engineering and Planning and Vincent Berhhout, head of operations. I also had lunch with Tim Streater who used to be at SLAC. The purpose of the visit was to better understand the current European NRN (National Research Networking) set up and learn of plans. We also discussed ESnet/SLAC and TEN-155 Differentiated Services experiments.

Production networking

DANTE is a non-profit organization (the Quantum project countries are the shareholders, customers and board members)  that provides coordinating, planning, contracting and management of the European high performance NRN backbone now called the TEN-155 network which came out of the Quantum project. The backbone is currently based on 155Mbps STM-1 (OC3) ATM circuits. The backbone lines are provided by Unisource, Belgium . There are transit nodes in AT, CH, DE, FR, IT, NL, SE, and the UK to which the NRNs are directly connected. In addition there international circuits to peripheral sites in BE, CZ, ES, GR, HU, IE, LU, PL and SI. TEN-155 was upgraded from TEN-34 which resulted in a huge increase in available capacity. As a result of the 1998 liberalization of European telecomms services, the overall cost of TEN-155 is similar to that of TEN-34. A full mesh of UBR-like PVCs is set up between the TEN-155 routers at the transit nodes. In addition DANTE has  2 STM-1 circuits from Frankfurt and London to New York (Telehouse at 25 Broadway) provided by Worldcom, that are not part of the Quantum project and are at DANTE's commercial risk. Several of the collaborating countries subscribe and us this link with the exceptions  being the UK (JANET has 2*155Mbps), Germany, the Scandinavian countries (NORDUNET had 2*155 which was upgraded to 3*155Mps in September 1999, and will go to 4*155Mbps in Jan 2000), Netherlands, France, and Switzerland. There is another add on connection to the TEN-155 networks to Israel (E3 London to Tel Aviv) and Cyprus. The day to day management is outsourced to the ULCC (University of London Computer Center) NOC (network Operations Center).

TEN-155 Managed Bandwidth Service (MBS)

DANTE provides MBS to serve specific research purposes and experiments in the European academic and research community. The network resources are defined as bandwidth requirements, lifetime of the established connections, traffic profile and a complete set of network parameters (http://www.dante.org.uk/mbs/information.phtml). An initial limit of 10% to 20% of the exiting bandwidth between the NRN and TEN-155 is expected to be used to cater for service requests. The MBS is an open ended service whose availability depends not only on the TEN-155 but also on the NRN network infrastructure. The tests so far have been successful, however one observation was that while ATM allows for great flexibility  - since it is a level 2 service - it places a heavy burden on the users of the service who must configure the IP level routing and set up an IP VPN themselves, better ways are needed to integrate IP and ATM services avoiding the need for reconfiguration at the IP level. There were also problems understanding and coordinating between the contacts and procedures of the various NRNs to implement the required VCs (Virtual Circuits). Other future requirements include: obtaining a list of prices for connectivity in every NRN; the need to achieve bandwidth brokerage faster from the set of providers involved (i.e. need a timescale equivalent to the lifetime of the link) 

Differentiated Services (DS) Experiment

DANTE has been involved with many of of the Quantum NRNs in a test of DS. There are test sites at CERN, DANTE, GRNET, INFN/GARR, RedIRIS, SWITCH, University of Stuttgart, University of Twente and University of Utrecht. They have made extensive tests of Cisco implementations of CAR (Committed Access Rate) for marking and policing and Cisco's CB-WFQ (Class Based Weighted Fair Queuing) for scheduling traffic streams through the routers. They have made these tests with both UDP and TCP and show that with the latest releases of the Cisco code both perform as advertised and are an effective mechanism for traffic isolation. They have also explored various settings of the possible parameters. We (SLAC networking and the ESnet team) need to study their results in some detail to help with the ESnet/SLAC QoS pilot. 

Visit to Daresbury

At Daresbury I met with Robin Tasker and Paul Kummer and gave a talk on Internet Quality of Service. Robin has been working on a new IEEE draft standard for Ethernet spanning trees (IEEE Draft P802.1w/D1) that is designed to provide rapid reconfiguration in a switched network. This is hoped to reduce the time to reconfigure after a loss of spanning tree from 30 seconds to sub-second time which will be a big help for providing more available networking. We also discussed where Daresbury might get more  involved with the PingER project and agreed that the area of looking at ICMP rate limiting looks promising. Maybe they could be a beta test site for the code when it is ready, it would give another site from which to look for the limiting.

Visit to RAL

At RAL I met with Dr Trevor Daniels, Dr Paul Jeffreys ,  John MacAllister and Bob Day. I also presented a talk on Internet Quality of Service.

RAL Security

Trevor Daniels says RAL looks for scans and blocks the appropriate networks. They have 50-70 blocks of this kind in the 3COM border routers. They leave the blocks in for a month. Occasionally they make a mistake in the configuration.. One mistake resulted in a compromised machine at RAL. They use NetXray with filters to look for SYNs (attempts to set up connections) and use the filters to expose scanning patterns. This is no longer very effective since reviewing of the results is manual and modern scans often attack at the same time so by the time the log is reviewed it may be too late. A side effect of the blocks is that the router is very busy. This is believed to cause congestion to the RAL site which may be the cause of the increased packet loss PingER has seen to RAL.

They have also divided the IP addresses on the site into two ranges, one of which is for machines that provide externally available services. then they block SYNs to non external range hosts. This went in very smoothly and works well. The worst compromise they saw was on Solaris 2.6 which affected a few hosts that had not been patched. There were 19 Solaris 2.6 hosts and all were scanned, and the 6 that had not bee fully patched were compromised. They discovered the hosts from the logs, and it took 1.5 days to block it.

They are starting to apply protocol filters in the internal routers at RAL (i.e. a server has to be registered before the protocol is allowed to it).  They have done this for the IT division so far, and will extend it to other divisions. They do no blocking of UDP from offsite and see very little UDP activity (I don't think they run AFS).

They are concerned about mail born viruses (e.g. Back Orifice) and are trying to centralize all mail services on Microsoft Exchange. They use Network Associates VirusScan in the desktops. They do nothing to protect against SPAM. To get a DHCP address/service one has to register the MAC address first.

Traceping

John MacAllister says he will have a Perl5/Linux version of traceping ready to try at SLAC in a couple of weeks. He will need a Linux account at SLAC to install, configure and manage. He will also need a directory on the Linux machine that is available to the web server. the new version has a route option (simply sends one packet to each node along the route), it also allows setting of the number of packets per hop, the packet size, and the interval. With the VMS version he ran out of disk space at SLAC, so he wants to add an option after which to archive, and another limit after which to drop archived data. he is also looking at the number of processes (currently one for each Beacon site), but is concerned about how to recover if a site hangs. He wants to look at providing graphical schematic output of the results.

UKERNA-JANet

JANet (Joint Academic Network) is the UK NRN that is run by UKERNA. Bob Day reported that they ran into problems with Macintoshes when they turned on CAR. Apparently the Macs did not work if the TOS bits in the returned packets had had their values changed. they had to back out the use of CAR until they can get the Macs up to a later release (it is OK with the Y2K compatible version of the  MacTCP stack - apparently someone in New Zealand discovered and identified the problem.

They have concluded contractual agreements with Teleglobe to provide IP transit for commodity traffic for JANet. They also have agreement for a link between the JANet router in New York and the Abilene and ESnet routers. They hope to get the peering set up in the next 2 weeks (the deadline is some big show, possibly SC99, in November). They are subscribing to 45Mbps of peering bandwidth in New York, with 34Mbps to Abilene and possibly 10Mbps to ESnet. They expect to upgrade from 2*155Mbps to 3*155Mbps in the early stages of 2000. They are very interested to know the timing of when ESnet will have a link to 60 Hudson St in New York or whether anything has been decided yet. JANet has no links to STAR-TAP. 

UKERNA charges institutions based on incoming usage from the US in order to help raise half the cost of the Trans-Atlantic link. This did NOT modify demand noticeably. Mandatory caching of web pages is also in place. There is no charging back of costs back to the users. Monitoring the usage is a big headache. They use Netflow which creates vast amounts of data, also as the load on the router increases then Netflow flows are lost (routing takes priority). The newer Cisco GSR router does not provide support for NetFlow.

The BaBar pilot managed bandwidth pilot will be extended started in May, will be extended to March 2001. They want to copy 5.5M events and provide a mirror Objectivity site in the UK. They expect to need 2Mbps continuously between RAL and Manchester and Imperial College.

For the next generation UK NRN called Super JANet 4, they expect to see big drops in telecommunications costs/bit in a year or so and will hold steady until these appear. Of some concern is the consolidation of Telecommunication companies by mergers etc. which may remove competition.

Full Itinerary

I report the whole itinerary of this trip, though the part from October 11 through October 19 was sponsored and paid for by NATO.

October 11, 1999  Leave Menlo Park
October 12 Arrive London
October 13 Leave London, arrive Tbilisi
October 14-15 Visit Internet Center in Tbilisi Georgia
October 16-18 NATO sponsored Advanced Networking Workshop, Georgia
October 19 Fly Tbilisi- London
October 20 Visit Daresbury, near Liverpool
October 21 Visit RAL, near Oxford
October 22 Visit DANTE near Cambridge
October 23-24 Weekend
October 25 Fly London - San Francisco, arrive Menlo Park

List of persons met during trip

Daresbury Robin Takser, Paul Kummer
RAL Trevor Daniels, Paul Jeffreys, John MacAllister, Bob Day
DANTE Roberto Sabatino, Vencent Berkhout, Tim Streater

[ Feedback ]