CERN Visit, July 96, Trip Report

Les Cottrell, Assistant Director, SLAC Computer Services (SCS)

Stanford Linear Acelerator Center, Stanford University, Stanford, California 94309

July 17-21, 1996

Purpose of Trip:
The trip was arranged to have discussions and share experiences with peers at CERN, concerning computing and networking support.

Private News Service

CERN is looking at how to provide a "private netnews service", similar to what Hypernews provides. They are developing such a service using majordomo to set up and maintain the private lists. When the email items for a private list are received by majordomo, they are piped to a public domain utility mhonarc (see http://www.oac.uci.edu/indiv/ehood/mhonarc.doc.html) which converts the mail item to html and it is then saved in an afs directory. Access to the news is via the Web (viewing with the local file option in the file menu), with afs providing the security. The contacts are Miguel.Marquinez@cern.ch and jluis@mail.cern.ch (Jean Luis Casas). Superficially what they are doing appears very similar to lwgate, except that file access is protected by AFS. They compared mhonarc with hypermail and preferred mhonarc since it allows more tailoring of the directory list, and also it has a command line option.

WWW - Contact Mike Sendall

CERN set up a WWW Policy group in Mar-96. They keep an eye on content of pages and usability. The group has representatives from the user community. Division leaders are responsible for Web pages. Each division has its own Web master who is a link between the division's Web cowboys and the policy group. The policy group is trying to push people into providing well managed servers. However, they do not expect to insist on a limited number of servers. Rather they will encourage what appears and is linked to from the CERN home page. They have a Norwegian visitor looking at setting up CERN corporate pages. They are looking at some guidelines for this, in particular the work done at FNAL by David Ritchie. They have also looked at some of the SLAC documents (in particular the style guide) and hope to develop CERN specific pages from them.

Mike provides a list of tools for users, including converters from various applications to HTML, and validators (such as Lint, MOMSpider).. This is available at: http://www.w3.org/hypertext/WWW/Tools/filters.html. CERN's main server has about 50K pages and there is a great need of a way to visualize the organization of these pages (i.e. Site maintenance tools). They are also looking for better search tools. They have plans to support Harvest. They want to create guidelines for page authors to ensure that hits to their pages from search engines are more likely to be relevant.

Users can install cgi scripts on the various Web servers at CERN. CERN do not appear to have addressed how to ensure the security of these cgi scripts and were very interested in SLAC's work in this area.

Nominally the CERN Web server is now supported by the W3C organization out of MIT and INRIA. In reality W3C is rethinking its approach to server support so Mike thinks little is going on in this area at W3C.

Home Access

CERN provides dumb terminal (or emulation) access, ARA and SLIP/PPP over dial up modems. Current CERN use of the analog (modem) dial in services is about 4 Khours/month or $15 KSFr per month which comes out to about 4 SFr/hr. They do charge back using the computer readable bills they get from the PTTs. They are considering providing ISDN access to the home. They are going to evaluate using a Cisco 4xxx router with PRI lines from both the Swiss and French PTTs. The choice of Cisco is since they can fairly easily borrow one from the CERN networking folks. They will use dial back in order to facilitate billing. In Switzerland the ISDN call charges are the same as for voice. I have more details on costs if you are interested. The contact person at CERN is John.Ogilvie@cern.ch. DESY are using Ascend routers with PRIs. The contact person at DESY is Michael Ernst. RAL (contact Paul Bryant) are looking at Cisco to provide their ISDN interfaces.

Local Area Networking - Contact Brian Carpenter

CERN is in the process of moving from a large bridged network to a routed network. At the same time they are moving to structured wiring. 20,000 outlets are being installed. The structured wiring started mid-95, and they hope to have all the office buildings completed by mid-97. There is roughly one outlet per 5 square meters of office space (rounded up). This comes out to roughly 2 outlets per phone (which is what SLAC has been using as a guide). The contract for the structured wiring was outsourced (Olivetti and Nokia) and includes the cable (including fiber) installation, termination and Ethernet hubs (HP). For routers they have chosen Cisco 4700s. They have 60 star points, each of which includes a fiber patch panel, a UTP5 patch, hubs, routers, bridges and monitors. The star points are in locked areas. The backbone will continue to be based on FDDI connected to the routers, with 12 port Ethernet interfaces also in the routers. They are using DEC bridges with typically ~ 40-48 Ethernet ports and separate ports for IP, IP+ATalk/IPX, and IP + DECnet. They are addressing the devices on the bridge ports so they will be able to subnet them later.

They are looking at Cisco catalyst switches for higher performance (100 Mbps support). They are not considering heavy use of VLANs at the moment, since there is little locality of traffic due to the use of centralized servers.

For cable management they are using a product from Mountain Top/Accugraph. Earlier they had (through Olivetti the outsourcer for the structure wiring) a product from ISICAD. The ISICAD product however was judged inadequate. The Mountain Top product was originally designed for phone type cabling and needed customization (50,000 lines of C code) for CERN's needs. It is backended to Oracle. It has been a major effort and still needs better error checking.

When CERN wires a new building they feed in the plans to Mountain Top, decide on the density of connections, where the plugs should be located and the cable ways and what should be plugged into each hub port. The contractor then does the rest, and CERN ends up with computerized drawings (which are printed in case the computers is unavailable, e.g. during network trouble shooting). The drawings are "never touched" since they contain the fixed information. They use SNMP to upload the state of what is connected to each hub from the hubs themselves. As they move from a bridged to a routed network their knowledge of the network configuration is improving since devices have to be registered in the Domain Name Servers.

The cable management package consumes between 0.25 and 0.5 of an FTE which includes coordinating with the vendors, and resolving inconsistencies. The contact person is John Gamble.

They are using the Cabletron Spectrum package for network management (topology, alarms, statistics). Trouble tickets are moving to Gnats. They have a WWW utility which enables a user to discover what other machines are on the same segment as you are on.

High Speed Connectivity - Contact Ben Segal

CERN has been using HIPPI for high speed interconnectivity for compute farms. Fiber Channel Standard (FCS) appears to be more focused for disk connectivity (as opposed to host connectivity), and there appear to be only prototype implementations currently commercially available and only one FCS switch vendor, Eicor.

CERN has measured HIPPI performance of 92 MBytes/sec memory to memory with 64 KByte TCP windows using SGI (IRIX 5.x). HIPPI on Ultra Sparc on the other hand is said to be a poor performer (<10 MBytes/sec) (contact Berndt Panzer for details). Digital Unix does not currently support TCP over HIPPI. Maybe Essential or Genroco have an offering for DEC PCI based Turbolaser machines.

CERN has been using Gigarouters to provide HIPPI to HIPPI connections (for up to 10 Km) with multi FDDI connections into each Gigarouter (i.e. the Gigarouters act as FDDI multiplexers). This allows connectivity to existing FDDI connected servers.

They have had good experience with Avaica HIPPI 8 port switches. They had problems with the 16 port Avaica switch and replaced it with two 8 port switches. They are interested in looking at a 16 port HIPPI switch from Essential. For IBM RS/6000 HIPPI interfaces they are looking at Essential and expect IBM to have an offering by end 1997. Sun Solaris may have a 3rd party HIPPI board but I did not get the name of a company from Ben. Ben says HP does not have a good HIPPI interface. Ben expressed doubts about NSC's HIPPI switch offering. Ben is also looking at how to connect an IBM SP2's internal switch to the outside world, and is considering a Gigarouter (they have or have announced an SP2 interface).

Tape Support - Contact Les Robertson

CERN has an IBM 3495 robot which supports 10-20 K (I think) 3480/3490 cartridges. CERN has about 234 K 3480/3490 cartridges. They also have a small (< 3000 10 GByte NTP cartridges) IBM 3494 robot which has 9 IBM 3590 drives. They also have 2 stand alone 3590 drives.

The LEP experiments have moved over to using DLT 2000 cartridges with 10 GBytes/cartridge and there are 5 Odetics DLT robots supporting 264 DLT cartridges each. The robots have a total of 15 DLT drives. There are also 32 stand alone DLT 2000 drives. Most of the active data is now on DLT cartridges. They do 6 Kmounts/week of DLT robot mounts out of a total of 18 Kmounts/week. About 50% of the mounts are to stand alone drives. DLT error rates are similar to 3480 error rates and about 1o times better than 8 mm error rates

CERN is planning to acquire sufficient automated data capacity to allow them to run in a lights out mode. They are starting from scratch and will presumably phase out the existing robots. This new acquisition is designed to last until the end of LEP or the start of the LHC. They are looking at many possible solutions including:

  • Testing Ampex DST 310 drives which use D2 technology and support up to 50 MBytes/sec on big (MByte) blocks. The drives cost about $140K each. DESY uses these and will buy more. Ampex cannot meet the robot size required by CERN, however someone else such as Grau (a German robot integrator) or Emass may bid Ampex drives.
  • Looking at DLT 7000 which were announced at the start of the year. However, only Digital appears to be behind it and do not have a product available yet. The speed is about 5 MBytes/sec and the drives are expected to be several times cheaper than the IBM NTP drives.
  • They have tested an STK Redwood drive, but had some hardware problems which took 3 weeks to fix. STK are thought to be anxious to get into CERN.
  • They already have IBM NTP drives and robots. The drives cost about $40K each. Philips are also making IBM 3490c drives and want to use the same mechanism with new heads to support NTP cartridges.
    • The decision is hoped to be made in late December with an order placed soon after.

    Disk - Contact Les Robertson

    CERN had been buying disks based on an acquisition won by Falcon and providing Seagate disks. CERN was very satisfied with this arrangement and was getting good support. Recently the contract was won by SGI who provide Quantum disks, The disks are bought in cabinets, and there have been many problems with connecting the disks up.

    Reliability is the main issue rather than transfer rates. Disk performance and connectivity, according to Les, has kept up with most CPU requirements. Les is looking at Ultra SCSI (SCSI-3) which should be twice as fast as the exiting interfaces. HP is already selling an Ultra SCSI (but it may not be fully standards compliant) and DEC is actively developing products. Les seemed uninterested in using RISC for high performance.

    Unix Support - Contact Alan Silverman

    CERN are dropping desk top support for Ultrix, SunOS and SGI. Solaris is now fully supported since it is stable and needed to support new hardware. They also support HPUX, AIX (about 150 machines plus an SP2) and Digital Unix. The support for AIX is outsourced to a full-time AIX administrator. There appears to be no move to support Linux, though it is attractive to universities and appears to have good free support via the Internet.

    Alan is chairing a committee looking at "Informatics Outsourcing for CERN". The first step is to collect information on areas that might be outsourced. From this they will make up a palette of service requirement. They will then do a market survey and identify companies who may be serious contenders (by reviewing what companies do and looking at reference sites). They will then make up a tender to about 20 companies and pick the 5/10 which are price/technically interesting. These companies will be put on a list of service companies and the list provided to groups at CERN who can decide what they want and can afford. A group will then write a service definition and purchasing will bid it. They hope to have this process ready by June 1997 so they can go to the Finance Committee for permission to implement the scheme. This effort is being driven by the member states who wish to see more money going in to the member states (i.e. member state contractors).

    Alan is procuring Windows NT machines for many of his Unix section people so they will be positioned to support Windows NT as the requirement for support for Windows NT ramps up. The Unix section is also looking at supporting tools to allow Unix desktops to access PC applications. Alan mentioned two products, WinCenter and WinDD (the latter from Tektronix and is proprietary). It is hoped that such products will elongate the usefulness of Unix desktops, and also provide a migration tools for Unix users to move to Windows.

    PC/Mac Desktop Support - Contact David Foster

    No decision has been made on moving to Windows NT (WNT), so the CERN PC desk top environment is Windows 95 (and possibly Windows 3.1) based. They provide a turn key solution called NICE where all one needs is a floppy disk to get started, then everything is downloaded over the network. There are 4 people (including the team leader David Foster) in the CN division's PC support area. They support about 3000 users. David wants to move to WNT on Intel platforms (and avoid WNT on multiple hardware platforms, at least at first) but does not have the people resources to make the move. Les Robertson is also interested in building a WNT production simulation farm. This will enable them to look at reliability issues in a production environment and provide connections to tape and disk servers. Les has identified a couple of people to work on this.

    Backup support for the Netware servers is mainly via the IBM ADSM product with two RS/6000s running AIX and connected to DLT robots. Recently IBM released a Windows NT ADSM server. CERN is glad to see it, though they do not currently have plans to move to it.

    David has concerns about the scalability of the Microsoft Exchange server. Microsoft Exchange is used to provide a IMAP type service for Microsoft Mail clients. David quoted a figure of about 150 users per Microsoft Exchange server being a maximum. He also reported that today he supports about 1800 NICE mail users (which has grown by about 10% in the last 6 months) who are using about 4 GBytes. The growth is about a GByte in 5 months

    CERN Computing Management - Contact David Williams

    CERN is merging people providing computing support (in ECP, Admin Support and CN divisions) under the CN division which will result in a stronger Computing division. I got the impression that this re-organization is universally acclaimed. Some people are unhappy about rejoining the CN division and may be looking for how to avoid it. There is also a change in the division management upcoming as David William retires (after 8 years as head of CN) and Jurgen Mai (ex DESY) takes the reins.

    CERN is using the Oracle Human Resources package. Apparently the user interface is inadequate so they are internally developing a new user interface for both PCs and Macs.

    Appendix: Itinerary of Trip

    July 16Leave Dresden
    July 16Arrive CERN
    July 17-21CERN
    July 23Return to Menlo Park

    Appendix: List of Persons Met During Trip

    Miguel Marquinez
    Jean Luis Casas
    Mike Sendall
    John Ogilvie
    Brian Carpenter
    Ben Segal
    Les Robertson
    Alan Silverman
    David Foster
    David Williams