Santa Fe August 31-September 2
Rough notes by: Les Cottrell
There were about 26 attendees most (18) of whom had laptops. There was a LAN hookup but the T1 Internet access provided by US West did not work until the end of the first day of the meeting. Reinhold Mann of ORNL was introduced as the new Biology representative replacing Weiss. Guy Almes from UCAID/Internet 2 and Linda Winkler for the NGI testbed & StarTAP were also present.
Terry Disz filled in for Rick Stevens at this meeting. There is an intention to meet with the new DoE CIO to brief him about ESnet. There was a meeting in June and the CIO had a voice session at this meeting (see below). A memo is needed to Seweryniak about FBI connection for Laboratoty computer security. Charles Karney of PPPL is joining the International group.
Bill Richardson still secretary, Ed Oliver is the new associate director, Dan Hitchcock will continue as MICS division head until next April.
There is a big emphasis on security especially following the China incident. They are trying to centralize security. CIO sets policy & guidance and Labs will implement, Office of Independent Oversight will check for proper policy and proper implementation. Unclassified and classified program security will be combined under new security czar (General Habiger). Presidential Decisions Directives are being applied with increased rigor. PDD-12 for foreign contacts., PDD-39 for counter terrorism, PDD-61 for counter-intelligence (hiring counter intelligence professionals to work at Labs) and PDD-63 is for protecting Americas critical infrastructures. There is a policy of zero tolerance for violations of security for sensitive materials. Several DOE facilities (including SLAC) are exempt from the foreign visitors requirements since they have no classified work. DoE sites will establish a program to continually monitor computer systems for security, random audits will be made. DOE sites will make better use of technology to combat the hacker and espionage threats. The new Office of Oversight and Performance Assurance will establish a program of continuous independent oversight of cyber-security with support from the Office of Counterintelligence.
GAO is doing an audit of computer based controls for security. ESnet briefed GAO on Aug 18, 1999, penetration tests will be done of Labs without notification. Sandy will put this (lack of notification) on the agenda at the SLCCC in the next few weeks.
ESnet has been asked for a connection by the FBI. It has been approved for implementation, the MOU is in its final stages. This is for the FBI to monitor email and do intrusion detection for ESnet sites that request this help. Initially (FY1999) the Labs who will take advantage of this will be LLNL, SNL. LANL & PNNL and each will have an FBI T1 link, in FY2000 the capabilities will be extended to a total of 12 sites across the DoE complex and some of the connections will be T3/OC3.
Large Scale Networking (LSN) has put together a plan to request NGI expansion to 5 years (currently only 3 years).
DOE NGI FY1999 see http://home,doe.gov/news/releases/ FY2000 request is for $15M ($6M application testbeds, $6M R&D, ...)
PITAC needs reporting on geographic reach, minority colleges. They are looking for specific instances of technology transfer to industry. There is a review of DOE in Oct 6, 1999. The last review indicate a strong need for end-to-end network performance measurements to document testbed performance. PITAC recommended continued funding for FY2000. PITAC said research is focused to much on near term results, no systematic measurement of bandwidth, and QoS to the desktops, they (PITAC) stated that end-to-end performance was about 10MBytes/s for IP and 0.6MBytes/s for TCP and they are disappointed in this. Next review is in Jan-2000.
SSI goal is to revolutionize our ability to solve scientific problems of extraordinary complexity and to apply these resources to scientific endeavor. They want to integrate DOE NGI/SSI programs, increased university connectivity, increased international bandwidth, DOECN, QoS services in testbed, follow on ESnet contract renewed.
SBIR (set by law to 2.6%) has White House interest awarded $600K phase 1, $3.8M this year.
DOEnet business case released. Phased implementation, 1 Nov 1999 first phase deadline, interest by CIO to use ESnet and a VPN test case has been set up between ORNL/JLAB/DOE HQ.
Summary ESnet continues to provide high quality production level services to ESnet community, need to work closely with other agencies and I2 to provide connectivity to Universities. Must be positioned to support the requirements of the SSI, must ensure support for NGI activities. There is increased emphasis on network security and new DOE directives. Don't expect any change in ESnet budget from last year.
List of bad sites, why are they bad, outlook. Seweryniak forwarded 1998 list of DOE universities ordered by packet loss & funding.
Worst were UConn & Brandeis. Talked to Richard JOnes of UConn a nuclear physicys with interest in accessing JLab Will be an I2 OC3 connection. Brandies talked to Craig Blocker hep works with FNAL & CERN, university has 2 T1 line. Campus is wiored but university does not support internet connectivity, endowment is small, near term solution is HEP group to get connection via MIT. Jim is looking at providing a direct connection but there are issues over the router use. Wayne State has moved to Abilene and should get better, it needs to be watched. Fred Harris of the University of Hawaii ping time is 150msec. University of Mississippi has T3 to Jackson State and this is hoped to be solved soon. UCinn will also join Internet 2. Duke just joined vBNS were not aware of problems.
Most universities plan to move to I2. Researchers are aware of I2 problems when 5-10% loss, but unaware at 1% loss problems. Researchers are interested in networking monitoring measurements and think it will provide ammunition to get the university to improve their internet connectivity.
Optimizing networking between universities and DoE Labs, example ColoState <=> SLAC. Have changed routing to to go via Sacramento instead of Chicago.
Data transfer rates on ESnet are comparable to what could write to Exabyte tapes. Expect researchers to exploit this method of data transfer, cheap televideo may be realized. Monitoring ESnet-Abilene interconnect will become increasingly important.
Spreading network monitoring to research collaborations. Increase awareness and understanding of internet to university researchers by placing network monitoring links in research collaboration pages. Get researchers to see monitoring and especially comparisons between different universities. Several pages have been set up focused on major collaborations. Remarks we need to increase visibility in collaborations, difficult to find network measurement pager in collaboration pages, need to simplify information so even PI can understand. Develop more useful metrics and translate measurements into more understandable quantities such as available transfer rates.
Most loss is < 1% and university researchers are unaware of problems at 1% level. Optimize ESnet-Abilene conectivity routing & monitoring, continue to increase researcher understanding of networks (collaboration pages, simplified presentations)
See http://www.slac.stanford.edu/grp/scs/net/talk/essc-aug99-uni/index.htm
Marty is very interested in providing web accessible plots of historical use of links by sites. Marty & Les talked to Jim about this. Jim has some concerns about making site specific data available via ESnet reports. Les & Marty will pursue further with the ESnet folks.
There will be a Chautauqua video conference in September which will involve several universities and Labs. It will be used to demonstrate applications such as controlling microscopes. Tests will be made on September 7-9 and 13, and with the real conference on the 14-15. It will be interesting to look at the impact of the conference on ESnet links etc. See http://chautauqua.bu.edu/ for more details.
Jim Leighton is interested in peering with the Israeli R&E network at Star-TAP.
See http://www.slac.stanford.edu/grp/scs/net/talk/essc-aug99-icfa-scis/index.htm
This is part of the President's IT**2 initiative and DoE is involved to the tune of $70M. Vision is to utilize power of advanced simulation to do scientific research. There are 3 major area, global systems, combustion and research & engineering. These are built on a Terascale (Tops) computing environment. 5Tops system by 2000, 50Tops system in 2003. A major requirement will be the environment (e.g. systems software, IP stacks, interfaces etc.) to take advantage of these machines. The applications level systems software includes distributed data management. Will need ultra high speed networks. They are doing a survey on what is being done in DoE in simulation and what is needed. Following this they want a Computer Simulation Initiative workshop (20-30 invitees total including networkers (15 including vendors), hardware (machines), applications) in early Fall (Oct-Nov) to cover networks and Terascale facilities, this will lead to working groups and a plan (Feb 2000) followed by a review. Currently there is no funding on the House side, the budget caps make it hard to get money for new start-ups, it is hard to understand what will happen, it (budget prediction) is very chaotic. Want to lay out a long-term plan (basic concepts & precepts) for the DoE Super Computing laboratory infrastructure.
There are big requirements from Fusion and HENP to access SDSC. NERSC is building on off-site computer center in Oakland, which will need an OC12 to ESnet. NERSC is willing to pay the incremental costs. The ASCI sites (LANL, SNL, LLNL) are asking for OC12 upgrades to talk between themselves.
There is a need to come up with a general planning/strategy for upgrading links to Labs. This will become more important when there are estimates of upgrade costs based on the new ESnet contract.
Jul 99 37.5 Gpkts, 18.9 TB accepted, Jul 98 29.4Gpkts & 10.8 TB. Exponential growth, almost factor of 2 per year.
Domestic issues:
NAP Peerings
University access
International links
New NY hub configuration proposal will have T3/OC3 between Telehouse & 60 Hudson. 60 Hudson has OC3 to BNL & T3 to PPPL and T3 to NJ/NAP
Research & Advanced technology
DOE business net - DOEnet, now called DOEnet. Interest is high at CIO level for investigation of the issues related to merging of ESnet and DOEnet. DOEnet personnel estimated cost of merging with ESnet to be $40M. Have held meeting with CIO, have held one meeting with DOEnet personnel - no follow up as promised; letter to DOEnet personnel is now being drafted. DOEnet project is proceeding: "unacceptable" security architecture required (i.e. no firewalls between sites, just a firewall to the outside world) - one site has refused to allow connection.
Security: GAO is doing an audit of unclassified cyber-security in DOE: have visited LBNL, requested review of ESnet, also requested review of ESnet via HQ, help review on 18 August, have now written and web-posted a new ESnet security plan which has been given to GAO.
ESnet3 procurement: they have the responses to the solicitation and are evaluating the proposals, they will rank and do extra discussions. Oct-Nov 99 sign contract. Sprint did not bid (the official statement is they are focusing on services rather than transport) so there will be a transition to a new vendor which will take about a year. There will be extra costs due to the overlap (manpower, equipment, installations), so cutting corners to save money (e.g. removing unused T1 lines, delete OC3 to Perryman, eliminating all outstanding order with Sprint (SLAC, BNL, Seattle OC3, all NY hub orders)). Propose to do an ESSC executive briefing (by video) of installation plans after costing negotiation completed (presumably around November 1999).
Digital collaboration services (DCS) replaces VCS with new hardware (new Accord MCU) & software. It is currently in the 1st month of production, VCS will retire in Oct. Bridge audio (Latitude audio bridge) and video and data conferencing (PictureTalk), will have individual user authentication (versus room based). It does not support VIC/VAT, will be added in next release scheduled for release in October/November 1999.
Strategic implementation plan - requested by the last review panel. But technology churn is extremely rapid so long term views are hard and possibly meaningless. Networking is dependent upon the voluntary cooperation with many peers so one has very limited control of the resources or the opportunity to plan for many resources, so need to be agile and accept/adopt/adapt to opportunities. Therefore any plan must be short term and can't be strategic except at the generic level.
ESnet will continue as a staff, vendor, community, academic, commercial & international collaborative effort. Major services will be provided under Sprint contract, under an overlapping contract for advanced services. On going search for cost-effective services including volume & special discounts, hubbing arrangements, local loop cost reduction, "free" peering arrangements. Connections to major national peering points will be emphasized for external connectivity: FIX-W, MAE-W, Chicago. NY NAP, MAE-E, PB NAP & others, private peering mya be necessary in near future. Will provide connectivity to universities.
International connectivity will focus on big 7: Canada, CERN, UK, France, Germany, Italy, Japan. Target plan calls for aprpoximately 5-10Mbps to each over next few years, increased funding will be needed at $200K increment/year. Approach meet at US peering locations: Canada @ Chicago, CERN @ Chicago ...
Research areas include: QoS - policing & shaping, priority of service end-to-end, reservations, edge & leaf node requirements, many ancillary services required; high performance technology - IP/SONET (but lose some nice features of ATM such as providing an PVC for managed bandwidth), (D)WDM reliability, striping, cost effective, high reliability high performance architectures; multi-media support - VoIP, video streaming, video conferencing.
Increased emphasis on doing applied network research in specific areas of interest: security - VPN, gateway, router-based technology; advanced protocols MPLS and IPv6.
Future services: continue to develop collaboration tools and services, streaming video.
ESnet in August was allocated the first ever production IPv6 address prefix worldwide by ARIN (The American Registry for Internet Numbers). This was the culmination of over 2 years of activity.
ESnet leading a world-wde initiative called the 6REN (IPv6 R&E network). It went into 7*24 production in mid-July (see http://www.6tap.net )
Now we need production applications running over IPv6. DoE is strongly supporting ESnet IPv6 leadership efforts for this. A 1st application has surfaced using remote tomography using electron-microscope application at San Diego via Java applets. Links from UCSD to Chicago to Osaka will be stress tested (40mbps). Sun will need to make Java run over IPv6 and there may be some problems with Cisco configurations, vBNS will need to expand their IPv6 net to UCSD/SDSC in September, SDSC is building a native IPv6 path to the CDMA lab in September.
ESnet continues its rollout of IPv6 to ANL & ORNL - SLAC & LBNL are already up (SLAC is working on bringing up PingER on IPv6). See www.6ren.net www.6ren.net and www.6bone.net
Securenet = classified WAN for ASCI currently running across ESnet. ASCI revolves around a few (one or two) machines at 2 or three sites which are used by users over the WAN. They need encryption done by hardware devices (GTE FastLane) in front of each sites' secure net. There is only one level of security (unless you count no extra security). The encryption devices allow them to use insecure protocols (FTP, Telnet ...) across ESnet since they are encrypted first. The use of the network enables them to have faster turn-around. The number of people using Securenet is increasing, especially for email. Next year they will do video teleconferencing across Securenet. They do not currently have plans to do VoIP over Securenet.
1999 3TOps, 2000 10TOps, 2001 30TOPs, 2004 100TOPs, and the scaling is 1 Gbps for 1 TOPs. the encryptors do not scale as fast (e.g. in 2004 the encryptors will only go at 10 Gbps (OC192), in 2002 only 2.5Gbps (OC48)). The links and encryptors will probably need to run as parallel pipes.
The original security model works but at a significant price on complexity, management, performance etc.
Lisa Erspamer said that there is no completion date for the BNL or SLAC OC3 link installations. The orders were placed several months ago. The problem, according to Sprint, is with the telecommunications companies (RBOCs). With the difficulties of dealing with Sprint, the new ESnet contract, and the need to save money to pay for the new ESnet contract and the migration to it, the Sprint installations will be cancelled (Sprint has not bid on the new ESnet contract). The new contract is planned to be signed in November 1999. It is unclear when the BNL & SLAC upgrades will happen, the new vendor will have new prices and need to build new connections, plus there will be an emphasis on a transparent migration. At best, it appears the BNL & SLAC upgrades will not happen until Spring of 2000.
Bill Lidinsky has an offer out to someone to join HEPNRC to work on Internet end-to-end performance monitoring. The person is an internal candidate from FNAL.
Bob Fink would be interested in presenting a computing seminar on IPv6 at SLAC. Bob is very interested in where SLAC is with its IPv6/PingER project.
Larry Price, Jim Leighton and Les discussed a request from KEK (Yukio Karita) to provide more direct connectivity to ESnet on the W. Coast (to reduce RTT by about 80 msec and thus improving performance). We also discussed a second request from KEK to ESnet to provide connectivity for KEK to US universities.
These are multi-lab/institution projects with a focus on a subset of tools & technologies with pilot projects to test tools & gain acceptance/visibility. Started thinking about 4 years ago, there are 7 projects: shared virtual reality, software infrastructure, collaboration management, security infrastructure, electronic notebooks, floor control, QoS. The separation of these projects has decreased as they began to work & interact together more. There are also 2 major projects with diesel combustion collaboratory and climate control, plus some other projects in fusion and HENP. See http://www.mcs.anl.gov/DOE2000/ for information on the projects. DoE has taken a leadership in this area.
Deployment issues include: integrating the emerging tools into on-going science programs; support of tools for installation, training & infrastructure over the long term; commercialization; interoperability & standards. Despite problems and rough edges, tools are being used, new science is happening in new ways, other collaboratory projects are starting driven by interest of scientists, user communities are growing, some very quickly.
The purpose is to enhance DOE's ability to solve the diesel combustion problem, involves LANL, LBNL, LLNL, SNL Cummins, Caterpillar, Detroit Diesel. Security is a big issue, they have firewalls which make things more complex, they are going to implement a PKI and are using VPNs. They use NetMeeting and version 3 (the newest version) has much improved security. With NetMeeting 2 one can do sharing with Sun forum. They provide a protected yet shared archive of data for researchers. Electronic notebook is a very valuable feature. They have a "conversion central" application to convert between formats (e.g. Word to PDF etc.)
This is a user facility that provides access to magnets for NMR work. Users make proposals which are reviewed. Typically time is scheduled in 1 to 2 week blocks. They have 2 proposal calls per year. Spectrometer time is free (except for proprietary use), only cost is user's time & travel, so the ability to collaborate remotely is very attractive. About 25% of the users login remotely and use the facility. Most important feature is a shared window, they also have a white board, electronic notebook. Minimum needs are a modern computer, camera, echo canceller and a network link. Collaboration is in real-time during the experiment. The electronic notebook is more offline collaboration, a particular use of the notebook is to share images rather than having to FAX them between users. Important to make the user interface easy & intuitive since the user (e.g. a chemist) may not be very computer literate. The EMSL Televiewer is platform independent (c.f. NetMeeting).
They did a live demo. One useful feature was the ability to read an analog readout (in this case a thermometer) via the camera.
The project involves PNNL, LBNL & ORNL. Use for researcher' lab observations, design, instrument log boook, experiment log book, legal record, notepad, group work space. Sources of information include raw data, meta data, summaries etc. Basically a securable shared WWW based space with interactive input of results, supports rich media types, provides querying & searching & automation of input from instruments & calculations and access to databases. They provide digital signatures, witnessing, timestamps, import/export and fine grained authorization. Legal defensibility is an issue, preservation & integrity of electronic records.
They have a common architecture that allows one to plug in new viewers or input formats/interfaces, editors, import/export etc. They have defined a set of notebook objects. It is in use at several sites, he gave 5 separate collaborations, 4 of which were not to do with EMSL but rather use by students at universities.
Applications motivate the engineering and the engineering enables the applications. What makes it hard are the need for high bandwidth over wide areas and the applications are intriniscly bursty, and at the same time there is a need for multicast & QoS and a need for measurements. The applications challenges include raising the awareness among all faculty, providing a "service-rich" network environment, supporting application developers. 1999 is a key year for multicast, in the past it meant MBONE, now PIM-sparse mode MBGP, MSDP etc. that enable scalable use of multicast flows.
Abilene is a network to support the Internet2 collaboration of about 150+ universities and about 35 GPoPs and a few interconnect backbones. The backbones are OC48 and OC12.
Interconnect issues are being addressed by NGIX effort with the NGI/Internet2 JET with connection points at NASA-Ames, UMD College Park (as soon as practical) & Chicago. International StarTap plus emphasize StarTap as the universal solution, optimize where appropriate (Canada as an important special case).
The engineering issues are: QoS (low latency, very low packet loss), multicast (PIM-sparseMode, MBGP, MSDP), measurements (Surveyor one way delay & loss, traffic utilization, end to end flows with GPoP help, OC3MON passive measurements). They are beginning multicast beta service, they did a demonstration with NASA-Ames (CalREN2, NREN, Abilene, NGIX-Ames) with 50-80Mbps multicast flows for a medical application. There are only a few campuses that can do this at the moment.
Traffic utilization is easy, boring but important to do well, the periods formats are important. Active measurements give very accurate one-way delay & loss, passive is cheaper in terms of network traffic.
There is a growing self consciousness to support the aggressive applications (not just network performance, and minimize trouble tickets, but also make the applications work well and the scientists get their work done). The bottom line is supporting inter-university networking, supporting advanced applications where users know apps but not the infrastructure, setting expectations. Supporting univ/gigapop/backbone/NGIX infrastructure with multiple support organizations
NGI testbed proposal submitted for University network technology testbeds and deployment of advanced differential services technology across autonomous networks. Basic idea is project support for QBone engineering and planning with a measurement architecture and a bandwidth broker. There will be workshops and targeted support for measurement equipment.
Big need is to understand the applications needs and what they will do to the network. Second issue is how to make any solution scale, and the third is the interoperability issue (different vendors, different network layers etc.)
Premium service guarantees no loss as long as the application stays at or below a threshold. Do this by preferring such traffic (e.g. Per Hop Behavior expedited forwarding, e.g. put at front of queues). Needs strict policing (drop out-of-profile packets) on input and needs to be done across clouds.
We care since threat is increasing amateur hacker => foreign intelligence professional. On the classified side there are national security implications. There is also value of our intellectual capital. Also need to recover congressional trust. Want to achieve world class cyber security. Nine points: immediate security refresher training; long term integrated computer security training; ... then 6 enhancements. Establish program to continually monitor sites, agressive training of sys-admins, random audit of individual computer users ...
There is a draft Cyber security action plan (latest version is Sep-99). Need clear concise policies, need trained people committed & accountable, need operations linked and centrally monitored, and technical capabilities (continuous evolution). It is a multi year program.
The plan is to integrate classified and unclassified programs (emphasis on programs not processes etc.). There was a discussion that this could lead to confusion since the classified and unclassified mechanisms, requirements are very different. There will be a requirement for contracts to adhere to cyber security standards. This could mean procurement contracts, software development contracts, site contractor contracts. Gilligan wants a blue ribbon review team to assess baseline security. Establish CIO or equivalent at all sites. Integrate the security review process (reduce the number of independent oversights). They want to ensure they have budget to perform actions. Gilligan want to increase staff at DoE HQ from 10 to 15.
His transparency said "Integrate Near Term Weapons Laboratory and Counterinterlligence efforts" and when asked said this refers to figuring out how to deploy the above efforts to the non-weapons Lab or as he put it "Deploy consistent security implementation Department wide". Sandy Merola reminded us that currently the security for unclassified Labs is the responsibility of the Lab directors, however the metrics (i.e. the recommendations from above) that the Lab director is expected to meet can still make it something the open Labs would have extreme difficulty living with. Sandy also pointed out that the Lab input needs to come from the SLCCC, and not primarily from the ESSC since ESnet per se has almost no security requirements.
Begin installation on Jan 3, 2000. They defined various levels of security from classified, national security/nuclear, business operations/personnel, industrial secrets/research, academic research, public open. As one goes down the list the sites have more control.
DOEnet in response to a request from Gilligan, estimated a $40M estimate to integrate with ESnet. There was a meeting with the leaders of ESSC (Price, Merola, Leighton) with Gilligan. ESCC stated the estimate was incredibly high, and it was accepted that requesting integration was the wrong question. Rather the request should be to state the requirements (to be gathered by Gilligan's people with ESnet &/or DOEnet people only acting as consultants), DOEnet and ESnet will then analyze the requirements and propose technical & financial approach by which either DOEnet or ESnet or a collaborative DOEnet/ESnet approach to satisfy those requirements. There were some action items which were not followed up on.
There was a 1996 act that mandated that each agency have a CIO. This person would look across the agency and provide: capital planning & oversight of projects; to define standards and architecture for the department; oversight (policy & guidance) of cyber security in particular for the unclassified systems (catch up to close gaps in risk versus protection); moving records etc. into electronic media; lead in addressing Y2K issues; operations (45 people) to provide long haul capability for the federal government, and also to provide operations & maintenance for the DoE central office.
G stated that we do not manage desktop and long haul and networks very well acros the department, it is highly fragmented. Individuals are system managers. need common enterprise wide management of infrastructure. There needs to be a vision for networks to manage bandwidth and provide customer support for throughput reliability etc. and we can do this as we coordinate activities. ESnet is the biggest networking activity in DoE, and it has good management and has it foot in advanced technology. G sees that there is a potential for integrating the DOEnet with ESnet. They are piloting DOEnet via VPNs on ESnet at JLab to see how to come up with a single network with potential savings. So he looks at ESnet not just as a network to support Energy Research but also as an operational network for DOE operations and offices.
Larry Price stated that ESSC is excited about this but wants to ensure that the transition is managed correctly. Larry said the ESSC would send a memo to Gilligan making suggestions on how to make a start on merging the networks. Sandy Merola made the point that ESnet has had high success and has been well regarded in reviews, is well liked by the users as represented by the ESSC, and is something for Gilligan to be proud of and wave the flag for.
Following the call there was a discussion on what the memo should say. It was felt that it is important to stress the effectiveness of the current ESnet operations, management and oversight. Sandy will redraft a memo and we will return to the discussion tomorrow.
Today we have distributed resources - people, data, computers, instruments, facilities, we also have increasing network capabilities. Tomorrow we want a seamless & ubiquitous access to facilities, data & colleagues via shared collaborative information and computational environments.
The DOE NGI theme is wide area data intensive & collaborative computing: ensure the underlying middleware & networking technologies are developed; intetgrate & test the technologies on DOE mission applications; ensure the tools developed can be used by researchers at universities. The concept is an integrated grid architecture serves to unify applications, middleware and networking. The lowest layer is the grid fabric and includes archives, networks, computers, display devices, associated local services, the next layer up is the grid services includinbg protocols, authentication, policy, resources management, instrumentation, discovery etc., above this are the application toolkits for remote data access, remote computing, remote vizualization, asynchronous collaboratory tools. Then at the top layer are the applications such as the combustion corridor, weather control, the PPDG etc. There are also some testbeds including QUALIT and EMERGE (ESnet Metropolitan Research & Education Network (MREN) regional Grid Experimental NGI Testbed).
Application proposals funded were: Earth system grid, combustion, PPDG, X-Ray crystallography, Corridor One. Then there were the 2 pilots QUALIT & EMERGE. Then there were 12 proposals for research in basic applications. 2 were in architecture, 2 in hardware (build interface cards to improve performance of NICs using FPGAs), 2 on integration & analysis. 2 on measurements (NetLogger and another). 2 on middleware (policy based resource management (ANL, UWisc, USC), technologies & tools for high performance distributed computing - MPI2). 2 on visualization (Deb Argawal's tools, Ohio State toolkit).
Need testbed to support advanced applications to evaluate new network technologies such as diff-serv, VPNs, and to coordinate efforts between sites.
NGI networking will have to support 5 applications and 3 NSPs which is a lot of diversity with requirements for bandwidth, jitter, latency, multicast, loss. Need the creation of intelligent networks & network aware applications. Will need a bandwidth broker, better instrumentation at the application level as well as elsewhere, deployment of diff-serv mechanisms to support interactive visualization to allow bulk data to co-exist. Need distributed caching systems and access to large data sets and the globus system.
Looked at allocating bandwidth out of production ESnet service (~30% of access link; 45-90Mbps/site). Testbed will carry best effort + premium traffic; needs network access control/allocation/scheduling; provide instrumentation assistance for applications to drive toward adaptive applications. No new $ for added bandwidth. ESnet & Abilene routers will police at ingress and set the appropriate PHB. Sites are responsible for routing, site routers will mark premium service flows, sites will ensure premium service to end nodes. The goal is to ease the manual configuration burden and to provide a persistent testbed infrastructure.
Current testbed sites (i.e. ones with the equipment already in place) are: LBNL, ANL, SNL, & SLAC, in the next 6-12 months other ESnet sites and Abilene sites will be added. We can get started with the initial 4 sites. The transition phase to ESnet 3 will also make things more complex.
ANL, LBNL, SNL-CA & UWisc. The term corridor came from ASCI, it is a broad corridor as compared to a narrow straw. Combustion is source of 85% of energy consumes in US & source of 98% of CO2 emitted in US so it is an important thing for DoE to address. Lots of data 100MBytes/second, data is complex (irregular not a simple lattice, but is hierarchical). One of the problems is the data for the pictures are too large to save in the desktop, yet unless one has all the data there then one cannot take advantage of all the visualization rendering tools. They are looking at new ways to handle this. With 4 updates/second 32 bits/pixel, 8 textures, 1000x1000 pixels gives 1 Gbps sustained bandwidth so they can use all the bandwidth they can get. The bandwidth is intermittent, e.g. while user is in the 6 wall CAVE the bandwidth is needed but not otherwise. The high bandwidth data distribution achieved between LBNL & ANL is 35Mbytes/sec.
This is for climate prediction, impact assessment and analysis (e.g. for policy evaluation). Data rates 3PB/yr, 100MB/sec (@5 TFops). The user community is large o(100+) sites, very multidisciplinary and had international connections. Used term "virtual proximity" for their requirements to enable a geographically distributed collaboration. Also need caching, security etc. They are using PCMDI, Globus, DPSS, STACs and ...
See http://www.slac.stanford.edu/grp/scs/net/talk/ngiperf/index.htm
NGI security requires identification, authentication, authorization, and accounting. This is complicated by the existence of multiple management domains with different security policies and users sometimes working across domains. A single user sign-on is desired without passwords transmitted over the network in clear text, or at all. A number of different approaches are documented in this presentation.
The project started in December 1997 between CERN, DESY & FNAL with VocalTec card in PCs. It worked but was rather poor quality. SLAC joined in September 1998. The project moved to using Cisco 3640 routers in 1998. The performance is good (toll quality), but interfacing to the phone PBX is non-trivial. It took a long time to get the router PBX interfaces working at all the sites. CERN, FNAL & DESY are using the ISDN/BRI interface, SLAC & LBL use the E&M interface. SLAC is looking at getting a PRI-ISDN. One can pass thru to make local calls at most Labs.
QoS performance measured by ping indicates that jitter, RTT & loss are within the ITU limits. However after route changes in the Internet there can be long convergence times during which there can be many seconds of loss of connectivity (periods of sequential packet loss) between end nodes. This will need further investigation to get an idea of the frequency and length of such reconvergence outages. There is a testbed between SLAC & LBL to further investigate the impact of load on VoIP and also of using DiffServ.
DoD (Pete Dean <pwdean@sandia.gov>) is very interested since it may enable them to replace STU secure phones (phones with type 1 encryption for classified calls) by using a secure network. Many features (e.g. caller ID) of today's PBXs are not available in the pilot and the reliability of the Internet has got a long way to get to the phone companies (99.999%). Main uses have been for people traveling and needing to get back to their home Lab or even to their home. We need a dial-plan and a gatekeeper.
DOEnet's real reason for existence is to link together DOE field office to do business operations. Many of the field offices are extremely small (few tens of people) with very little network/security expertise at each site so there can be a big security exposure.
VPNs are growing in importance/interest. There are LAN-LAN & LAN-client instantiations. There are systems from Red Creek, Compatible systems, Nortel, Cisco. Interoperability is poor at the moment but IPSec is coming. This is a potential solution to allowing DOE site offices to exist inside Lab LANs. JLab is piloting such a solution with Red Creek equipment across ESnet and the traffic pops out at ORNL DoE operations office. ORNL has about 2000 VPN users, the clients are free but the server was expensive (~$80K). If successful this can be extended to provide direct connectivity to DoE head office in Washington or other sites, with the option of not using ESnet. A concern was raised about the ER sites having to take on the administration of the VPN infrastructure for DOEnet (e.g. configuring VPN client software, consulting etc.).
Wireless 802.11 standard has been ratified which is leading to high demand and thus low price. The PCMCIA cards are ~$100 each and the hubs are $250. It is very secure, and un-sniffable. There will be a demo/test at the fall ESCC meeting at JLab. The bad news is that the IEEE standard does not specify handshaking so interoperability is left to the vendors and cards are new enough that interoperability claims are suspect. The dream is full portability of laptops across the DOE complex. Go to a meeting expect to be on the net (security via VPN).
VoIP is just the start. Long term will move to integrated networks that will enable "soft phones". Any phone = an operator switchboard, any phone can have any number, any phone could read email, send a FAX, be a pager or act as a security system, an IPv6 cell phone could interact with ANY IPv6 device - they will be ubiquitous. We need to watch, prototype and stay ahead, to understand, to set expectations etc.
Gig-E and layer 3 switches provide the first obvious backbone replacement since FDDI. There are lots of vendors such as Foundry, Packet Engines, good news they all support basic IP. Bad news is figuring out what other features you want.
SC99 in Portland will have the "hottest net in the world" with OC192 WAN, 16xOC192 LAN (DWDM).
The next ESCC is at JLab in October.
Huge effort going into securing sensitive but unclassified information. So doing secure voice over their protected IP net is very interesting. There is also a goal to make the classified network more functional and more secure.
We need to nominate people to attend the SSI meeting. The following people were proposed: Larry Price (chair of ESSC), Bill Wing (chair of ESCC), Jim Leighton (ESnet), Ray Bair (has been involved in SSI), and Sandy Merola (inter agency issues).
The draft memo to Gilligan was discussed It recognized the desire to have a single DOE network, it also identifies the success of ESnet evolving out of 3 incompatible networks (MFEnet, HEPnet and ARPA/Milnet) supporting various DoE programs. It identifies the need to ensure ESnet is not compromised technically, fiscally or administratively. ESnet/ESSC is prepared to move forward with an investigation of the CIO and other DoE business/administrative requirements that may be meetable via a separate DOEnet, ESnet or some jointly planned and managed interconnectivity between DOEnet and ESnet. This latter might include only the provision of bandwidth and networking management services by ESnet (DOEnet provides services (e.g. VPN)). The network related requirements should be gathered not by the network providers but by the DoE clients (CIO & other business/administrative functions) represented by an internal working group (DoE staff who are both knowledgeable of their business requirements and respected in their community to ensure needed credibility) to gather the requirements. DOEnet & ESnet staff would be to analyze the requirements, propose a technical and financial approach which either DOEnet and ESnet or a collaborative DOEnet/ESnet approach could satisfy those requirements. There was a long discussion of how much overlap there is between the two networks, how well VPNs would work for say 9 of the 40 DOEnet sites, if ESnet provides services to DOEnet then what are these services and how are they funded? It was agreed to take this offline with a smaller group who will work with Gilligan to come up with an acceptable proposal/solution. Out of this will come a charter. Sandy will compose a piece of email to Gilligan proposing a private meeting between Gilligan, Merola & Price to share information, understanding and set expectations.
There was a discussion on the testbed. Can we operate a testbed without impacting the production traffic, how much bandwidth should be reserved (strawman 30%), and issues of the persitence of the network (how long does it stay around), who manages it, and how will the grid services be integrated. It was decided that the upper limit would be set at 30% of the "scarcest link" in the path. Esnet will use their usual tools to monitor utilization and performance, and there will be regulsar reports on the testbed and its impact on the production network.
[ Feedback ]