"The first deployment of workload management services on the EU DataGrid Testbed: feedback on design and implementation."

G. Avellino, S. Beco, B. Cantalupo, F. Pacini, A. Terracina, A. Maraschini (DATAMAT Ingegneria dei Sistemi S.p.A.) D. Colling (Imperial College, London) S. Monforte, M. Pappalardo (INFN, Sezione di Catania) L. Salconi (INFN, Sezione di Pisa) F. Giacomini, E. Ronchieri (INFN, CNAF) D. Kouril, A. Krenek, L. Matyska, M. Mulac, J. Pospisil, M. Ruda, Z. Salvet, J. Sitera, M. Vocu (CESNET) M. Mezzadri, F. Prelz (INFN, Sezione di Milano) A. Gianelle, R. Peluso, M. Sgaravatto (INFN, Sezione di Padova) S. Barale, A. Guarise, A. Werbrouck (Università di Torino and INFN, Sezione di Torino) For comments: Francesco.Prelz@mi.infn.it This talk is available at: http://www.mi.infn.it/~prelz/chep03

-

Just a quick reminder from CHEP 2001:

• DataGrid goal: Grid software projects meet real-life scientific applications (and their deadlines), with mutual benefit.
• DataGrid goal: bring the issues of data identification, location, transfer and access into the picture (but this is WP2).
• WP1 goal: define and implement a suitable architecture for distributed scheduling and resource management on a GRID environment

The following functionality has been delivered by WP1 with a continuous effort on the EU DataGrid testbed over one and a half years:

• Ability to submit a job (described via an appropriate Job Description Language, or JDL) to the DataGrid testbed from any user machine.
• Ability to monitor and control (terminate) the job, and to transfer a "small" amount of data directly to and from the client machine and the executing machine.
• WP1's RB chooses an appropriate computing resource for the job, based on the constraints specified in the JDL.
  • Jobs are sent 'close' to where the data required in the JDL is, based on Replica Catalog and Information System information.

The WP1 solution integrates the following technology:

• The JDL is based on Condor ClassAds, and parsed with the Condor ClassAd library
• Information on the state of the testbed is retrieved through Globus MDS services.
• Globus GRAM is used to communicate with the CE's, and the Condor-G layer is included to provide increased reliability.
• GridFtp is used for the transfer of the job "sandboxes".

The release 1 prototype was confronted with stress tests and semi-production exercises, and, as expected, many problems were discovered and identified.
Experience with this first release was assimilated, as planned, in the components for the second major release of EU DataGrid (now slated for integration and delivery in April 2003).

• Another talk (presented by Massimo Sgaravatto) will focus on the WP1 components for Release 2.
• This talk focuses on the issues that were found (following from design oversights and/or implementation mistakes), and tries to derive few principles that summarize the lessons that WP1 learned so far.
• Although many lessons were also learned on organizational and sociological matters, this talk will only cover what could be understood and done at the technical level to the job management pipeline.

• Service scalability problems, with increasing user and job load.
  • The distribution of Resource Broker and Logging and Book-keeping services was left undetermined, but they were not tied together hierarchically.
  • Typical integrated installation, and typical usage patterns, could not be easily predicted.
  • We imagined we would simply hit the resource roof on one node.
  • This could in principle be addressed by determining the appropriate organisation and distribution of Resource Broker and Logging and Book-keeping services...

• Figuring out and controlling the interplay of the various integrated components, either directly or through the contention for system resources.
• Addressing stability issues and bottlenecks. This is complicated by two fundamental non-linear effects, that make standard queuing theory hard to apply:
  • "Idle" job queues can introduce feedback on system load, and this can sometimes become significant.
  • Hard timeouts are hidden in many services.
• It is well known that working on a bottleneck on the job pipeline will expose the next. The following applet is used to demonstrate how these effects impact on the troubleshooting of the integrated system:

• From the exercise above:
  • Queues of various kinds of requests for processing should be allowed to form where they have a minimal and understood impact on system resources.
  • Limits should always be placed on dynamically allocated objects, threads and/or subprocesses.
  • Special care needs to be taken around the pipeline areas where serial handling of requests is needed.
• We don't claim we couldn't have learned these from some book, or by heeding to good advice, but...
• An expert is someone who made most of the possible mistakes in any given field.
• There are more lessons that were learned in this spirit.

• By trying to communicate job information and state among the various services and back to users:
  • Communication among services should always be reliable
    • Always applying double-commit and rollback for network communications.
    • Going through the filesystem for local communications.
  • Every process, object or entity related to the job lifecycle should have another process, object or entity in charge of its well-being.
  • Information repositories should be minimized (with a clear identification of authoritative info).
• By integrating software that's under development (and by dealing with the latest C++ machinery):
  • Monolithic, long-lived processes should be avoided
    • They are more subject to memory and other resource leaks.
  • More thought should be devoted to efficiently and correctly recovering a service rather than to starting and running it.

• WP1 has been distributing jobs over the EDG testbed in a continuous fashion for one and a half years now.
• Existing grid technology was integrated wherever possible.
• Understanding direct and indirect component interplay was not a simple scalability evaluation.
• Understanding and removing bottlenecks is significantly complicated by non-linear and non-continuous effects in the system.
• Few principles that apply to distributed systems operations were learned the hard way.
• WP1 tried to incorporate these in its second major software release.
• The proof will necessarily have to be in the pudding.