arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

D. Colling

63 papers · 2003–2023

2003: 62004: 42005: 32006: 42007: 32008: 22009: 22010: 62011: 22012: 32013: 52014: 52015: 62016: 02017: 22018: 22019: 42020: 12021: 12022: 12023: 1
2003peak 6 in 20032023

Frequent co-authors

Papers

showing 1–25 of 63 RSS

GridPP: the UK grid for particle physics

D. Britton, A. Cass, P. E. L. Clarke +9 more

Computer Networks and Communications · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · · cited 80× · DOI ↗ · more like this ·

The start-up of the Large Hadron Collider (LHC) at CERN, Geneva, presents a huge challenge in processing and analysing the vast amounts of scientific data that will be produced. The architecture of the worldwide grid that will handle 15 PB of particle physics data annually from this machine is based on a hierarchical tiered structure. We describe the development of the UK component (GridPP) of this grid from a prototype system to a full exploitation grid for real data analysis. This includes the physical infrastructure, the deployment of middleware, operational experience and the initial exploitation by the major LHC experiments.

The EU DataGrid Workload Management System: towards the second major release

G. Avellino, S. Barale, S. Beco +26 more

cs.DC · · cited 25× · arXiv ↗ · more like this ·

In the first phase of the European DataGrid project, the 'workload management' package (WP1) implemented a working prototype, providing users with an environment allowing to define and submit jobs to the Grid, and able to find and use the ``best'' resources for these jobs. Application users have now been experiencing for about a year now with this first release of the workload management system. The experiences acquired, the feedback received by the user and the need to plug new components implementing new functionalities, triggered an update of the existing architecture. A description of this revised and complemented workload management system is given.

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS

Justas Balcas, S. Belforte, Brian Bockelman +9 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 20× · DOI ↗ · more like this ·

CMS will require access to more than 125k processor cores for the beginning of Run 2 in 2015 to carry out its ambitious physics program with more and higher complexity events. During Run1 these resources were predominantly provided by a mix of grid sites and local batch resources. During the long shut down cloud infrastructures, diverse opportunistic resources and HPC supercomputing centers were made available to CMS, which further complicated the operations of the submission infrastructure. In this presentation we will discuss the CMS effort to adopt and deploy the glideinWMS system as a common resource provisioning layer to grid, cloud, local batch, and opportunistic resources and sites. We will address the challenges associated with integrating the various types of resources, the efficiency gains and simplifications associated with using a common resource provisioning layer, and…

Enabling QoS for Service-Oriented Workflow on GRID

Li Guo, A. Stephen McGough, Asif Akram +3 more

Computer Networks and Communications · · cited 15× · DOI ↗ · more like this ·

As the main computing paradigm for resource- intensive scientific applications, Grid[1] enables resource sharing and dynamic allocation of computational resources. It promotes access to distributed data, operational flexibility and collaboration, and allows service providers to be distributed both conceptually and physically to meeting different requirements. Large- scale grids are normally composed of huge numbers of components from different sites, which increases the requirements of workflows and quality of service (QoS) upon these workflows as many of these components have real-time requirements. In this paper, we describe our web services based QoS-aware workflow management system(WfMS) from GridCC project[7] and show how this WfMS can aid to ensure workflows meet the pre-defined QoS requirements and optimise them accordingly.

GRIDCC

A. Stephen McGough, Asif Akram, Li Guo +7 more

Computer Networks and Communications · · cited 14× · DOI ↗ · more like this ·

The Grid is a concept which allows the sharing of resources between distributed communities, allowing each to progress towards potentially different goals. As adoption of the Grid increases so are the activities that people wish to conduct through it. The GRIDCC project is a European Union funded project addressing the issues of integrating instruments into the Grid. This increases the requirement of workflows and Quality of Service upon these workflows as many of these instruments have real-time requirements. In thispaper we present the workflow management service within the GRIDCC project which is tasked with optimising the workflows and ensuring that they meet the pre-defined QoS requirements specified upon them.

The first deployment of workload management services on the EU DataGrid Testbed: feedback on design and implementation

G. Avellino, S. Beco, B. Cantalupo +26 more

cs.DC · · cited 13× · arXiv ↗ · more like this ·

Application users have now been experiencing for about a year with the standardized resource brokering services provided by the 'workload management' package of the EU DataGrid project (WP1). Understanding, shaping and pushing the limits of the system has provided valuable feedback on both its design and implementation. A digest of the lessons, and "better practices", that were learned, and that were applied towards the second major release of the software, is given.

Real Time Monitor of Grid job executions

D. Colling, J. Martyniak, A. Stephen McGough +4 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 9× · DOI ↗ · more like this ·

In this paper we describe the architecture and operation of the Real Time Monitor (RTM), developed by the Grid team in the HEP group at Imperial College London. This is arguably the most popular dissemination tool within the EGEE [1] Grid. Having been used, on many occasions including GridFest and LHC inauguration events held at CERN in October 2008. The RTM gathers information from EGEE sites hosting Logging and Bookkeeping (LB) services. Information is cached locally at a dedicated server at Imperial College London and made available for clients to use in near real time. The system consists of three main components: the RTM server, enquirer and an apache Web Server which is queried by clients. The RTM server queries the LB servers at fixed time intervals, collecting job related information and storing this in a local database. Job related data includes not only job state (i.e.…

CMS analysis operations

Julia Andreeva, M. Calloni, D. Colling +9 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 8× · DOI ↗ · more like this ·

During normal data taking CMS expects to support potentially as many as\n2000 analysis users. Since the beginning of 2008 there have been more\nthan 800 individuals who submitted a remote analysis job to the CMS\ncomputing infrastructure. The bulk of these users will be supported at\nthe over 40 CMS Tier-2 centres. Supporting a globally distributed\ncommunity of users on a globally distributed set of computing clusters\nis a task that requires reconsidering the normal methods of user support\nfor Analysis Operations. In 2008 CMS formed an Analysis Support Task\nForce in preparation for large-scale physics analysis activities. The\ncharge of the task force was to evaluate the available support tools,\nthe user support techniques, and the direct feedback of users with the\ngoal of improving the success rate and user experience when utilizing\nthe distributed computing environment. The…

Using the grid for the BaBar experiment

C. Bozzi, T. Adye, D. Andreotti +9 more

Computer Networks and Communications · IEEE Transactions on Nuclear Science · · cited 8× · DOI ↗ · more like this ·

The BaBar experiment has been taking data since 1999. In 2001 the computing group started to evaluate the possibility to evolve toward a distributed computing model in a grid environment. We built a prototype system, based on the European Data Grid (EDG), to submit full-scale analysis and Monte Carlo simulation jobs. Computing elements, storage elements, and worker nodes have been installed at SLAC and at various European sites. A BaBar virtual organization (VO) and a test replica catalog (RC) are maintained in Manchester, U.K., and the experiment is using three EDG testbed resource brokers in the U.K. and in Italy. First analysis tests were performed under the assumption that a standard BaBar software release was available at the grid target sites, using RC to register information about the executable and the produced n-tuples. Hundreds of analysis jobs accessing either Objectivity or…

The Diverse use of Clouds by CMS

Anastasios Andronis, D. Bauer, Olivier Chaze +9 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 7× · DOI ↗ · more like this ·

The resources CMS is using are increasingly being offered as clouds. In Run 2 of the LHC the majority of CMS CERN resources, both in Meyrin and at the Wigner Computing Centre, will be presented as cloud resources on which CMS will have to build its own infrastructure. This infrastructure will need to run all of the CMS workflows including: Tier 0, production and user analysis. In addition, the CMS High Level Trigger will provide a compute resource comparable in scale to the total offered by the CMS Tier 1 sites, when it is not running as part of the trigger system. During these periods a cloud infrastructure will be overlaid on this resource, making it accessible for general CMS use. Finally, CMS is starting to utilise cloud resources being offered by individual institutes and is gaining experience to facilitate the use of opportunistically available cloud resources.We present a snap…

GRIDCC - Providing a real-time grid for distributed instrumentation

PR Hobson, AS McGough, D. Colling

Computer Networks and Communications · Brunel University Research Archive (BURA) (Brunel University London) · · cited 7× · OpenAlex ↗ · more like this ·

The GRIDCC project is extending the use of Grid computing to include access to and control of distributed instrumentation.
\nAccess to the instruments will be via an interface to a Virtual Instrument Grid Service (VIGS). VIGS is a new concept and its design and implementation, together
\nwith middleware that can provide the appropriate Quality of Service (QoS), is a key part of the GRIDCC development plan. An overall architecture for GRIDCC has been
\ndefined and some of the application areas, which include distributed power systems, remote control of an accelerator and the remote monitoring of a large particle physics
\nexperiment, are briefly discussed.

Scalability tests of R-GMA-based grid job monitoring system for CMS Monte Carlo data production

D. Bonacorsi, D. Colling, Laurence Field +8 more

Computer Networks and Communications · IEEE Transactions on Nuclear Science · · cited 7× · DOI ↗ · more like this ·

High-energy physics experiments, such as the compact muon solenoid (CMS) at the large hadron collider (LHC), have large-scale data processing computing requirements. The grid has been chosen as the solution. One important challenge when using the grid for large-scale data processing is the ability to monitor the large numbers of jobs that are being executed simultaneously at multiple remote sites. The relational grid monitoring architecture (R-GMA) is a monitoring and information management service for distributed resources based on the GMA of the Global Grid Forum. We report on the first measurements of R-GMA as part of a monitoring architecture to be used for batch submission of multiple Monte Carlo simulation jobs running on a CMS-specific LHC computing grid test bed. Monitoring information was transferred in real time from remote execution nodes back to the submitting host and…

Running CMS software on GRID Testbeds

D. Bonacorsi, P. Capiluppi, A. Fanfani +27 more

physics.comp-ph · · cited 6× · arXiv ↗ · more like this ·

Starting in the middle of November 2002, the CMS experiment undertook an evaluation of the European DataGrid Project (EDG) middleware using its event simulation programs. A joint CMS-EDG task force performed a "stress test" by submitting a large number of jobs to many distributed sites. The EDG testbed was complemented with additional CMS-dedicated resources. A total of ~ 10000 jobs consisting of two different computational types were submitted from four different locations in Europe over a period of about one month. Nine sites were active, providing integrated resources of more than 500 CPUs and about 5 TB of disk space (with the additional use of two Mass Storage Systems). Descriptions of the adopted procedures, the problems encountered and the corresponding solutions are reported. Results and evaluations of the test, both from the CMS and the EDG perspectives, are described.

Using the CMS High Level Trigger as a Cloud Resource

D. Colling, Adam Huffman, Alison McCrae +9 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 5× · DOI ↗ · more like this ·

The CMS High Level Trigger is a compute farm of more than 10,000 cores. During data taking this resource is heavily used and is an integral part of the experiment's triggering system. However, outside of data taking periods this resource is largely unused. We describe why CMS wants to use the HLT as a cloud resource (outside of data taking periods) and how this has been achieved. In doing this we have turned a single-use cluster into an agile resource for CMS production computing. While we are able to use the HLT as a production cloud resource, there is still considerable further work that CMS needs to carry out before this resource can be used with the desired agility. This report, therefore, represents a snapshot of this activity at the time of CHEP 2013.

GRIDCC: A Real‐Time Grid Workflow System with QoS

A. Stephen McGough, Asif Akram, Li Guo +9 more

Computer Networks and Communications · Scientific Programming · · cited 5× · DOI ↗ · more like this ·

The over‐arching aim of Grid computing is to move computational resources from individual institutions where they can only be used for in‐house work, to a more open vision of vast online ubiquitous `virtual computational′ resources which support individuals and collaborative projects. A major step towards realizing this vision is the provision of instrumentation – such as telescopes, accelerators or electrical power stations – as Grid resources, and the tools to manage these resources online. The GRIDCC project attempts to satisfy these requirements by providing the following four co‐dependent components; a flexible wrapper for publishing instruments as Grid resources; workflow support for the orchestration of multiple Grid resources in a timely manner; the machinery to make reservation agreements on Grid resources; and the facility to satisfy quality of service (QoS) requirements on…

The GRIDCC Project the GRIDCC Collaboration

A. Stephen McGough, D. Colling

Computer Networks and Communications · · cited 5× · DOI ↗ · more like this ·

The GRIDCC project is integrating into the grid remote interaction with instruments, along with distributed control and real time interaction. The GRIDCC middleware is being designed with use cases from a very diverse set of applications and so the GRIDCC architecture provides access to the instruments in as generic a way as possible. The middleware will be validated on a representative subset of these applications. GRIDCC is also developing an adaptable user interface and a mechanism for performing complex workflows in order to increase both the usability and the usefulness of the system. Wherever possible the GRIDCC middleware builds on top of other middleware stacks allowing the effort to be concentrated on the more novel elements of the project. The GRIDCC project is a collaboration between 10 organizations in 4 different countries and is funded by the European Union

HEP Applications Evaluation of the EDG Testbed and Middleware

I. Augustin, F. Carminati, J. Closier +24 more

cs.DC · · cited 5× · arXiv ↗ · more like this ·

Workpackage 8 of the European Datagrid project was formed in January 2001 with representatives from the four LHC experiments, and with experiment independent people from five of the six main EDG partners. In September 2002 WP8 was strengthened by the addition of effort from BaBar and D0. The original mandate of WP8 was, following the definition of short- and long-term requirements, to port experiment software to the EDG middleware and testbed environment. A major additional activity has been testing the basic functionality and performance of this environment. This paper reviews experiences and evaluations in the areas of job submission, data management, mass storage handling, information systems and monitoring. It also comments on the problems of remote debugging, the portability of code, and scaling problems with increasing numbers of jobs, sites and nodes. Reference is made to the…

Possibilities for Named Data Networking in HEP

Duncan Rand, Simon Fayer, D. Colling

Computer Networks and Communications · Journal of Physics Conference Series · · cited 4× · DOI ↗ · more like this ·

Named Data Networking is a novel networking architecture which places emphasis on the naming and signing of data. Once so named, the location of data sources becomes less important and the emphasis moves from host to host transfers to pulling data from the network. Furthermore data are cached en route to their destination. We believe this approach has interesting possibilities for High Energy Physics (HEP) and report on work we have done in this area including the development of a scalable repository, a ROOT plugin and a small local testbed, the submission of jobs to a grid cluster and some ideas on an authentication system for LHC VOs.

The GridPP DIRAC project - DIRAC for non-LHC communities

D. Bauer, D. Colling, Robert Currie +5 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 4× · DOI ↗ · more like this ·

The GridPP consortium in the UK is currently testing a multi-VO DIRAC service aimed at non-LHC VOs. These VOs (Virtual Organisations) are typically small and generally do not have a dedicated computing support post. The majority of these represent particle physics experiments (e.g. NA62 and COMET), although the scope of the DIRAC service is not limited to this field. A few VOs have designed bespoke tools around the EMI-WMS & LFC, while others have so far eschewed distributed resources as they perceive the overhead for accessing them to be too high. The aim of the GridPP DIRAC project is to provide an easily adaptable toolkit for such VOs in order to lower the threshold for access to distributed resources such as Grid and cloud computing. As well as hosting a centrally run DIRAC service, we will also publish our changes and additions to the upstream DIRAC codebase under an…

CMS computing model evolution

C. Grandi, D. Bonacorsi, D. Colling +2 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 4× · DOI ↗ · more like this ·

The CMS Computing Model was developed and documented in 2004. Since then the model has evolved to be more flexible and to take advantage of new techniques, but many of the original concepts remain and are in active use. In this presentation we will discuss the changes planned for the restart of the LHC program in 2015. We will discuss the changes planning in the use and definition of the computing tiers, that were defined with the MONARC project. We will present how we intend to use new services and infrastructure to provide more efficient and transparent access to the data. We will discuss the computing plans to make better use of the computing capacity by scheduling more of the processor nodes, making better use of the disk storage, and more intelligent use of the networking.

Performance of R-GMA Based Grid Job Monitoring System for CMS Data Production

Rob Byrom, D. Colling, S.M. Fisher +7 more

Computer Networks and Communications · IEEE Symposium Conference Record Nuclear Science 2004. · · cited 3× · DOI ↗ · more like this ·

High Energy Physics experiments, such as the Compact Muon Solenoid (CMS) at the CERN laboratory in Geneva, have large-scale data processing requirements, with stored data accumulating at a rate of 1 Gbyte/s. This load comfortably exceeds any previous processing requirements and we believe it may be most efficiently satisfied through Grid computing. Management of large Monte Carlo productions (/spl sim/3000 jobs) or data analyses and the quality assurance of the results requires careful monitoring and bookkeeping, and an important requirement when using the Grid is the ability to monitor transparently the large number of jobs that are being executed simultaneously at multiple remote sites. R-GMA is a monitoring and information management service for distributed resources based on the Grid Monitoring Architecture of the Global Grid Forum. We have previously developed a system allowing us…

Efficiency of resource brokering in grids for high-energy physics computing

Paul Andrew Crosby, D. Colling, D. Waters

Computer Networks and Communications · IEEE Transactions on Nuclear Science · · cited 3× · DOI ↗ · more like this ·

This paper presents a study of the efficiency of resource brokering in a computational grid constructed for CPU and data intensive scientific analysis. Real data is extracted from the logging records of an in-use resource broker relating to the running of Monte Carlo simulation jobs, and compared to detailed modeling of job processing in a grid system. This analysis uses performance indicators relating to how efficiently the jobs are run, as well as how effectively the available computational resources are being utilized. In the case of a heavily loaded grid, the delays incurred at different stages of brokering and scheduling are studied, in order to determine where the bottlenecks appear in this process. The performance of different grid setups is tested, for instance, homogeneous and heterogeneous resource distribution, and varying numbers of resource brokers. The importance of the…

Open access to high-level data and analysis tools in the CMS experiment at the LHC

A. Calderón, D. Colling, Adam Huffman +5 more

Computer Networks and Communications · Journal of Physics Conference Series · · cited 2× · DOI ↗ · more like this ·

The CMS experiment, in recognition of its commitment to data preservation and open access as well as to education and outreach, has made its first public release of high-level data under the CC0 waiver: up to half of the proton-proton collision data (by volume) at 7 TeV from 2010 in CMS Analysis Object Data format. CMS has prepared, in collaboration with CERN and the other LHC experiments, an open-data web portal based on Invenio. The portal provides access to CMS public data as well as to analysis tools and documentation for the public. The tools include an event display and histogram application that run in the browser. In addition a virtual machine containing a CMS software environment along with XRootD access to the data is available. Within the virtual machine the public can analyse CMS data; example code is provided. We describe the accompanying tools and documentation and discuss…

Adding Standards Based Job Submission to a Commodity Grid Broker

D. Colling, A. Stephen McGough, Jazz Mack Smith +4 more

Computer Networks and Communications · Oxford University Research Archive (ORA) (University of Oxford) · · cited 2× · DOI ↗ · more like this ·

The Condor matchmaker provides a powerful mechanism for matching together both users job requirements and resource providers requirements in such a manner that not only is a paring selected which satisfies both requirements but is optimal for both. This has made the Condor system a good choice for use as a meta-scheduler within the Grid. Integrating Condor within a wider Grid context is possible but only through modification to the Condor source code. Here we describe how the standards for job submission and resource descriptions can be integrated into the Condor system to allow arbitrary Grid resources which support these standards to be brokered through Condor.

Performance of R-GMA for Monitoring Grid Jobs for CMS Data Production

Rob Byrom, D. Colling, S.M. Fisher +6 more

Computer Networks and Communications · Brunel University Research Archive (BURA) (Brunel University London) · · cited 2× · DOI ↗ · more like this ·

High energy physics experiments, such as the Compact Muon Solenoid (CMS) at the CERN laboratory in Geneva, have large-scale data processing requirements, with data accumulating at a rate of 1 Gbyte/s. This load comfortably exceeds any previous processing requirements and we believe it may be most efficiently satisfied through grid computing. Furthermore the production of large quantities of Monte Carlo simulated data provides an ideal test bed for grid technologies and will drive their development. One important challenge when using the grid for data analysis is the ability to monitor transparently the large number of jobs that are being executed simultaneously at multiple remote sites. R-GMA is a monitoring and information management service for distributed resources based on the grid monitoring architecture of the Global Grid Forum. We have previously developed a system allowing us to…