The BABAR Prompt Reconstruction System, or Getting the Results out Fast: an Evaluation of Nine Months Experience Operating a Near Real-time Bulk Data Production System

The BABAR experiment at SLAC has been operational since May 1999. An ambitious program to completely reconstruct 100% of all physics events within two hours of their acquisition was launched. In addition, this system was intended to provide a nearly continuous “rolling calibration” and extensive detector monitoring for feedback into the working experiment. We succeeded in processing the very first PEP-II collisions recorded by the detector within a few hours. Unfortunately, problems with code reliability, computing and network infrastructure, Objectivity and operational efficiency prevented us from maintaining this short latency. Increasing accelerator luminosity and the cancellation of a much-needed scheduled down period complicated our ability to upgrade the system. After months of struggling with code release policies, hardware upgrades, and extensive Objectivity development, we are now within sight of our primary performance goals. The focus of this paper is to summarize the more important steps required to make this project a success with emphasis on lessons learned. Overall performance, current status of the running system and future plans will also be presented.