Collaboration for improved operations: how OPOS supports the experiments

The current OPOS team, from left: Andres Felipe Alba Hernandez from Colombia, Qiulan Huang from China, Paola Buitrago from Colombia, Vito Di Benedetto from Italy and Tyler Propp from the US. Photo courtesy of the OPOS team

The current OPOS team, from left: Andres Felipe Alba Hernandez from Colombia, Qiulan Huang from China, Paola Buitrago from Colombia, Vito Di Benedetto from Italy and Tyler Propp from the US. Photo courtesy of the OPOS team

You might not have heard of the Scientific Computing Division’s Offline Production Operations Service (OPOS) team, but their behind-the-scenes work is essential for many Fermilab experiments. The team increases the resources of the experiments by efficiently processing jobs, providing the scientists with quality data and increasing the time the scientists have to analyze the data.

The current OPOS team, which consists of one application physicist and three visiting scientists, was formed in January 2015 under the direction of Anna Mazzacane with the intention of supporting the Intensity Frontier experiments by helping transform the raw data collected by detectors into a format that scientists can interpret to give it a physical meaning. Now, OPOS is working with DUNE, MINERvA, MINOS+ and NOvA. MicroBooNE, which is taking data and has high-priority need for data processing, is one of the next experiments OPOS will take on board.

The team’s main responsibility is processing jobs to reduce the experiments’ workload, but as Mazzacane explained, that’s far from all OPOS does.

“Many people think this work is just push the button and you’re done,” Mazzacane said, “but it’s much more than processing jobs on the grid.”

The success of their work depends on forging relationships both within the team and with experimenters through improved cooperation and communication. The team, having studied at different universities in different countries, including the Institute of High Energy Physics in China, the Universidad de los Andes in Colombia, the Università degli Studi di Napoli Federico II in Italy and the Università del Salento, also in Italy, have practiced cooperation and communication daily to coordinate their skill sets and areas of expertise. This internal effort mirrors the work the team does with the experiments. Mazzacane credits OPOS with creating a bridge between the experiments and Computing in order to use Computing tools to support the experiments, sharing their expertise to increase the efficiency of the production and increasing the manpower of the experiments to achieve the goals of the collaboration on the production side.

Team members leverage their knowledge to direct the experimenters to the correct Computing staff member, department or tool to more efficiently solve problems and overcome obstacles. In addition, by working with multiple experiments simultaneously, they can look for new ways of utilizing common tools and workflows for increased productivity. And by coupling their knowledge of available resources with their job processing expertise, they can collaborate with newer experiments to help come up with efficient job-processing workflows tailored to the experiment’s needs.

For job processing– no small feat considering the number of experiments they support–team members use a tool created and managed by SCD, the Production Operations Management System, to handle the influx of jobs. OPOS is responsible for scheduling, running and monitoring the large-scale production tasks while providing status updates to the experiment groups. Team members also produce simulated Monte Carlo event samples for the experiments.

More information about the OPOS team can be found at their website.