Fermilab computing experts bolster NOvA evidence, 1 million cores consumed

How do you arrive at the physical laws of the universe when you’re given experimental data on a renegade particle that interacts so rarely with matter, it can cruise through light-years of lead? You call on the power of advanced computing.

The NOvA neutrino experiment, in collaboration with the Department of Energy’s Scientific Discovery through Advanced Computing (SciDAC-4) program and the HEPCloud program at DOE’s Fermi National Accelerator Laboratory, was able to perform the largest-scale analysis ever to support the recent evidence of antineutrino oscillation, a phenomenon that may hold clues to how our universe evolved.

Using Cori, the newest supercomputer at the National Energy Research Scientific Computing Center (NERSC), located at Lawrence Berkeley National Laboratory, NOvA used over 1 million computing cores, or CPUs, between May 14 and 15 and over a short timeframe one week later. This is the largest number of CPUs ever used concurrently over this duration — about 54 hours — for a single high-energy physics experiment. This unprecedented amount of computing enabled scientists to carry out some of the most complicated techniques used in neutrino physics, allowing them to dig deeper into the seldom seen interactions of neutrinos. This Cori allocation was more than 400 times the amount of Fermilab computing allocated to the NOvA experiment and 50 times the total computing capacity at Fermilab allocated for all of its rare-physics experiments. A continuation of the analysis was performed on NERSC’s Cori and Edison supercomputers one week later. In total, nearly 35 million core-hours were consumed by NOvA in the 54-hour period. Executing the same analysis on a single desktop computer would take 4,000 years.

“The special thing about NERSC is that it enabled NOvA to do the science at a new level of precision, a much finer resolution with greater statistical accuracy within a finite amount of time,” said Andrew Norman, NOvA physicist at Fermilab. “It facilitated doing analysis of real data coming off the detector at a rate 50 times faster than that achieved in the past. The first round of analysis was done within 16 hours. Experimenters were able to see what was coming out of the data, and in less than six hours everyone was looking at it. Without these types of resources, we, as a collaboration, could not have turned around results as quickly and understood what we were seeing.”

The experiment presented the latest finding from the recently collected data at the Neutrino 2018 conference in Germany on June 4.

“The speed with which NERSC allowed our analysis team to run sophisticated and intense calculations needed to produce our final results has been a game-changer,” said Fermilab scientist Peter Shanahan, NOvA co-spokesperson. “It accelerated our time-to-results on the last step in our analysis from weeks to days, and that has already had a huge impact on what we were able to show at Neutrino 2018.”

In addition to the state-of-the-art NERSC facility, NOvA relied on work done within the SciDAC HEP Data Analytics on HPC (high-performance computers) project and the Fermilab HEPCloud facility. Both efforts are led by Fermilab scientific computing staff, and both worked together with researchers at NERSC to be able to support NOvA’s antineutrino oscillation evidence.

The current standard practice for Fermilab experimenters is to perform similar analyses using less complex calculations through a combination of both traditional high-throughput computing and the distributed computing provided by Open Science Grid, a national partnership between laboratories and universities for data-intensive research. These are substantial resources, but they use a different model: Both use a large amount of computing resources over a long period of time. For example, some resources are offered only at a low priority, so their use may be preempted by higher-priority demands. But for complex, time-sensitive analyses such as NOvA’s, researchers need the faster processing enabled by modern, high-performance computing techniques.

SciDAC-4 is a DOE Office of Science program that funds collaboration between experts in mathematics, physics and computer science to solve difficult problems. The HEP on HPC project was funded specifically to explore computational analysis techniques for doing large-scale data analysis on DOE-owned supercomputers. Running the NOvA analysis at NERSC, the mission supercomputing facility for the DOE Office of Science, was a task perfectly suited for this project. Fermilab’s Jim Kowalkowski is the principal investigator for HEP on HPC, which also has collaborators from DOE’s Argonne National Laboratory, Berkeley Lab, University of Cincinnati and Colorado State University.

“This analysis forms a kind of baseline. We’re just ramping up, just starting to exploit the other capabilities of NERSC at an unprecedented scale,” Kowalkowski said.

The project’s goal for its first year is to take compute-heavy analysis jobs like NOvA’s and enable it on supercomputers. That means not just running the analysis, but also changing how calculations are done and learning how to revamp the tools that manipulate the data, all in an effort to improve techniques used for doing these analyses and to leverage the full computational power and unique capabilities of modern high-performance computing facilities. In addition, the project seeks to consume all computing cores at once to shorten that timeline.

The Fermilab HEPCloud facility provides cost-effective access to compute resources by optimizing usage across all available types and elastically expanding the resource pool on short notice by, for example, renting temporary resources on commercial clouds or using high-performance computers. HEPCloud enables NOvA and physicists from other experiments to use these compute resources in a transparent way.

For this analysis, “NOvA experimenters didn’t have to change much in terms of business as usual,” said Burt Holzman, HEPCloud principal investigator. “With HEPCloud, we simply expanded our local on-site-at-Fermilab facilities to include Cori and Edison at NERSC.”

Building on work the Fermilab HEPCloud team has been doing with researchers at NERSC to optimize high-throughput computing in general, the HEPCloud team was able to leverage the facility to achieve the million-core milestone. Thus, it holds the record for the most resources ever provisioned concurrently at a single facility to run experimental HEP workflows.

“This is the culmination of more than a decade of R&D we have done at Fermilab under SciDAC and the first taste of things to come, using these capabilities and HEPCloud,” said Panagiotis Spentzouris, head of the Fermilab Scientific Computing Division and HEPCloud sponsor.

“NOvA is an experimental facility located more than 2,000 miles away from Berkeley Lab, where NERSC is located. The fact that we can make our resources available to the experimental researchers near real-time to enable their time-sensitive science that could not be completed otherwise is very exciting,” said Wahid Bhimji, a NERSC data architect at Berkeley Lab who worked with the NOvA team. “Led by colleague Lisa Gerhardt, we’ve been working closely with the HEPCloud team over the last couple of years, also to support physics experiments at the Large Hadron Collider. The recent NOvA results are a great example of how the infrastructure and capabilities that we’ve built can benefit a wide range of high energy experiments.”

Going forward, Kowalkowski, Holzman and their associated teams will continue building on this achievement.

“We’re going to keep iterating,” Kowalkowski said. “The new facilities and procedures were enthusiastically received by the NOvA collaboration. We will accelerate other key analyses.”

NERSC is a DOE Office of Science user facility.

Fermilab is America’s premier national laboratory for particle physics and accelerator research. A U.S. Department of Energy Office of Science laboratory, Fermilab is located near Chicago, Illinois, and operated under contract by the Fermi Research Alliance LLC, a joint partnership between the University of Chicago and the Universities Research Association Inc. Visit Fermilab’s website at www.fnal.gov and follow us on Twitter at @Fermilab.

DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

 

At present scientists think that at the highest energies and earliest moments in time, all the fundamental forces may have existed as a single unified force. As the universe cooled just one microsecond after the Big Bang, it underwent a “phase transition” that transformed or “broke” the unified electromagnetic and weak forces into the distinct forces observed today.

One can compare the phase transition to the transformation of water into ice. In this familiar case, we call the transition a change in a state of matter. In the early-universe case, we call the transition electroweak symmetry breaking.

In the same way that we characterize the water-to-ice phase transition as occurring when the temperature drops below 32 degrees, we characterize the amount of symmetry breaking with a parameter called the weak mixing angle, whose value has been measured over the years.

By recreating the early-universe conditions in accelerator experiments, we have observed this transition and can measure the weak mixing angle that controls it. Our best understanding of the electroweak symmetry breaking involves the so-called Higgs mechanism, and the Nobel Prize-winning Higgs boson discovery in 2012 was a milestone in our understanding.

Measuring the weak mixing angle: a global pursuit

Previous determinations of the weak mixing angle from around the world disagreed, allowing for the possibility that maybe there are new fundamental particles to be discovered, or maybe even pointing to a misunderstanding in how we think about the fundamental forces. A new result from Fermilab helps to resolve the discrepancy and reinforces our standard theory of the fundamental forces.

Our combined understanding of particle physics and cosmology hinges on our understanding of the particles and their interactions: how the Higgs boson breaks the symmetry, how the different forces mesh with each other, whether there are new and unknown particles like dark matter or other unseen forces that could have a dramatic effect on the evolution of the universe.

The details of this symmetry breaking affect why matter is stable at all and how stars and galaxies form. The much improved agreement on the measurement of the weak mixing angle helps cement our understanding of the past, the character of what we observe today, and what we believe is in store for our future.

For two decades, the most precise measurements of the weak mixing angle came from experiments that collided electrons and positrons at the European laboratory CERN and SLAC National Accelerator Laboratory in California, each of which gave different answers. Their results have been puzzling because the probability of the two measurements to agree was less than one part in a thousand, suggesting the possibility of new phenomena. More input was needed.

Although the environment in Fermilab’s proton-antiproton Tevatron Collider was much harsher than either CERN’s or SLAC’s collider, with many more background particles, the large and well-understood data sets of the Tevatron’s CDF and DZero experiments allowed a new combined measurement that gives almost the same precision as the electron-positron measurements.

The new measurement of the mixing angle sin²θ_eff^lept is 0.23148 ± 0.00033, and it lies about midway between the CERN and SLAC measurements and thus is in good agreement with both of them, as well as with the average of all previous direct and indirect measurements of weak mixing. Thus, Occam’s razor suggests that those new particles and forces are not yet necessary and that our present particle physics and cosmology models remain good descriptors of the observed universe.

Read more in Physical Review D and on the Tevatron Run II webpage.

Breese Quinn is a member of the DZero collaboration and a physicist at the University of Mississippi. Willis Sakumoto is a member of the CDF collaboration and a physicist at the University of Rochester.

Note: A shortened version of this release was issued earlier today by the U.S. Department of Energy.

Yesterday, the U.S. Department of Energy and the Italian Embassy, on behalf of the Italian Ministry of Education, Universities and Research, signed an agreement for collaboration on research with the international Short-Baseline Neutrino (SBN) program hosted at DOE’s Fermi National Accelerator Laboratory.

The SBN program, started in 2015, comprises the development, installation and operation of three neutrino detectors spread over a distance of 600 meters on the Fermilab site. Italy and its National Institute of Nuclear Physics (INFN) are making major contributions to the SBN program, including the delivery and installation of one of the three detectors. Scientists will use the detectors and a neutrino beam from Fermilab’s particle accelerator complex to measure the properties of neutrino particles with unprecedented precision and search for a new type of particle known as a sterile neutrino.

“Italy is a strong partner of Fermilab and the department in advancing scientific research,” said Secretary of Energy Rick Perry. “Their expertise in the state-of-the-art technologies, essential to the SBN program and the international Deep Underground Neutrino Experiment, makes them a key partner in the global effort to solve the mysterious behavior of neutrinos.”

Previous experiments at DOE’s Los Alamos National Laboratory and at Fermilab have found evidence for neutrino interactions that scientists can’t explain within the best theory they currently have for the properties and interactions of the universe’s building blocks. Instead, the explanation could be the existence of a new type of neutrino with properties different from the three known types (electron, muon and tau neutrinos) or some other unknown phenomenon.

The SBN program comprises three neutrino detectors to be installed along one of Fermilab’s neutrino beamlines, known as the Booster Neutrino Beamline:

• The Short Baseline Near Detector, located 110 meters from the neutrino beam source: This detector will provide a measurement of the initial composition of the neutrino beam, which at this distance is expected to comprise almost exclusively muon neutrinos.
• The MicroBooNE detector, located 470 meters from the neutrino source: This detector, about the size of a school bus, will look for the first sign of the transformation of muon neutrinos into electron neutrinos.
• The ICARUS detector, located 600 meters from the neutrino source: Provided by INFN and first operated at the Gran Sasso Laboratory in Italy from 2010-2014, the ICARUS detector is the largest of the three detectors of the SBN program. It was refurbished at the European research center CERN before being shipped to Fermilab in 2017.

“After a long and productive scientific life at Gran Sasso National Laboratory, the ICARUS detector, refurbished at CERN, is starting a new adventure at Fermilab, a U.S. DOE laboratory where there is a long tradition of collaboration with INFN physicists,” said Fernando Ferroni, president of INFN. “The ICARUS detector, under the leadership of Nobel laureate Carlo Rubbia, will help to clarify the issue of the possible existence of sterile neutrinos. If discovered, they will be a revolution for the field.”

Once all the detectors have been installed in the Booster Neutrino Beamline, scientists will record data to solve the sterile neutrino puzzle. Using different distances from the neutrino source but the same liquid-argon technology, the three neutrino detectors will be able to distinguish whether their measurements are due to transformations between neutrino types involving a sterile neutrino or are due to other, previously unknown interactions.

“Together with the expertise of neutrino physicists from around the world, the three detectors of the SBN program will resolve the anomalies observed in previous experiments,” said Fermilab Director Nigel Lockyer. “This program will provide the best measurements for understanding these neutrino interactions.”

The signing of the SBN program collaboration agreement is an addendum to the umbrella agreement on neutrino physics research that the United States and Italy signed on July 17, 2015.

“The liquid-argon TPC technology was first introduced by the ICARUS team at INFN in Pavia over 20 years ago,” said INFN’s Carlo Rubbia, spokesperson for the ICARUS collaboration and a Nobel laureate. “The first physics experiment with 700 tons of ultrapure liquid argon was completed by ICARUS at the Gran Sasso Laboratory. Now at Fermilab, ICARUS will be the largest of three detectors to search with unprecedented accuracy neutrino events beyond the standard theory. This represents an important step forward in searching for new phenomena.”

Rubbia continued, “It is exciting to see that the same technology will be further developed by the huge 40-kiloton Deep Underground Neutrino Experiment detector, with the participation of more than 1,000 scientists worldwide.”

About 250 scientists from more than 50 institutions in Brazil, Italy, Switzerland, Turkey, the United Kingdom and the United States work on the international Short-Baseline Neutrino program at Fermilab. Funding is provided by CERN, the United States (DOE Office of Science and National Science Foundation), Italy (INFN), United Kingdom (STFC), Switzerland (NSF) and Brazil (FAPESP).

To learn more about the Short-Baseline Neutrino program at Fermilab, visit sbn.fnal.gov.

Fermilab is America’s premier national laboratory for particle physics and accelerator research. A U.S. Department of Energy Office of Science laboratory, Fermilab is located near Chicago, Illinois, and operated under contract by the Fermi Research Alliance LLC, a joint partnership between the University of Chicago and the Universities Research Association, Inc. Visit Fermilab’s website at www.fnal.gov and follow us on Twitter at @Fermilab.

The DOE Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

INFN, Istituto Nazionale di Fisica Nucleare, is the public Italian research institute dedicated to the study of the fundamental constituents of matter and their interactions. INFN conducts theoretical and experimental research in the fields of subnuclear, nuclear and astroparticle physics. Fundamental research in these areas requires the use of cutting-edge technology and instruments, developed by the INFN at its own laboratories and in collaboration with industries. All of the INFN’s research activities are conducted in close collaboration with Italian universities and undertaken within an international framework.

For more than 30 years, many Brazilian scientists have called Fermilab experiments their scientific home. The partnership between Fermilab and scientists from Brazil dates back to the 1980s, when the laboratory brought four young scientists to Illinois to conduct research for two years before returning to their home institutions in São Paulo and Rio de Janeiro. Today, about 80 scientists and students from 16 Brazilian institutions participate in Fermilab experiments, including the neutrino experiments DUNE, NOvA and MINERvA and the Dark Energy Survey.

The Brazilian Center for Research in Physics (Centro Brasileiro de Pesquisas Físicas, or CBPF) was the home of three of the four Brazilians who made the first trek to Fermilab in 1984 and today remains a leading institution for particle physics research in Brazil.

During a visit to Fermilab at which representatives from global funding agencies discussed the progress of and support for the Long-Baseline Neutrino Facility and Deep Underground Neutrino Experiment as part of the Resources Review Board, CBPF Director Ronald Shellard discussed the past, present and the future of collaboration between Brazilian scientists and Fermilab.

Q: What were the origins of the connection between Brazilian physicists and Fermilab?

A: The history goes back to the 1980s when [Fermilab Director Emeritus] Leon Lederman wanted to mobilize Latin America to join high-energy physics experiments around the world. Lederman visited many places and corresponded with the Brazilian Physical Society, which at the time was dominated by condensed matter researchers. Lederman offered fellowships to young Brazilians who came to Fermilab, and then they brought their students to Fermilab. That provoked a reaction from CERN, who offered one fellowship to a young Brazilian scientist, which I took.

Fermilab was absolutely essential in creating a large and quite vigorous community of high-energy physics researchers not only in Brazil, but in Latin America.

Q: On which Fermilab experiments did Brazilian scientists leave their mark?

A: The first Brazilian scientists at Fermilab worked on a fixed-target experiment exploring the charm quark. Later, there was a group that joined the DZero collider experiment, which has now moved to the CMS experiment [at the Large Hadron Collider]. Many Peruvian, as well from other Latin American countries, scientists took their degrees in Brazilian institutions on themes associated to Fermilab. Today they have a community in their countries. Also, there is a large collaboration of Brazilians working on the Pierre Auger Observatory. [Fermilab scientist] Paul Mantsch was crucial to making Auger happen on time and on schedule in a period when Argentina had a huge economic crisis.

Now I am here because of my responsibilities as director of CBPF, which is involved in DUNE. As a scientist I have been following with much interest the development of DUNE. It’s an excellent opportunity to have new, young physicists tackling the main, exciting problems in high-energy physics. And neutrino research is one of the windows to solve some of the fundamental questions in particle physics.

Q: How do you see the partnership between Brazil and Fermilab evolving over the next decade?

A: I would like to see an increase in the amount of people in their 30s participating in Fermilab’s flagship experiment, so that they have leadership roles in DUNE in 10 years. I also see the possibility of mobilizing local resources, local industries for providing equipment and services to DUNE, which is very important to creating a solid base for the continued growth of high-energy physics in Brazil. We are at the stage where we need to get involved not only in doing the analysis from the experiment, but actually building parts of it here in Brazil. Some of our younger scientists have already been developing detectors for DUNE.

We, in South America, are also building the ANDES project (which stand for Agua Negra Deep Experimental Site) that we hope to use for neutrino physics. This is connected to a road tunnel under the Andes that will connect Argentina and Chile all year long; the existing tunnel is closed two months a year. We need international collaboration to help fill it [with experimental equipment].

Q: How has high-energy physics research had broader impacts in Brazil?

A: Our high-energy physics community has now reached “adulthood,” and we have people doing great work. We have a light source in Brazil [SIRIUS] in the final stages of construction, a state of the art synchrotron radiation facility, which will be open for all.

There has also been an impact on education. For more than 10 years, high school teachers from Brazil have participated in a CERN course together with colleagues from Portugal. More than 300 teachers have participated from 23 states in Brazil. This has been very successful; they feel part of the scientific enterprise. The impact has been enormous.

Q: Where do you see Brazilian high-energy physics heading next?

A: One of the paradoxes of science is that the more you know, the less you know. We are stuck with a Standard Model and lots of things that don’t fit. In the 1970s when I did my Ph.D., progress was slow. Now it’s fast, even though experiments take a long time to build.

Brazilian science has come a long way. Within my lifetime we went from few scientists to a significant number of people and enough resources to give a big step forward and be more proactive in the experiment where we participate.

We have work to do. We need to spread high-energy physics around our country more; at the moment it is concentrated in the south around Rio de Janeiro and São Paulo. We need to mobilize industry to serve and benefit from these experiments. These things are happening — not at the rate I would like, but they are happening.

For more on CBPF and its relationship to Fermilab and to DUNE, view the video interview with Shellard below.

For more on the history of Brazilian scientific collaboration with Fermilab, read ““Brazil in Batavia: How a timely invitation sparked 30 years of partnership.”