Dark Energy Survey releases first three years of data

A Fermilab team built and tested the first new superconducting accelerator cryomodule for the LCLS-II project, which will be the nation’s only X-ray free-electron laser facility.

Earlier this week, scientists and engineers at the U.S. Department of Energy’s Fermilab in Illinois loaded one of the most advanced superconducting radio-frequency cryomodules ever created onto a truck and sent it heading west.

Today, that cryomodule arrived at the U.S. DOE’s SLAC National Accelerator Laboratory in California, where it will become the first of 37 powering a three-mile-long machine that will revolutionize atomic X-ray imaging. The modules are the product of many years of innovation in accelerator technology, and the first cryomodule Fermilab developed for this project set a world record in energy efficiency.

These modules, when lined up end to end, will make up the bulk of the accelerator that will power a massive upgrade to the capabilities of the Linac Coherent Light Source at SLAC, a unique X-ray microscope that will use the brightest X-ray pulses ever made to provide unprecedented details of the atomic world. Fermilab will provide 22 of the cryomodules, with the rest built and tested at the U.S. DOE’s Thomas Jefferson National Accelerator Facility in Virginia.

The quality factor achieved in these components is unprecedented for superconducting radio-frequency cryomodules. The higher the quality factor, the lower the cryogenic load, and the more efficiently the cavity imparts energy to the particle beam. Fermilab’s record-setting cryomodule doubled the quality factor compared to the previous state-of-the-art.

“LCLS-II represents an important technological step which demonstrates that we can build more efficient and more powerful accelerators,” said Fermilab Director Nigel Lockyer. “This is a major milestone for our accelerator program, for our productive collaboration with SLAC and Jefferson Lab and for the worldwide accelerator community.”

Today’s arrival is merely the first. From now into 2019, the teams at Fermilab and Jefferson Lab will build the remaining cryomodules, including spares, and scrutinize them from top to bottom, sending them to SLAC only after they pass the rigorous review.

“It’s safe to say that this is the most advanced machine of its type,” said Elvin Harms, a Fermilab accelerator physicist working on the project. “This upgrade will boost the power of LCLS, allowing it to deliver X-ray laser beams that are 10,000 times brighter than it can give us right now.”

With short, ultrabright pulses that will arrive up to a million times per second, LCLS-II will further sharpen our view of how nature works at the smallest scales and help advance transformative technologies of the future, including novel electronics, life-saving drugs and innovative energy solutions. Hundreds of scientists use LCLS each year to catch a glimpse of nature’s fundamental processes.

To meet the machine’s standards, each Fermilab-built cryomodule must be tested in nearly identical conditions as in the actual accelerator. Each large metal cylinder — up to 40 feet in length and 4 feet in diameter — contains accelerating cavities through which electrons zip at nearly the speed of light. But the cavities, made of superconducting metal, must be kept at a temperature of 2 Kelvin (minus 456 degrees Fahrenheit).

To achieve this, ultracold liquid helium flows through pipes in the cryomodule, and keeping that temperature steady is part of the testing process.

“The difference between room temperature and a few Kelvin creates a problem, one that manifests as vibrations in the cryomodule,” said Genfa Wu, a Fermilab scientist working on LCLS-II. “And vibrations are bad for linear accelerator operation.”

In initial tests of the prototype cryomodule, scientists found vibration levels that were higher than specification. To diagnose the problem, they used geophones — the same kind of equipment that can detect earthquakes — to rule out external vibration sources. They determined that the cause was inside the cryomodule and made a number of changes, including adjusting the path of the flow of liquid helium. The changes worked, substantially reducing vibration levels — to a 10th of what they were originally — and have been successfully applied to subsequent cryomodules.

Fermilab scientists and engineers are also ensuring that unwanted magnetic fields in the cryomodule are kept to a minimum, since excessive magnetic fields reduce the operating efficiency.

“At Fermilab, we are building this machine from head to toe,” Lockyer said. “From nanoengineering the cavity surface to the integration of thousands of complex components, we have come a long way to the successful delivery of LCLS-II’s first cryomodule.”

Fermilab has tested seven cryomodules, plus one built and previously tested at Jefferson Lab, with great success. Each of those, along with the modules yet to be built and tested, will get its own cross-country trip in the months and years to come.

Read more about the LCLS-II project in SLAC’s press release.

This project is supported by DOE’s Office of Science. LCLS is a DOE Office of Science user facility.

Fermilab is America’s premier national laboratory for particle physics research. A U.S. Department of Energy Office of Science laboratory, Fermilab is located near Chicago, Illinois and operated under contract by the Fermi Research Alliance, LLC. Visit Fermilab’s website at http://www.fnal.gov and follow us on Twitter @Fermilab.

The DOE Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit http://science.energy.gov.

In one sense, the two ProtoDUNE detectors are small. As prototypes of the much larger planned Deep Underground Neutrino Experiment, they are only representative slices, each measuring about 1 percent of the size of the final detector. But in all other ways, the ProtoDUNE detectors are simply massive.

Once they are complete later this year, these two test detectors will be larger than any detector ever built that uses liquid argon, its active material. The international project involves dozens of experimental groups coordinating around the world. And most critically, the ProtoDUNE detectors, which are being installed and tested at the European particle physics laboratory CERN, are the rehearsal spaces in which physicists, engineers and technicians will hammer out nearly every engineering problem confronting DUNE, the biggest international science project ever conducted in the United States. It is hosted by the Department of Energy’s Fermi National Accelerator Laboratory.

 

Gigantic detector, tiny neutrino

DUNE’s mission, when it comes online in the mid-2020s, will be to pin down the nature of the neutrino, the most ubiquitous particle of matter in the universe. Despite neutrinos’ omnipresence — they fill the universe, and trillions of them stream through us every second — they are a pain in the neck to capture. Neutrinos are vanishingly small, fleeting particles that, unlike other members of the subatomic realm, are heedless of the matter through which they fly, never stopping to interact.

Well, almost never.

Once in a while, scientists can catch one. And when they do, it might tell them a bit about the origins of the universe and why matter predominates over antimatter — and thus how we came to be here at all.

A global community of more than 1,000 scientists from 31 countries are building DUNE, a megascience experiment hosted by Fermilab. The researchers’ plan is to observe neutrinos using two detectors separated by 1,300 kilometers — one at Fermilab outside Chicago and a second one a mile underground in South Dakota at the Sanford Underground Research Facility. Having one at each end enables scientists to see how neutrinos transform as they travel over a long distance.

The DUNE collaboration is going all-in on the bigger-is-better strategy; after all, the bigger the detector, the more likely scientists are to snag a neutrino. The detector located in South Dakota, called the DUNE far detector, will hold 70,000 metric tons (equivalent to about 525,000 bathtubs) of liquid argon to serve as the neutrino fishing net. It comprises four large modules. Each will stand four stories high and, not including the structures that house the utilities, occupy a footprint roughly equal to a soccer field.

In short, DUNE is giant.

Lots of room in ProtoDUNE

The ProtoDUNE detectors are small only when compared to the giant DUNE detector. If each of the four DUNE modules is a 20-room building, then each ProtoDUNE detector is one room.

But one room large enough to envelop a small house.

As one repeatable unit of the ultimate detector, the ProtoDUNE detectors are necessarily big. Each is an enormous cube—about two stories high and about as wide — and contains about 800 metric tons of liquid argon.

Why two prototypes? Researchers are investigating two ways to use argon and so are constructing two slightly different but equally sized test beds. The single-phase ProtoDUNE uses only liquid argon, while the dual-phase ProtoDUNE uses argon as both a liquid and a gas.

“They’re the largest liquid-argon particle detectors that have ever been built,” said Ed Blucher, DUNE co-spokesperson and a physicist at the University of Chicago.

As DUNE’s test bed, the ProtoDUNE detectors also have to offer researchers a realistic picture of how the liquid-argon detection technology will work in DUNE, so the instrumentation inside the detectors is also at full, giant scale.

“If you’re going to build a huge underground detector and invest all of this time and all of these resources into it, that prototype has to work properly and be well-understood,” said Bob Paulos, director of the University of Wisconsin–Madison Physical Sciences Lab and a DUNE engineer. “You need to understand all the engineering problems before you proceed to build literally hundreds of these components and try to transport them all underground.”

Partners in ProtoDUNE

ProtoDUNE is a rehearsal for DUNE not only in its technical orchestration but also in the coordination of human activity.

When scientists were planning their next-generation neutrino experiment around 2013, they realized that it could succeed only by bringing the international scientific community together to build the project. They also saw that even the prototyping would require an effort of global proportions — both geographically and professionally. As a result, DUNE and ProtoDUNE actively invite students, early-career scientists and senior researchers from all around the world to contribute.

“The scale of ProtoDUNE, a global collaboration at CERN for a U.S.-based megaproject, is a paradigm change in the way neutrino science is done,” said Christos Touramanis, a physicist at the University of Liverpool and one of the co-coordinators of the single-phase detector. For both DUNE and ProtoDUNE, funding comes from partners around the world, including the Department of Energy’s Office of Science and CERN.

The successful execution of ProtoDUNE’s assembly and testing by international groups requires a unity of purpose from parties that could hardly be farther apart, geographically speaking.

Scientists say the effort is going smoothly.

“I’ve been doing neutrino physics and detector technology for the last 20 or 25 years. I’ve never seen such an effort go up so nicely and quickly. It’s astonishing,” said Fermilab scientist Flavio Cavanna, who co-coordinates the single-phase ProtoDUNE project. “We have a great collaboration, great atmosphere, great willingness to make it. Everybody is doing his or her best to contribute to the success of this big project. I used to say that ProtoDUNE was mission impossible, because—in the short time we were given to make the two detectors, it looked that way in the beginning. But looking at where we are now, and all the progress made so far, it starts turning out to be mission possible.”

Inside the liquid-argon test bed

So how do neutrino liquid-argon detectors work? Most of the space inside serves as the arena of particle interaction, where neutrinos can smash into an argon atom and create secondary particles. Surrounding this interaction space is the instrumentation that records these rare collisions, like a camera committing the scene to film. DUNE collaborators are developing and constructing the recording instruments that will capture the evidence of these interactions.

One signal is ionization charge: A neutrino interaction generates other particles that propagate through the detector’s vast argon pool, kicking electrons — called ionization electrons — off atoms as they go. The second signal is light.

The first signal emerges as a streak of ionization electrons.

To record the signal, scientists will use something called an anode plane array, or APA. An APA is a screen created using 24 kilometers of precisely tensioned, closely spaced, continuously wound wire. This wire screen is positively charged, so it attracts the negatively charged electrons.

Much the way a wave front approaches the beach’s shore, the particle track — a string of the ionization electrons — will head toward the positively charged wires inside the ProtoDUNE detectors. The wires will send information about the track to computers, which will record its properties and thus information about the original neutrino interaction.

A group in the University of Wisconsin–Madison Physical Sciences Lab led by Paulos designed the single-phase ProtoDUNE wire arrays. The Wisconsin group, the Science and Technology Facilities Council’s Daresbury Laboratory in the UK and several UK universities are building APAs for the same detector. The first APA from Wisconsin arrived at CERN last year; the first from Daresbury Lab arrived earlier this week.

“These are complicated to build,” Paulos said, noting that it currently takes about three months to build just one. “Building these 6-meter-tall anode planes with continuously wound wire—that’s something that hasn’t been done before.”

The anode planes attract the electrons. Pushing away the electrons will be a complementary set of panels, called the cathode plane. Together, the anode and cathode planes behave like battery terminals, with one repelling electron tracks and the other drawing them in. A group at CERN designed and is building the cathode plane.

The dual-phase detector will operate on the same principle but with a different configuration of wire arrays. A special layer of electronics near the cathode will allow for the amplification of faint electron tracks in a layer of gaseous argon. Groups at institutions in France, Germany and Switzerland are designing those instruments. Once complete, they will also send their arrays to be tested at CERN.

Then there’s the business of observing light.

The flash of light is the result of a release of energy from the electron in the process of getting bumped from an argon atom. The appearance of light is like the signal to start a stopwatch; it marks the moment the neutrino interaction in a detector takes place. This enables scientists to reconstruct in three dimensions the picture of the interaction and resulting particles.

On the other side of the equator, a group at the University of Campinas in Brazil is coordinating the installation of instruments that will capture the flashes of light resulting from particle interactions in the single-phase ProtoDUNE detector.

Two of the designs for the single-phase prototype — one by Indiana University, the other by Fermilab and MIT — are of a type called guiding bars. These long, narrow strips work like fiber optic cables: they capture the light, convert it into light in the visible spectrum and finally guide it to an external sensor.

A third design, called ARAPUCA, was developed by three Brazilian universities and Fermilab and is being partially produced at Colorado State University. Named for the Guaraní word for a bird trap, the efficient ARAPUCA design will be able to “trap” even very low light signals and transmit them to its sensors.

 

“The ARAPUCA technology is totally new,” said University of Campinas scientist Ettore Segreto, who is co-coordinating the installation of the light detection systems in the single-phase prototype. “We might be able to get more information from the light detection — for example, greater energy resolution.”

Groups from France, Spain and the Swiss Federal Institute of Technology are developing the light detection system for the dual-phase prototype, which will comprise 36 photomultiplier tubes, or PMTs, situated near the cathode plane. A PMT works by picking up the light from the particle interaction and converting it into electrons, multiplying their number and so amplifying the signal’s strength as the electrons travel down the tube.

With two tricked-out detectors, the DUNE collaboration can test their picture-taking capabilities and prepare DUNE to capture in exquisite detail the fleeting interactions of neutrinos.

Bringing instruments into harmony

But even if they’re instrumented to the nines inside, two isolated prototypes do not a proper test bed make. Both ProtoDUNE detectors must be hooked up to computing systems so particle interaction signals can be converted into data. Each detector must be contained in a cryostat, which functions like a thermos, for the argon to be cold enough to maintain a liquid state. And the detectors must be fed particles in the first place.

CERN is addressing these key areas by providing particle beam, innovative cryogenics and computing infrastructures, and connecting the prototype detectors with the DUNE experimental environment.

DUNE’s neutrinos will be provided by the Long-Baseline Neutrino Facility, or LBNF, which held a groundbreaking for the start of its construction in July. LBNF, led by Fermilab, will provide the construction, beamline and cryogenics for the mammoth DUNE detector, as well as Fermilab’s chain of particle accelerators, which will provide the world’s most intense neutrino beam to the experiment.

CERN is helping simulate that environment as closely as possible with the scaled-down ProtoDUNE detectors, furnishing them with particle beams so researchers can characterize how the detectors respond. Under the leadership of scientist Marzio Nessi, last year the CERN group built a new facility for the test beds, where CERN is now constructing two new particle beamlines that extend the lab’s existing network.

 

In addition, CERN built the ProtoDUNE cryostats — the largest ever constructed for a particle physics experiment — which also will serve as prototypes for those used in DUNE. Scientists will be able to gather and interpret the data generated from the detectors with a CERN computing farm and software and hardware from several UK universities.

“The very process of building these prototype detectors provides a stress test for building them in DUNE,” Blucher said.

CERN’s beam schedule sets the schedule for testing. In December, the European laboratory will temporarily shut off beam to its experiments for upgrades to the Large Hadron Collider. DUNE scientists aim to position the ProtoDUNE detectors in the CERN beam before then, testing the new technologies pioneered as part of the experiment.

“ProtoDUNE is a necessary and fundamental step towards LBNF/DUNE,” Nessi said. “Most of the engineering will be defined there and it is the place to learn and solve problems. The success of the LBNF/DUNE project depends on it.”

This article also appears in symmetry magazine.

Fermilab is America’s premier national laboratory for particle physics research. A U.S. Department of Energy Office of Science laboratory, Fermilab is located near Chicago, Illinois and operated under contract by the Fermi Research Alliance, LLC. Visit Fermilab’s website at http://www.fnal.gov and follow us on Twitter @Fermilab.

The DOE Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit http://science.energy.gov.

Being on time is important – just ask Lewis Carroll’s leporine friend – and one group who knows this more than most are particle physicists, whose work revolves around keeping track of near-light speed blips of matter.

As particle accelerators and experiments have become increasingly complex and choreographed over the decades, technology behind the scenes has had to innovate to keep up. One such example is White Rabbit, a clever timing and data transfer system that is playing a key role in modern particle physics.

“We are always pushing our experiments to higher and higher precisions,” said Angela Fava, scientist on Fermilab’s ICARUS neutrino detector and part of the team exploring White Rabbit at Fermilab. “White Rabbit is really useful because it can reach time precisions down to less than a billionth of a second.”

Keeping time

In modern particle accelerators, many separate components have to be activated in sequence in a timely manner to identify and track particles passing by at the speed of light. This requires very precise synchronization and timing systems to determine when these events should occur – an egg timer won’t cut it here.

Until recently, this timing has usually been achieved with devices that are hard-wired into experimental equipment, such as the General Machine Timing (GMT) system at CERN. But GMT has limitations, including a low data bandwidth, the capacity to only send signals one way through the network, and an inability to self-calibrate — to internally calculate how long a signal has taken to travel — which results in timing errors.

As experiments grow in complexity and require nanosecond coordination, physicists have been left with a need for a one-size-fits-all system that can provide the required time synchronization and still be compatible with systems from multiple sources and vendors that are already in place.

The solution is White Rabbit, an open-source system that builds on common and accessible Ethernet technology – the same technology behind wired internet access. The system works kind of like an everyday computer network, too, with circuit boards called “nodes,” controlled by a specially written program.

Up to around 1,000 nodes can be linked in one White Rabbit network, all connected together with a web of optical fibers – up to 10 kilometers long – to exchange information. As the technology develops, the system will likely be able to support even more nodes separated by greater distances.

Since precise timing is so important in modern experiments, White Rabbit’s power comes in its ability to keep itself synchronized, no matter the cable length between nodes or other external factors. Even relatively small changes in cable temperature can affect travel time on the scale of nanoseconds, for example.

A White Rabbit system works kind of like a hierarchy, where one of the nodes in a network is designated a “master” and is responsible for keeping all the other nodes in check. The external time is fed into the master from high-precision atomic oscillators via orbiting GPS satellites, the same technology on which Google Maps navigation is based.

This exact time is digitally attached to blips of data – which, for example, include control instructions for accelerators – that constantly fly around the network. By sending the time tags back and forth between nodes, which GMT isn’t able to do, the system can calculate the time delays it takes for data to travel through cables and correct for them, keeping all the nodes in synchronization with the correct time and ensuring experimental events are kept coordinated.

Fava and scientist Donatella Torretta, together with William Badgett at Fermilab, are currently working on installing White Rabbit into some of Fermilab’s experiments, including the Short-Baseline Neutrino (SBN) Program, which will study neutrinos – tiny, elusive particles. The first use of White Rabbit in North America, the system can be used to time-tag the neutrinos from their production at the beam source through to the detector at the end of the experiment.

On the SBN ICARUS detector, White Rabbit can also be used to get an extremely accurate tagging of unwanted cosmic particles that come from space and get in the way of the experiment, potentially hiding the neutrino signatures.

“It would be possible to run ICARUS without White Rabbit, but it’s lot easier if we use it,” said Fava. “And it’s all in real-time too, so it saves on our computing power and storage.”

Open science

White Rabbit was first conceived in around 2008 as an international collaboration between CERN, the GSI Helmholtz Centre for Heavy Ion Research in Germany, and other partners, and was introduced to boost the abilities of the Large Hadron Collider.

From the beginning, the collaboration has made both the hardware and software for the timing system openly available to anyone around the world. The physical equipment can be purchased from commercial vendors, while the software is completely free and easily accessible online.

“Everybody benefits when science is open,” said Torretta, who learned about White Rabbit at a demonstration workshop at CERN. “As the technology develops, it’s becoming more and more popular.”

Torretta has since attended further workshops to learn more, including one recently in Barcelona, which was organized by White Rabbit experts from CERN.

The CERN development team also took care to ensure the design was as general as possible, so as to allow a large range of practical applications for the technology, including outside of science. A group in the Netherlands has even used White Rabbit to transmit official time between Dutch cities with nanosecond accuracies.

 

With information technology integrated into so many aspects of our lives, having some computer science skills will be essential for the future workforce. However, less than half of all schools teach computer science. Furthermore, while 71 percent of STEM jobs are currently in computer science, only 8 percent of STEM graduates study computer science. These gaps are concerning, but there are efforts under way to close them.

Hour of Code is a worldwide movement designed to demystify code, to show that anybody can learn the basics and to broaden participation in computer science and other technical fields. Over 400 partners and 200,000 educators globally currently support the movement.

As part of Computer Science Education Week on Dec. 4-10, Fermilab partnered with Argonne National Laboratory on an initiative to bring Hour of Code activities and coding role models to local schools. Fourteen employees from Fermilab, along with several from various Argonne organizations, visited area elementary, middle and high schools and spoke about their labs, their careers and coding in general. They also assisted students with coding exercises.

This is the second year Penelope Constanta, application developer and system analyst, participated in Hour of Code. Over the course of one day, Constanta visited 13 classrooms at Oswego’s Fox Chase Elementary School, from first grade to fifth grade, with varying levels of coding experience. She explained to the children who had never coded what a computing program is, and for those who had, she helped with the coding tasks that they were asked to do.

“I did love the kids’ reactions when I asked them to ‘program me’ to move to a location in the classroom or do some simple task,” Constanta said. “At all levels, they were able to grasp the simple concepts that I introduced.” For all but two of the classes she attended, this was the first time these kids had heard about programming.

Scientist Adam Lyon spent a morning at Thomas Jefferson Junior High School in Woodridge.

“It was a lot of fun, and the teachers and kids were great,” he said. “The Minecraft and Flappy Bird projects were by far the most popular, though several kids told me that Minecraft is so 2015.”

Application developer and system analyst Kris Brandt attended both introductory and AP computer science classes at St. Charles East High School, where the students are already coding, so she didn’t need to introduce the concept of coding.

“Instead, my talk focused on scientific versus core computing at Fermilab and computing careers in general,” Brandt said. “They were also curious and impressed by some of the stats from ‘Computing by the Numbers,’ especially the amount of data we manage and the cybersecurity stats. Quantum computing was also a hot topic. They were a very curious and smart group of kids, which made the experience fun.”

Fermilab Chief Information Officer Rob Roser highlighted Fermilab Computing’s positive impact in the community.

“I am very proud of the enthusiastic participation from all branches of Computing in this important outreach event,” Roser said.  “Reaching out through the schools and showing these kids that coding is both fun and accessible to them and that the end result can change the world is very powerful.”

The Fermilab participants and the schools they visited were:

Kris Brandt: St. Charles East High School
Penelope Constanta: Fox Chase Elementary School, Oswego
Lynn Garren: Churchill Elementary School, Oswego
Ken Herner: Lemont High School
Burt Holzman: Wolf’s Crossing Elementary, Aurora
Tanya Levshina: Morrill Math & Science Elementary School, Chicago
Adam Lyon: Thomas Jefferson Junior High School
Marco Mambelli: UIC College Prep, Chicago
Craig Mohler: Timber Ridge Middle School, Plainfield
Keenan Newton: Thornton Fractional South High School, Lansing
Irene Shiu: Boulder Hill Elementary, Oswego
Margaret Votava: Bower Elementary, Warrenville
Tammy Whited: Grace McWayne Elementary, Batavia
Michael Zalokar: Rotolo Middle School, Batavia

Marcia Teckenbrock is the communications manager in the Office of the Chief Information Officer.