The coevolution of particle physics and computing – Symmetry magazine
In the mid-twentieth century, particle physicists were peering deeper into the history and makeup of the universe than ever before. Over time, their calculations became too complex to fit on a blackboardor to farm out to armies of human computers doing calculations by hand.
To deal with this, they developed some of the worlds earliest electronic computers.
Physics has played an important role in the history of computing. The transistorthe switch that controls the flow of electrical signal within a computerwas invented by a group of physicists at Bell Labs. The incredible computational demands of particle physics and astrophysics experiments have consistently pushed the boundaries of what is possible. They have encouraged the development of new technologies to handle tasks from dealing with avalanches of data to simulating interactions on the scales of both the cosmos and the quantum realm.
But this influence doesnt just go one way. Computing plays an essential role in particle physics and astrophysics as well. As computing has grown increasingly more sophisticated, its own progress has enabled new scientific discoveries and breakthroughs.
Illustration by Sandbox Studio, Chicago with Ariel Davis
In 1973, scientists at Fermi National Accelerator Laboratory in Illinois got their first big mainframe computer: a 7-year-old hand-me-down from Lawrence Berkeley National Laboratory. Called the CDC 6600, it weighed about 6 tons. Over the next five years, Fermilab added five more large mainframe computers to its collection.
Then came the completion of the Tevatronat the time, the worlds highest-energy particle acceleratorwhich would provide the particle beams for numerous experiments at the lab. By the mid-1990s, two four-story particle detectors would begin selecting, storing and analyzing data from millions of particle collisions at the Tevatron per second. Called the Collider Detector at Fermilab and the DZero detector, these new experiments threatened to overpower the labs computational abilities.
In December of 1983, a committee of physicists and computer scientists released a 103-page report highlighting the urgent need for an upgrading of the laboratorys computer facilities. The report said the lab should continue the process of catching up in terms of computing ability, and that this should remain the laboratorys top computing priority for the next few years.
Instead of simply buying more large computers (which were incredibly expensive), the committee suggested a new approach: They recommended increasing computational power by distributing the burden over clusters or farms of hundreds of smaller computers.
Thanks to Intels 1971 development of a new commercially available microprocessor the size of a domino, computers were shrinking. Fermilab was one of the first national labs to try the concept of clustering these smaller computers together, treating each particle collision as a computationally independent event that could be analyzed on its own processor.
Like many new ideas in science, it wasnt accepted without some pushback.
Joel Butler, a physicist at Fermilab who was on the computing committee, recalls, There was a big fight about whether this was a good idea or a bad idea.
A lot of people were enchanted with the big computers, he says. They were impressive-looking and reliable, and people knew how to use them. And then along came this swarm of little tiny devices, packaged in breadbox-sized enclosures.
The computers were unfamiliar, and the companies building them werent well-established. On top of that, it wasnt clear how well the clustering strategy would work.
As for Butler? I raised my hand [at a meeting] and said, Good ideaand suddenly my entire career shifted from building detectors and beamlines to doing computing, he chuckles.
Not long afterward, innovation that sparked for the benefit of particle physics enabled another leap in computing. In 1989, Tim Berners-Lee, a computer scientist at CERN, launched the World Wide Web to help CERN physicists share data with research collaborators all over the world.
To be clear, Berners-Lee didnt create the internetthat was already underway in the form the ARPANET, developed by the US Department of Defense. But the ARPANET connected only a few hundred computers, and it was difficult to share information across machines with different operating systems.
The web Berners-Lee created was an application that ran on the internet, like email, and started as a collection of documents connected by hyperlinks. To get around the problem of accessing files between different types of computers, he developed HTML (HyperText Markup Language), a programming language that formatted and displayed files in a web browser independent of the local computers operating system.
Berners-Lee also developed the first web browser, allowing users to access files stored on the first web server (Berners-Lees computer at CERN). He implemented the concept of a URL (Uniform Resource Locator), specifying how and where to access desired web pages.
What started out as an internal project to help particle physicists share data within their institution fundamentally changed not just computing, but how most people experience the digital world today.
Back at Fermilab, cluster computing wound up working well for handling the Tevatron data. Eventually, it became industry standard for tech giants like Google and Amazon.
Over the next decade, other US national laboratories adopted the idea, too. SLAC National Accelerator Laboratorythen called Stanford Linear Accelerator Centertransitioned from big mainframes to clusters of smaller computers to prepare for its own extremely data-hungry experiment, BaBar. Both SLAC and Fermilab also were early adopters of Lees web server. The labs set up the first two websites in the United States, paving the way for this innovation to spread across the continent.
In 1989, in recognition of the growing importance of computing in physics, Fermilab Director John Peoples elevated the computing department to a full-fledged division. The head of a division reports directly to the lab director, making it easier to get resources and set priorities. Physicist Tom Nash formed the new Computing Division, along with Butler and two other scientists, Irwin Gaines and Victoria White. Butler led the division from 1994 to 1998.
These computational systems worked well for particle physicists for a long time, says Berkeley Lab astrophysicist Peter Nugent. That is, until Moores Law started grinding to a halt.
Moores Law is the idea that the number of transistors in a circuit will double, making computers faster and cheaper, every two years. The term was first coined in the mid-1970s, and the trend reliably proceeded for decades. But now, computer manufacturers are starting to hit the physical limit of how many tiny transistors they can cram onto a single microchip.
Because of this, says Nugent, particle physicists have been looking to take advantage of high-performance computing instead.
Nugent says high-performance computing is something more than a cluster, or a cloud-computing environment that you could get from Google or AWS, or at your local university.
What it typically means, he says, is that you have high-speed networking between computational nodes, allowing them to share information with each other very, very quickly. When you are computing on up to hundreds of thousands of nodes simultaneously, it massively speeds up the process.
On a single traditional computer, he says, 100 million CPU hours translates to more than 11,000 years of continuous calculations. But for scientists using a high-performance computing facility at Berkeley Lab, Argonne National Laboratory or Oak Ridge National Laboratory, 100 million hours is a typical, large allocation for one year at these facilities.
Although astrophysicists have always relied on high-performance computing for simulating the birth of stars or modeling the evolution of the cosmos, Nugent says they are now using it for their data analysis as well.
This includes rapid image-processing computations that have enabled the observations of several supernovae, including SN 2011fe, captured just after it began. We found it just a few hours after it exploded, all because we were able to run these pipelines so efficiently and quickly, Nugent says.
According to Berkeley Lab physicist Paolo Calafiura, particle physicists also use high-performance computing for simulationsfor modeling not the evolution of the cosmos, but rather what happens inside a particle detector. Detector simulation is significantly the most computing-intensive problem that we have, he says.
Scientists need to evaluate multiple possibilities for what can happen when particles collide. To properly correct for detector effects when analyzing particle detector experiments, they need to simulate more data than they collect. If you collect 1 billion collision events a year, Calafiura says, you want to simulate 10 billion collision events.
Calafiura says that right now, hes more worried about finding a way to store all of the simulated and actual detector data than he is about producing it, but he knows that wont last.
When does physics push computing? he says. When computing is not good enough We see that in five years, computers will not be powerful enough for our problems, so we are pushing hard with some radically new ideas, and lots of detailed optimization work.
Thats why the Department of Energys Exascale Computing Project aims to build, in the next few years, computers capable of performing a quintillion (that is, a billion billion) operations per second. The new computers will be 1000 times faster than the current fastest computers.
The exascale computers will also be used for other applications ranging from precision medicine to climate modeling to national security.
Innovations in computer hardware have enabled astrophysicists to push the kinds of simulations and analyses they can do. For example, Nugent says, the introduction of graphics processing units has sped up astrophysicists ability to do calculations used in machine learning, leading to an explosive growth of machine learning in astrophysics.
With machine learning, which uses algorithms and statistics to identify patterns in data, astrophysicists can simulate entire universes in microseconds.
Machine learning has been important in particle physics as well, says Fermilab scientist Nhan Tran. [Physicists] have very high-dimensional data, very complex data, he says. Machine learning is an optimal way to find interesting structures in that data.
The same way a computer can be trained to tell the difference between cats and dogs in pictures, it can learn how to identify particles from physics datasets, distinguishing between things like pions and photons.
Tran says using computation this way can accelerate discovery. As physicists, weve been able to learn a lot about particle physics and nature using non-machine-learning algorithms, he says. But machine learning can drastically accelerate and augment that processand potentially provide deeper insight into the data.
And while teams of researchers are busy building exascale computers, others are hard at work trying to build another type of supercomputer: the quantum computer.
Remember Moores Law? Previously, engineers were able to make computer chips faster by shrinking the size of electrical circuits, reducing the amount of time it takes for electrical signals to travel. Now our technology is so good that literally the distance between transistors is the size of an atom, Tran says. So we cant keep scaling down the technology and expect the same gains weve seen in the past."
To get around this, some researchers are redefining how computation works at a fundamental levellike, really fundamental.
The basic unit of data in a classical computer is called a bit, which can hold one of two values: 1, if it has an electrical signal, or 0, if it has none. But in quantum computing, data is stored in quantum systemsthings like electrons, which have either up or down spins, or photons, which are polarized either vertically or horizontally. These data units are called qubits.
Heres where it gets weird. Through a quantum property called superposition, qubits have more than just two possible states. An electron can be up, down, or in a variety of stages in between.
What does this mean for computing? A collection of three classical bits can exist in only one of eight possible configurations: 000, 001, 010, 100, 011, 110, 101 or 111. But through superposition, three qubits can be in all eight of these configurations at once. A quantum computer can use that information to tackle problems that are impossible to solve with a classical computer.
Fermilab scientist Aaron Chou likens quantum problem-solving to throwing a pebble into a pond. The ripples move through the water in every possible direction, simultaneously exploring all of the possible things that it might encounter.
In contrast, a classical computer can only move in one direction at a time.
But this makes quantum computers faster than classical computers only when it comes to solving certain types of problems. Its not like you can take any classical algorithm and put it on a quantum computer and make it better, says University of California, Santa Barbara physicist John Martinis, who helped build Googles quantum computer.
Although quantum computers work in a fundamentally different way than classical computers, designing and building them wouldnt be possible without traditional computing laying the foundation, Martinis says. We're really piggybacking on a lot of the technology of the last 50 years or more.
The kinds of problems that are well suited to quantum computing are intrinsically quantum mechanical in nature, says Chou.
For instance, Martinis says, consider quantum chemistry. Solving quantum chemistry problems with classical computers is so difficult, he says, that 10 to 15% of the worlds supercomputer usage is currently dedicated to the task. Quantum chemistry problems are hard for the very reason why a quantum computer is powerfulbecause to complete them, you have to consider all the different quantum-mechanical states of all the individual atoms involved.
Because making better quantum computers would be so useful in physics research, and because building them requires skills and knowledge that physicists possess, physicists are ramping up their quantum efforts. In the United States, the National Quantum Initiative Act of 2018 called for the National Institute of Standards and Technology, the National Science Foundation and the Department of Energy to support programs, centers and consortia devoted to quantum information science.
In the early days of computational physics, the line between who was a particle physicist and who was a computer scientist could be fuzzy. Physicists used commercially available microprocessors to build custom computers for experiments. They also wrote much of their own softwareranging from printer drivers to the software that coordinated the analysis between the clustered computers.
Nowadays, roles have somewhat shifted. Most physicists use commercially available devices and software, allowing them to focus more on the physics, Butler says. But some people, like Anshu Dubey, work right at the intersection of the two fields. Dubey is a computational scientist at Argonne National Laboratory who works with computational physicists.
When a physicist needs to computationally interpret or model a phenomenon, sometimes they will sign up a student or postdoc in their research group for a programming course or two and then ask them to write the code to do the job. Although these codes are mathematically complex, Dubey says, they arent logically complex, making them relatively easy to write.
A simulation of a single physical phenomenon can be neatly packaged within fairly straightforward code. But the real world doesnt want to cooperate with you in terms of its modularity and encapsularity, she says.
Multiple forces are always at play, so to accurately model real-world complexity, you have to use more complex softwareideally software that doesnt become impossible to maintain as it gets updated over time. All of a sudden, says Dubey, you start to require people who are creative in their own rightin terms of being able to architect software.
Thats where people like Dubey come in. At Argonne, Dubey develops software that researchers use to model complex multi-physics systemsincorporating processes like fluid dynamics, radiation transfer and nuclear burning.
Hiring computer scientists for research projects in physics and other fields of science can be a challenge, Dubey says. Most funding agencies specify that research money can be used for hiring students and postdocs, but not paying for software development or hiring dedicated engineers. There is no viable career path in academia for people whose careers are like mine, she says.
In an ideal world, universities would establish endowed positions for a team of research software engineers in physics departments with a nontrivial amount of computational research, Dubey says. These engineers would write reliable, well-architected code, and their institutional knowledge would stay with a team.
Physics and computing have been closely intertwined for decades. However the two developtoward new analyses using artificial intelligence, for example, or toward the creation of better and better quantum computersit seems they will remain on this path together.
Original post:
The coevolution of particle physics and computing - Symmetry magazine