The Untold Story of the Boldest Supply-Chain Hack Ever

But they had been at it only 24 hours when they found the passage they’d been looking for: a single file that appeared to be responsible for the rogue traffic. Carmakal believes it was December 11 when they found it.

The file was a .dll, or dynamic-link library—code components shared by other programs. This .dll was large, containing about 46,000 lines of code that performed more than 4,000 legitimate actions, and—as they found after analyzing it for an hour—one illegitimate one.

The main job of the .dll was to tell SolarWinds about a customer’s Orion usage. But the hackers had embedded malicious code that made it transmit intelligence about the victim’s network to their command server instead. Ballenthin dubbed the rogue code “Sunburst”—a play on SolarWinds. They were ecstatic about the discovery. But now they had to figure out how the intruders had snuck it into the Orion .dll.

This was far from trivial. The Orion .dll file was signed with a SolarWinds digital certificate, which was supposed to verify that the file was legitimate company code. One possibility was that the attackers had stolen the digital certificate, created a corrupt version of the Orion file, signed the file to make it look authentic, then installed the corrupt .dll on Mandiant’s server. Or, more alarmingly, they might have breached SolarWinds’ network and altered the legitimate Orion .dll source code before SolarWinds compiled it—converting the code into software—and signed it. The second scenario seemed so far-fetched that the Mandiant crew didn’t really consider it—until an investigator downloaded an Orion software update from the SolarWinds website. The backdoor was in it.

The implication was staggering. The Orion software suite had about 33,000 customers, some of whom had started receiving the hacked software update in March. That meant some customers might have been compromised for eight months already. The Mandiant team was facing a textbook example of a software-supply-chain attack—the nefarious alteration of trusted software at its source. In a single stroke, attackers can infect thousands, potentially millions, of machines.

In 2017 hackers had sabotaged a software supply chain and delivered malware to more than 2 million users by compromising the computer security cleanup tool CCleaner. That same year, Russia distributed the malicious NotPetya worm in a software update to the Ukrainian equivalent of TurboTax, which then spread around the world. Not long after, Chinese hackers also used a software update to slip a backdoor to thousands of Asus customers. Even at this early stage in the investigation, the Mandiant team could tell that none of those other attacks would rival the SolarWinds campaign.

SolarWinds Joins the Chase

it was a Saturday morning, December 12, when Mandia called SolarWinds’ president and CEO on his cell phone. Kevin Thompson, a 14-year veteran of the Texas company, was stepping down as CEO at the end of the month. What he was about to hear from Mandia—that Orion was infected—was a hell of a way to wrap up his tenure. “We’re going public with this in 24 hours,” Mandia said. He promised to give SolarWinds a chance to publish an announcement first, but the timeline wasn’t negotiable. What Mandia didn’t mention was that he was under external pressure himself: A reporter had been tipped off about the backdoor and had contacted his company to confirm it. Mandia expected the story to break Sunday evening, and he wanted to get ahead of it.

Thompson started making calls, one of the first to Tim Brown, SolarWinds’ head of security architecture. Brown and his staff quickly confirmed the presence of the Sunburst backdoor in Orion software updates and figured out, with alarm, that it had been delivered to as many as 18,000 customers since the spring of 2020. (Not every Orion user had downloaded it.) Thompson and others spent most of Saturday frantically pulling together teams to oversee the technical, legal, and publicity challenges they faced. They also called the company’s outside legal counsel, DLA Piper, to oversee the investigation of the breach. Ron Plesco, an attorney at Piper and former prosecutor with forensic expertise, was in his backyard with friends when he got the call at around 10 pm.