The Equifax Data Breach Pt. I: A Big Data Bubble

Link To Part II

Facebook collects so much information about you, that its algorithm can sense when you’re falling out of love with your partner. But it started out, in 2003, as a game for college kids. Google can tell when you’re pregnant before you do. Its original purpose was to index the world’s collective knowledge.

Typically speaking, you don’t just build a big data company. The story goes that a couple of young founders start in a garage, with a small business idea–selling books maybe, or a kitschy concept for an app. Only years later–after massive growth and success–does a company seize their potential for information trafficking.

But in their 120 year history, Equifax never sold anything, or provided any service to ordinary folks in order to grow their business. It’s a multi-billion-dollar corporation today, so it appears to the eye quite different than it did at the beginning. But before we dive into the story of the notorious data breach of 2017 – it’s important to understand that underneath the surface, though, Equifax is the exact same company it has always been. And it’s because, long before high speed internet–before a single employee was hired–the founder of the company had an insight that most of us would come to only a century later: that there was big money to be made in collecting and selling people’s information, whether they liked it or not.

Equifax’s Origins

Cator Woolford was a former bank employee from Maryland who, in 1898, owned a grocery store in Chattanooga, Tennessee. Back then, being a grocer meant something different than it does today. Woolford personally knew many of the folks who walked in his front doors. A few of them were his suppliers. Others were his neighbors. The shop was part of a community, more so than any Stop ‘n Shop or Whole Foods is today.

Being part of a community meant acting as a community member, in a way that might seem foreign to us today. For example: if a customer were down on his luck, he might be lent a jar of milk, or some eggs, on the promise that he’d pay it back when times were better. Lending was not uncommon among food suppliers of the time, whether you were a grocer or a butcher or a baker. But Woolford thought about lending differently than his colleagues would have. Having worked in banking, he knew all about lending. In particular, he knew that the key to a sustainable loan system was credit.

At his store, Woolford began taking detailed notes on his customers, to determine their credit “worthiness”. Could they be relied upon, to repay their loan? Woolford was a pretty serious-looking guy–a receding hairline yet thick eyebrows, dark brown eyes, one lazy, and ears so large they might hear your inner thoughts. Getting in Cator’s book, well…you’d hope he wrote something good about you, because he wasn’t messing around.

Soon it became clear that these notes were of value, not just to Cator but to other businesses who wished to buy copies. He brought his younger brother Guy–a lawyer–on board and, together, they moved to Atlanta, Georgia, in 1899 to found the Retail Credit Company. RCC began small–operating out of a single room on the fifth floor of a building, located where Georgia State University’s campus now stands–but the brothers were enterprising. They made connections with food suppliers in Atlanta, then elsewhere in the state, copying customer credit information, until the job got so big they had to hire three, then five more employees to help.

After the first couple of years, the Woolford business model diversified. The company began going door to door among all kinds of businesses, inquiring about customers and reflecting their findings in detailed ledgers. The ledgers were thorough, yet simple, with categories such as “prompt” and “slow”, or “requires cash”. The notes were put altogether into a book, called the “Merchant’s Guide”, which was then sold back to those stores from which the information came.

After an initial slow period, the Merchant’s Guide became a big hit, especially for insurance companies. The benefit of this information, to businesses across the Eastern United States, was tangible. No longer did loans have to be significant risks, taken on as a result of personal ties or emotional pleas. With information about the character of the customer, merchants could now make smart loans based on real evidence. This applied to your regulars, but also to customers who’d never walked into your store before, because the Merchant’s Guide had information on them, too.

But not once yet have we heard from any customers themselves. The Merchant’s Guide was a product informed by merchants, sold to merchants, about customers. All the while you–the customer whose been the object of this entire process–might not know a thing about the Merchant’s Guide. What’s more: there’s nothing you could do about it, even if you did know about it. Any merchant is free to say whatever they want about you, and to have that information published and sold to any other merchant in the country.

And it would’ve been one thing, if all the Woolfords were collecting was information about your spending. But their scope was much larger, and without any boundaries. Alongside any information about your debts might be information known or otherwise overheard about your political, social or sexual life. Rumors about you, that might speak to your character. Were you of an undesirable political affiliation–one which might make you someone not to be dealt with? Did you frequent brothels, with such frequency that it might reasonably threaten your financial security down the line? The Retail Credit Company wasn’t just taking notes about your finances, they were building a profile of the very person you were. All behind your back. For sale to essentially anyone but you.

A Futile Push Back Against Equifax

As the company grew into a veritable corporation over the next century, their reputation as a giant, invisible snoop grew with them. Where once Cator and Guy went door to door, now thousands of investigators around the nation were hired to do the same. By the mid-20th century they’d compiled information on millions of Americans. In the 1960s, they began making arrangements to computerize it all, which massively assisted their data throughput. In the ‘70s, the FTC accused RCC of attempting to construct a monopoly, as they bought out competing credit bureaus in states across the country. A decade-long court battle ensued, but nothing came of it, and in the ‘80s alone they bought out over 100 smaller companies in the market.

Congress, in 1974, accused RCC of rewarding employees based on how much negative information they could dig up on customers–whether it be factual information about an unpaid debt, or mere rumors about marital infidelity. But just as the U.S. government was condemning their predatory practices, J. Edgar Hoover’s FBI was mulling through their databases and using it for their own ends.

Meanwhile, some customers tried suing the company. You can go back and find some of these cases, like that of Paul Roemer, who in 1975 managed to prove the information in his credit file false before a U.S. District Court. Or Cassius L. Peacock Jr., a man whose illustrious life is surpassed only by his illustrious name. His credit file indicated he’d falsified financial reports from his company, and slept with many prostitutes along the way. Cassius sued for invasion of privacy and libel, and lost. At least he had plenty of prostitutes, to comfort him in his time of need.

Overall those lawsuits, the government edicts and press reports had relatively little effect on Retail Credit Company’s business practices, or their bottom line. And yet, cumulatively, the bad press was undeniable. In an effort to rebrand, the company changed its name to Equifax.

Richard Smith

“Equifax”, short for “equitable factual information”, is still headquartered in Atlanta. It’s a gray, angular building–far-flung from any Chattanooga grocery store. The company is now valued at 14 billion dollars, employs 10,000 people, and holds data on nearly one billion people. Its data stores are estimated to be over 1,000 times larger than that of the largest library in the world, the Library of Congress. Equifax in 2019 looks nothing like it did in 1899, but under the hood, it really is the same company it always was.

Most people know Equifax as one of the big three credit reporting agencies, along with Experian and TransUnion. These agencies exist to collect data on you, from banks and other lenders, in order to then organize and sell it back to any other merchant who might be considering you for a loan. If you fall behind on your credit card payments, it’ll be reflected in the credit score Equifax calculates on your behalf. If you apply for a mortgage on a new home, the bank will request your credit score, and use it to decide whether offering you that house is in their interests. That makes your credit score one of the most powerful influences over your life, and the arbiters of that score, one of the most powerful influencers over your life.

When Richard Smith signed on as CEO of Equifax in 2005, he saw a successful yet stagnant company. Being in competition with only two other companies meant Equifax didn’t really have to work that hard to succeed, and retain their power in the market. There were never any fourth or fifth reporting agencies, so it was no wonder that a culture of complacency took hold. Richard Smith became the company’s most consequential CEO since the Woolford brothers, because he approached the same problem from an entirely opposite perspective.

I used the words “most people know Equifax as one of the big three credit reporting agencies” just now, because, really, Equifax became much more than a credit reporting agency ten years into Richard Smith’s tenure as CEO. Early on he made an expensive and profitable bet by purchasing Talx, a company with hundreds of millions of employment records. Equifax expanded into two dozen countries, building out departments in analytics and AI, developing dozens of new products, and expanding the range of their data collection. In doing so, the company’s revenue more than doubled and its stock price quadrupled.

Nowadays, credit reports is just one component of their larger business model. The company collects demographic data, and financial data not directly related to credit such as wealth, property valuations, and the salaries of nearly half of America’s workforce. They scrape data from Facebook and other social media, personal identification information from government records, and much more. They have teams dedicated to analyzing their data, to predict consumer behavior. They sell software-as-a-service, allowing other businesses to access and leverage their data. Oh and, I forgot: have I mentioned the word “data” yet?

Calling Equifax a credit reporting agency is a little like calling Google a search engine–it’s what we know them for, but it’s a very narrow view of a very large and multifaceted big data corporation. Their product isn’t really credit reports, it’s data. Lots of it, whether you like it or not, about you.

No matter how much Equifax grew in 120 years, though, their problems remained the same. The invasion of privacy. The inaccuracies. But one new problem cropped up in recent years, that wasn’t so relevant for the Woolfords in 1899: there were now people out in the world, with the tools to infringe upon and steal their records.

Small Scale Breaches

In a speech given shortly after becoming CEO, Richard Smith boasted that, quote, “we have been blessed in our rich history to never have a major breach.” And that was true, at the time. In company documents, Equifax boasted about being secure gatekeepers of everyone’s data. But by 2016, if you were to peek inside, a bastion of big data security is not what you would see.

Equifax experienced a few, smaller-scale data breaches around 2013 and 2014, that affected only small groups of people. But there may have been more incidents that were not revealed to the public. For example, in May 2016, an attack on one of their tax-oriented sites exposed the personal information of 430,000 customers of Kroger, a retail conglomerate. A class action suit was filed, where the lawyers for the plaintiff argued that Equifax, quote, “willfully ignored known weaknesses in its data security, including prior hacks into its information systems.” What were these “prior hacks”?

The Kroger hack would prove prescient to the later history of Equifax, for two main reasons. First: they sought to see the case tossed out of court, on the basis that injury to Kroger customers was merely “speculative and hypothetical” in nature. Second: the cause of this hack was so obvious you’d think anybody with a musical theater degree would’ve known to avoid it. It occurred because Equifax was giving out PINs to employees of client companies, allowing them to access Equifax data. The PINs, however, were not random. They were comprised of the last four digits of the employee’s social security number, and their birth year–information any run-of-the-mill hacker might be able to obtain.

The Kroger suit was eventually dismissed from court, on the condition that Equifax address their glaring PIN security flaw. That did not occur.

Later in 2016 more dire flaws were uncovered, like one September analysis finding that Equifax was running thousands of servers exposed on the internet – the equivalent of having thousands of safes, sitting out in public.

Another security researcher, who spoke anonymously to Motherboard, was able to take control of several Equifax servers in December of the same year, and found many others vulnerable to simple SQL injection attacks. Even more importantly, though, after a few hours of checking out Equifax websites and public servers, they came upon one particularly suspect domain. It appeared to be a portal for employees of the company, but required no authentication to access. The researcher tried “forced browsing”–a method whereby, through brute force guessing of URLs, they were able to access areas of the site not otherwise linked to. The researcher arrived at a page with several search fields which, at the click of a button–and without asking any questions–revealed the personal information of just about any American citizen. Names, birthdays, locations, social security numbers and more, all from this website that was openly accessible on the net, and vulnerable to the kind of attack that might take all of a few minutes. The researcher could have downloaded every American’s personal information in a matter of minutes. Instead, they downloaded data on a hundred-thousand people or so, to demonstrate their findings to Equifax itself.

Equifax could have solved the problem instantly, by taking down the site, but they did not.

Then in February 2017, Equifax became aware that hackers were targeting Talx–the company we mentioned earlier, which holds employment data on half of American workers, by now renamed “Equifax Workforce Solutions”. These hackers had personal identification information for employees of Equifax client companies, and used it to log into those employees’ Workforce Solutions online accounts, in order to file fraudulent returns and steal the tax refunds before anyone were the wiser.

There are things we don’t know about this breach, but what does seem apparent is that Richard Smith took it more seriously than any other before it. He personally oversaw the investigation, hired the FireEye-owned cybersecurity firm Mandiant to help, and classified it as “top secret”. And yet, in only a few weeks, the entire operation broke down. Mandiant warned Equifax about their vulnerable infrastructure–the kind of stuff they already should’ve known about, when they were warned of it months earlier. And yet, for reasons we don’t know, Equifax and Richard Smith came to believe Mandiant’s people were not doing their jobs very well.

Smith’s “Number One Worry”

To look at these three incidents–the obvious PIN numbers, the open and vulnerable infrastructure, and the faulty tax portal–you might assume Equifax didn’t care much for cyber security. But, surprisingly, they did.

Back before all of this, in 2005, soon after taking on the CEO job, Richard Smith hired a well-regarded security expert, Tony Spinelli, as his chief security officer. Spinelli began rewriting the company’s security protocols from the ground up, leading a team in rehearsing various crisis management scenarios.

Something seemed to have changed, however, around 2013, when Spinelli and many in his team began exiting the company. Equifax lost a good chunk of cyber security talent around this time and, in response, hired Susan Mauldin–formerly CSO at a credit card company–to replace Spinelli. Equifax hired some lesser-known talents to fill out Mauldin’s team, but all in all, they seemed to have their heads in the right place. They were strict with the IT department, about filtering any and all activity through security. How they invested millions in expensive new security technologies, and arranged a team dedicated to patching vulnerabilities in their systems.

Richard Smith, in particular, was thinking about information security long before his company experienced any significant breaches. In a speech to the University of Georgia’s Terry College of Business, he described it as his “number one worry”.

It’s interesting, actually, that speech. When Smith calls security his “number one worry,” he’s responding to a question from an audience member. It’s fifteen minutes into the speech, during the Q&A period, when that audience member receives a microphone. Smith is turned to him, standing upright, with a hint of a smile on his face. Then the question is posed. Quote: “Thank you, Richard. Data fraud must be a great concern of yours and everybody in the company.” Just as quickly as the thought finishes Smith drops his head down, and starts fiddling with something on his podium. It’s a moment so subtle, you probably wouldn’t have even noticed it, had you been sitting in that audience that day.

But if you were able to glance into Richard Smith’s head in that moment, you might have known why that momentary head turn, that little fidgeting, reflected something much more dire. Subconscious body language, perhaps a projection of inner anxiety, about knowledge he had in that moment which he could not share with anyone else in the room.

Richard Smith’s speech at UGA occurred on August 17th, 2017. Three weeks earlier, he’d received some very, very bad news. Seventy miles west of where he was speaking now, a top-secret operation was underway at his company, to try and mitigate the single greatest leak of personal information in world history.

Apache Struts 2

Apache Struts 2 is an open-source framework for developing web applications in Java. It’s used by companies all over the world including, for example, two-thirds of Fortune 500 companies. Virgin Atlantic, Citigroup, Office Depot, even the IRS uses Struts. Even knowing all that, though, it’s impossible to imagine that Nike Zheng, a cybersecurity researcher from Shanghai, could ever have realized what might ensue, after he discovered a bug in the program.

The bug was rather technical, but not so complicated to understand. By uploading a particular file type, a hacker could trick a web server hosting a Struts application into allowing them to remotely execute code. With this power, a hacker could upload malware to that server, allowing them to steal information held within.

Now, discovering a bug is one thing, and publishing its discovery is another. Announcing a vulnerability in a popular software program is tricky, because there’s no particular way to tell only the good guys about it. Apache announced the Struts bug only as soon as they had a patch ready for it, on March 8th, 2017. Because the vulnerability could allow hackers remote access and the ability to steal data from any company that used Struts, the warning was heard loudly across the international cyber community–both by the companies that needed to hear it, and the hackers who knew how to exploit it. The race was on.

Now if discovering a bug is one thing, and publishing its discovery is another, actually implementing a patch to Struts 2 was an entirely different matter. Oftentimes updating web applications is a relatively seamless process, but that wasn’t so with Struts 2. As noted by Ars Technica, the ubiquity of Struts meant that the largest websites were often running not just one, but tens or hundreds of separate instances of Struts-supported apps that might be scattered across many servers on more than one continent. One would have to first download the patched version of Struts, then use it to rebuild every single app that used the old, buggy version, and finally test the updated apps to make sure they hold up to scrutiny.

All of the work was worth it, though, because within 24 hours of Apache’s announcement, hackers were already scanning the internet for vulnerable servers running un-patched Struts applications. Before the week was finished, over 500 unique IP addresses were discovered to have been trying to exploit the bug, 300 of which resided in China. Two known exploits were already floating around which, according to researchers, required no authentication, were simple to use, and highly effective.

After Apache’s announcement, the U.S. government’s Computer Emergency Readiness Team circulated a security advisory to American companies. That notice was circulated, in an email, to all those on Equifax’s IT administration mailing list. Nobody seemed to notice, at the time, that the list hadn’t been recently updated to include all systems administrators at the company. Without everybody on board, Equifax’s Struts patch would remain incomplete.

Two days later, on March 10th, an unidentified hacker or hackers scanned Equifax’s servers. What they discovered was a server hosting the company’s customer portal for submitting disputes, running the vulnerable version of Struts. The unknown hackers accessed the server, tested what they could do with it, and then left without making a fuss.

On March 15th, Equifax administrators scanned their own systems for instances of the Struts vulnerability. For whatever reason, that online dispute website did not show up in the search. Still, even if they had found and patched the dispute portal, it wouldn’t have mattered by March 15th. They’d waited too long to check. On March 13th, using the hole they’d found three days prior, the hackers installed a back door to the Equifax server hosting their dispute portal–a web shell, which allowed them remote administration over that server. In other words, even if Equifax had discovered the vulnerability on March 15th, and patched it right away, it would’ve been for naught. The hackers had planted their flag, and were now underway in reaping the company for all it was worth.

Equifax’s Breach

Much of what we know of the Equifax hack comes by way of Moloch–an open-source tool they used in their systems, that stores and indexes a history of network traffic. In other words, Moloch is like a security camera for a computer network. Once investigators were able to look at its recordings, they found that these hackers were not your ordinary, run-of-the-mill cyber criminals, teenage wizkids or lone wolfs. Equifax was hacked by an entire cyber intrusion apparatus.

Beginning March 13th, 2017, a team of actors began infiltrating Equifax’s server through the Struts vulnerability. But they had trouble getting past firewalls, and other security protocols of the system. Then, according to Bloomberg magazine, a second, more advanced team took over the job. The more sophisticated team proceeded to methodically evade firewalls, and learn more and more about Equifax’s network. They broke into database after database, moving laterally through the network, escalating privileges, all the while helped by the fact that Equifax’s infrastructure was not properly segmented.

The key that allowed Equifax’s hackers to move so swiftly was their ability to leverage personal identification information from one location, to access another. A postmortem report published by the Government Accountability Office a year after the incident described how, by using the online dispute portal to query other databases, the hackers obtained unencrypted personal identification and login information that then allowed them to move on to the next point, and the next point, and so on. By the end, they’d installed thirty web shells across various Equifax sites, allowing them to remotely execute the same commands normally restricted only to the highest company admins.

Before long, the attackers had their hands in four dozen Equifax databases, sending around 9,000 queries in order to extract sensitive documents, and long and detailed tables containing just about any information you might want on over 100 million Americans.

The next step, after having hit on all those database tables, was to exfiltrate the data. But this was a sophisticated team of hackers, remember, and they knew that pulling all that data all at once would ring the alarm bells of even the least cyber secure corporation. So, rather than run off with all the spoils, they slowly, gradually siphoned off data, so that it would appear to network administrators like normal traffic.

Meanwhile, the device Equifax used to theoretically keep track of, and alert to such traffic was not able to do so, as a result of a digital certificate which was, by this point, ten months expired. Had the device been properly updated and configured, it would have immediately caught the activity in its tracks. How do I know that? Because as soon as administrators at the company did renew that digital certificate, the device immediately did pick up on the malicious activity. This was July 29th, and the attackers had been operating without any issue for 76 days already. The original point of entry–that dispute portal–was taken down the next day, and the attack was stopped. Within 24 hours. Two and a half months too late.

Coming up Malicious Life, I’ll tell you about the fallout of the Equifax hack: how the company changed their ways for the better, those responsible got punished, and everybody’s personal information became safe once again.

Just kidding! In Part Two we’ve got insider trading, corrupt government officials, really terrible cyber security, and how half of the U.S. population is now screwed for life! Have some chamomile tea, or a warm bath, before tuning in next time to Malicious Life.

Link To Part II