How the 60-Year-Old IRS Computer System Failed on Tax Day.
April 17.
In 2018, this was Tax Day. It’s the deadline for American taxpayers to submit their income tax returns to the federal government. Yet on Tax Day 2018, the IRS electronic filing system was down. Millions were attempting to e-File on that day alone and nearly every one of them were denied. Turns out the IRS's computers were still partially relying on old assembly code meant for mid-20th century IBM mainframes, and a buggy update had caused their modern systems to come crashing down. What happened? This is LGR Tech Tales, where we take a look at noteworthy stories of technological inspiration, failure, and everything in-between. This episode tells the tale of the 2018 Tax Day outage and the cold war-era technology that contributed to it.
In 1959, the Internal Revenue Service had some big problems.
260 million of them, actually. That was the number of tax documents they had to check from 1958, with over 60 million individuals and 975,000 corporations all submitting their tax returns that year. Even though the IRS employed 50,000 people, checking 60 million documents by hand was unfeasible, much less the millions of tax audits on top of that. And that’s where International Business Machines came in, with the latest and greatest of their so-called “brain machines,” “electronic brains,” or simply “computers.” And in 1959, the new hotness were these gigantic, room-filling mainframes from the IBM 700 and 7000 series. Specifically, the IRS made use of the IBM 7074 starting in the early 1960s, capable of calculating word lengths of up to ten digits in addition to a sign, with a total memory capacity ranging from 10 to 20 kilobytes in its standard configuration. Now, the entire tax return and history of each American citizen could fit onto a single four-inch long, half-inch wide strip of magnetic tape, accessible on equipment costing the IRS just $4,000 in daily rental fees. Or $33,150 per day adjusted for inflation.
By 1966, the IRS had gone onto invest in enough IBM hardware that the entire country’s taxes could be checked by computer every year. And their latest machine was an absolute monster: The Martinsburg Monster, as it was known, both for its size and its location in Martinsburg, West Virginia. And also for its reputation as Uncle Sam’s cold, calculating tax monster, expected to catch billions of dollars in unreported income. In reality, the monster was an IBM System/360, one of a family of mainframes that IBM delivered to governments, businesses, universities, and anyone who had a few hundred thousand dollars from 1965 to 1978. And for the time this was absolutely state of the art, thanks to an impressive tape storage solution: the Individual Master File. The IMF was stored on a couple thousand reels of tape and held the data for each individual taxpayer, corporation, legal entity, and whatever else the IRS needed. When an employee needed to look something up, the appropriate tape could be loaded using the Martinsburg Monster and checked against a selection of punch cards to verify everything from an individual’s name, address, money owed, marital status, tax credits, deductions, and so on. And for a while this was rather disconcerting to the general public, with stories in the news describing it with such phrases as “un-American,” “Orwellian,” “an ultimate weapon,” and “frightening.” - There are those at Internal Revenue who say that if this building in Martinsburg, West Virginia were filled with hay instead of computers it still would put the fear of god in us all so long as it said National Computer Center on the outside. This is where we are at, the end of the line for all of us, lined up together on the shelves of internal revenues National Computer Center in Martinsburg, West Virginia. The Martinsburg Monster. The very idea of computer checking returns conjures up images of frightening efficiency, which guarantee either that you can't cheat or that the other fellow can't cheat, depending on how you look at it. - Of course, the US government did what the US government does and released a series of short propaganda films to try and explain exactly what the Monster was doing, and how the computer wasn’t evil at all, but merely super good at its job. - This is the real heart of the Martinsburg Monster. Nearly everyone in the United States has some concern with this mechanical marvel and its electronic relatives. Electronic computers have made it possible for IRS to process millions more returns than would have been possible by hand. Returns go through a carefully planned cycle of processing operations. When information from each return is later transferred to the punch cards, the machines of course check the math. - Whether or not the public embraced this new faceless overlord though, it didn’t really matter. What the federal government really cared about was its proficiency doling out refunds and catching more tax cheaters than humans could. And considering that by 1968 the IRS was bringing in an additional 25 billion dollars in taxes year over year, the government’s continued investment in tax computing was justified. By the mid 1970s, there were mainframes, terminals, and now minicomputer systems in all of the most important regional IRS centers across the country. Allowing them to collect an additional 5 billion dollars annually by 1977 due to the larger number of successful audits, with computerized detection having become so accurate that the odds of an audited taxpayer losing a case were 4 to 1. Meaning that you’d have a better chance succeeding at a game of Russian Roulette than against the Martinsburg Monster. By the 1980s, the microcomputer revolution was in full swing, with machines that packed unprecedented performance in a form factor that fit onto a desktop. Not only that, but modem usage was increasing alongside them, allowing microcomputer users to dial into far-off computer systems remotely. Tech-savvy tax practitioners in particular got an early jump onto the personal computer bandwagon in order to help them prepare taxes, even if the IRS still required them to print out and physically mail in each return. It wasn’t until the mid-80s that the IRS Research Division started experimenting with a new Electronic Filing System. The first e-filing in the US happened in 1986, performed by just five tax preparers in three metropolitan areas: Cincinnati, Phoenix, and Raleigh-Durham. The process went like this: a tax preparer would use their personal computer’s modem to dial into the main IRS Cincinnati Service Center, and an IRS employee would pick up the call, plug the phone line into a Mitron magnetic tape terminal, and the completed digital tax return was received. It was then processed by the IRS on a minicomputer system using a newly-created e-File program written using COBOL. This final step was one of the trickier parts of the process, requiring the assistance of retired programmers to develop software which could interface with the aging IRS computer systems from the 60s and 70s, which also ran COBOL.
After 25,000 digital tax returns were processed successfully in 1986, the e-File system was deemed a success and work began on expanding its usage. Nationwide e-filing commenced in 1990, and although the system still only allowed returns that were due a refund, 4.2 million of them were electronically filed that year.
In 1999 electronic payments through credit and debit cards were introduced, along with the ability to sign returns electronically instead of by mail. The e-File system continued to expand alongside the explosive growth of the internet in the 2000s, with Free File and Modernized e-File debuting in 2003 and 2004, respectively. This resulted in a record 68.4 million returns filed electronically in 2005, and in 2011, e-filed returns crossed 100 million that tax season, meaning that approximately three out of every four US tax returns were now filed electronically. With all this in mind then: what happened in 2018 that caused the whole system to fail? Are they still using all those old mainframes and minicomputers running COBOL or what? Well, in a word, yes. Kind of. While the old IBM machines themselves are long gone, the actual programs they ran remained in use throughout the emergence of the modern e-filing system. And as of the making of this video, the IRS still relies on those old programs written in COBOL and IBM assembly languages. Remember the Individual Master File, the giant database of all individual taxpayers? Well, the IMF is still relied upon to reference all that taxpayer data, meaning that even though the computer hardware itself has been upgraded, they still have to emulate the computer systems from the Kennedy Administration. Not only that, but every time the US tax code changes, the old programs have to be updated, and this has caused a massive headache for the IRS. According to a Government Accountability Office report in 2016, some 20 million lines of code are still used that date back to the creation of the IMF in the 1960s. And this ancient programming is only going to result in greater challenges as time marches on, something the government has been aware of for decades. But despite ongoing efforts at modernization and hundreds of millions spent since the late 90s, the Individual Master File remains begrudgingly in use. And there were copious problems with its planned replacement, the Customer Account Data Engine. Despite lofty ambitions and nearly half a billion in funding, CADE was only ever used as a hybrid system tied to the old master file, delivering on only 15 percent of its promised capabilities before being canceled in 2009. And this brings us to the outage of April 17, 2018. According to a Treasury Inspector General report later that year, a known firmware bug caused a Tier 1 high-availability storage array to fail at around 3 AM on Tax Day. This was an 18-month-old piece of hardware installed to support the Individual Master File. That morning it detected a deadlock condition after a warmstart due to cache overflow, causing 59 systems in total to fail. Since almost all other IRS services and systems ingest data from the IMF mainframe, they too failed, and e-File systems were offline for 11 hours during Tax Day. It was later determined that buggy firmware on their IBM hardware was to blame, something that IBM had known about since June of 2017. And IBM released a firmware update fixing the bug that November, five months before Tax Day. But after a December 2017 meeting between them and Unisys, the agency’s storage contractor, the IRS decided not to update on the advice of Unisys, since the older firmware was thought to be more stable in supporting the all-important Individual Master File. And so, millions of people were freaking out all over the United States, unable to pay their taxes. And droves of IT specialists were freaking out in government computer centers, scrambling to get everything back online while the higher-ups breathed down their necks. All over a bad piece of firmware on a storage device holding a bunch of primordial software programmed before the moon landing. While there are currently plans to replace the Individual Master File with a new system, CADE 2, that still hasn’t happened either. Will the system continue to hold together until the IMF is finally replaced? That remains to be seen, but until then, I hope you’ve enjoyed this episode of LGR Tech Tales! Never thought I’d do a video about taxes, but whatever, I just like computers, stories about computers, and using computers to make videos about computers. If ya like this kinda thing then stick around, I’ve got new videos every week right here on LGR. And as always, thank you very much for watching!
