How the Internet Changed the NSA [ML BSide]

Transcription edited by Suki T

[Ran] Hi and welcome to Cybereason’s Malicious Life B-Sides, I’m Ran Levy. Over the years, we’ve dedicated quite a few episodes of Malicious Life to the NSA, because well, you’ve gotta admit, love it or hate it, it’s a very interesting organization whose influence on cyber security is huge. Eternal Blue, Prism, TAO, these are probably well-known names for any security professional. We’ve covered the NSA’s early history in an episode titled Super Spies, the NSA’s Cold War shenanigans in the world’s first keylogger, and a few other episodes, and in this B-side interview we’ll be covering an extremely important period in the NSA’s history, yet one that I think is rarely talked about. The period in the late 80s to the early 2000s, when the National Security Agency transitioned from being hardware-based, that is, creating and operating physical spying devices, to being mainly software-based, excelling in hacking networks, tracking people online, etc. As you’ll shortly hear, that transition was by no means easy. The NSA, by that point, was a big organization, and big organizations are notorious for being very resistant to change.
Jeff Mann, our guest today, was one of the first people at the NSA to make the transition from hardware to software. In fact, he designed the NSA’s very first encryption software. Jeff spoke with Nate Nelson, our Senior Producer, about his days in the secretive organization, about how the NSA’s public image changed over the years, and the frustrations of dealing with the never-ending government bureaucracy. Enjoy the episode.

[Nate] I am here with Jeff Mann. Jeff, if you could start off by introducing yourself.

[Jeff] Sure, my name is Jeff Mann. I am what’s considered a curmudgeon in this industry. I’ve been doing information security since the early 80s, and I’ve been working in the infosec field in the commercial world for over 20 years now. I spent the first part of my career working for the government, which is why I think you want to talk to me today.

[Nate] Correct. So, to start off, you first arrived at the NSA back in the mid-80s. Take us back to that time. How did it feel back then inside those walls? What was the work environment like? What kinds of people did you work with, and what was the general energy of the place?

[Jeff] Great question. Back in the mid-80s, I went to work for NSA during the Reagan administration, where we still had an adversary that we recognized and understood in the traditional sense, which was the Soviet Union. So, most of the defense budget for the DoD, and in particular for NSA, was devoted to fighting what was known as the Cold War. There’s a lot of espionage involved, a lot of collecting information about the adversary, which is a lot of what NSA is known for, and there was sort of an unlimited budget, if you will, for accomplishing that mission.
When I started at NSA, the Cold War was winding down. The Soviet Union was dissolved in the late 80s. The Berlin Wall came down. These are things that you can learn about in the history books or by Googling it. So, NSA very quickly was in a time of transition, and the transition was not only because the world stage was changing, but also I started at the very beginning of the age of computers and the age of the internet. And where I ended up, which kind of set my career path in a lot of ways, was on what was known at that time as communication security, or ComSec, or very quickly after that became known as Infosec, information security, which was, for lack of a better term, the defensive side of the house. NSA, what NSA was and is known for is collecting communications and information, traffic signals from our adversaries and the rest of the world, called that offense. And then there was a much smaller portion of NSA that was devoted to providing secure communications and secure communications equipment and capabilities to our customer, which was primarily the defense department, all the military services. So, the defensive side, if you will.
Where I went to work on my first assignment was called the Manual Crypto Systems Shop. It was the shop that was responsible for putting out all the paper systems, which was mostly something called a one-off system. It was called a one-time pad. One-time pads, ironically, are a secure form of encrypted communication. It’s not mathematically or cryptographically breakable as long as you keep the key secret. And the key is a paper notepad. It’s usually the size of an index card that has a series of pages, and on each page is written down random characters, which is essentially the key that’s used in a very manually intensive cryptographic system where you write down your message one character at a time. You apply some sort of substitution algorithm between the plaintext character and the key character to produce a third character, which is called cipher. And then that is sent. The message is collected and sent. And the person on the other end has essentially the same copy of the one-time pad with the same random key printed on the page. And you write down the cipher, one character at a time, matched up with the key, and you apply the same algorithm, and you get back to plaintext. It cannot be broken as long as the key is protected. So I was assigned to the manual cryptosystems shop because they had decided that the manual cryptosystems that were still being used by a lot of the military needed an evaluation, a refresh to make sure that they were still cryptographically sound. It wasn’t all one-time pads, there was other things involved.
And looking back on it now, it’s like my whole career at NSA was sort of somewhere in the wake of or in the midst of the past meets the future, which I think is a lot of what NSA struggles with even to this day and some of the current debates that are going on, like the idea of being able to have backdoors in communications devices, mobile devices, and having the secret backdoor keys so that NSA can do, or other law enforcement entities can do what they do. It all has to do with sort of this crossroads between the past and the legacy and the technological capabilities of the future.

[AD] The attack surface has never been larger or more diverse, yet defenders are still forced to piece together intelligence from numerous siloed solutions that produce a flood of alerts in order to detect and end complex malicious operations. No more. Defenders can now leverage AI-driven Cybereason XDR powered by Google Chronicle to predict, understand, and end sophisticated attacks with the only solution on the market that delivers planetary-scale protection that allows them to predict attacker behavior through a revolutionary, operation-centric detection and response approach. Cybereason and Google Cloud are dedicated to teaming with defenders to end cyber attacks from endpoints to the enterprise to everywhere. Learn more about Cybereason XDR powered by Google Chronicle at cybereason.com/platform/XDR.

[Nate] I’d like to ask about the reputation of the NSA. As you may well know, the NSA has this reputation for being a sort of big brother organization, a black box that does an uncomfortable amount of spying on innocent people. You worked in that black box. How does that general perception of the NSA vibe with your own personal experience there?

[Jeff] I think that the perception that you speak of has largely come about over the last couple years. A lot of this perception of NSA comes from the fallout of the whole Edward Snowden thing. In fact, one of the reasons why I like to tell stories about NSA and my experience of NSA is to just sort of put context on some of the conversation about the reputation of NSA based on experiences that I’ve had. Case in point, the idea that NSA, how did you phrase it, that they’re big brother and doing stuff to innocents. Well, NSA has a charter. NSA is bound by law, but what was hammered into me from day one as part of our mission was that we had a charter and the charter stated that NSA does, I’m paraphrasing, that NSA does what NSA does to foreign entities, to our adversaries. We do not do what NSA does to US citizens. The perspective of doing it to the innocents where that’s gotten blurred a little bit with the whole Snowden thing has a lot to do with 9-11, has a lot to do with the laws that were passed subsequent to 9-11 from the Patriot Act on downward.
In my experience at NSA, NSA takes the law very seriously and takes the charter very seriously and goes to great lengths. I have personal experiences that I won’t go into a whole lot of detail here today, but I have personal experiences of how much NSA cares about following the law and how diligent they are almost to a fault of taking a very historically conservative approach to even the interpretation of the laws and the charters. If I may, a little history. Based on some of my experiences and also on the whole Snowden revelations, the charter that I speak of largely came about, it goes back to the time of the Watergate break-in. After the Watergate break-in, there was a Senate committee formed that ended up being called the Church Committee. Senator Frank Church was the chair of that committee. They investigated what were the laws and what were the checks and balances that were in place for the nation’s three-letter agencies, in particular NSA, FBI, and the CIA. They produced volumes of reports, which I believe are mostly available on the internet, so you can find them.
Essentially, what they discovered was all these three-letter agencies wield a whole lot of power to do a whole lot of evil and bad things to anybody, and they really don’t have any checks and balances on paper in terms of what they’re allowed to do legally, by charter, by law, or by direction. The outcome of the Church proceedings was, among other things, the thing that I know is the charter that became the mantra that I lived by at NSA, which is NSA does not do what NSA does to U.S. citizens. That all came as a result of these Church proceedings that were part of the fallout of the Watergate break-in back in the 70s. I bring that up in terms of Snowden because there’s only two times in my life that I’ve ever heard about the Church proceedings. One was when I was getting in trouble at NSA for supposedly breaking the charter, which had something to do with penetration tests and red teaming, which we’re going to talk about in a little while, but the second time I heard about it was in the context of when Snowden did his thing and hearing news reporting about the Church proceedings. It’s a not well-known part of history, but it is certainly a part of history that impacted NSA very directly and NSA takes very seriously. To round out and try to answer your question, NSA, by charter, by their mission, does not necessarily do what they do to quote-unquote, innocence. They’re doing it to foreign powers. They’re doing it to who we consider our enemy or our adversary or countries and nation states that we are concerned about, all in the name of national defense and national security.
The lines are blurred somewhat more these days in terms of who is our enemy, when our enemy might be among us, when our enemy might be on our soil, and of course, with the technological advances where the idea of borders become kind of blurred, extremely blurred in terms of the internet. But if you take the time to investigate and study, the laws are still there. The laws are still in place. And what I think Snowden did, which I think is a fair debate that we should have in the public world, is the debate of whether NSA should be doing these types of things or not. But I think what people often jump to is, oh, NSA is doing something illegal. NSA is going beyond its boundaries. Again, in my experience, that’s not what NSA does. Question what NSA is doing, but not whether it’s legal or not. Look at the laws and the charters and the things that are binding them, which are congressional acts primarily, and have a debate in the public forum about whether those laws should be there or not and whether that direction should be given to NSA. I think that’s more where the discourse should be.

[Nate] Could you tell our listeners about some of the more notable projects you worked on for the NSA?

[Jeff] You know, it’s funny, as I look back on my career, probably the thing that I’m most proud of, and I actually thought it at the time, wow, this is the coolest thing that I’ll probably ever do in my whole career, is something that I did in the first year, year and a half that I was there. So I’m in the manual cryptosystem shop, you know, and I’m dealing with one-time pads and providing one-time pads to our customers who are primarily military, the DoD and other spooky people. And you can imagine that one-time pads are used for secret communications. And we were approached by a customer one day that said, hey, you know, we’re the guys that are in the offices and the controlled spaces. We’re talking to the spies and the people we’ve recruited in the field. And, you know, maybe they have to do, you know, sneak off into dark shadows and use little miniature versions of the one-time pads that they can hide in the heel of their shoe and things like that. But we had the large print edition, you know, in fact, their one-time pad was more like a legal pad size. And even then, it would take them hours to do the encryption and the decryption of the messages and communications that were going back and forth between the people that they were working with.
And they asked us, there’s this thing on our desk and, you know, happened to be an IBM PC. Is there any way we can take advantage of that thing and just speed up or automate this very manually intensive process? And me being young and naive and kind of new to all of the business said, yeah, that seems reasonable. So I ended up becoming a project manager, design manager for producing a one-time pad that ended up being on floppy disks, rather on the paper pad. And I went and found a development group and got somebody that can actually write the code that would do the encryption and the decryption algorithm. And essentially, I had to in a hardware shop and NSA, I should say at that time, historically only produced little black boxes. In fact, one of my early bosses, in fact, he was a chief scientist, said one time, you know, there really is no such thing as software, there’s only hardware. That was the mindset of NSA at the time, again, in a time where technology was changing.
So I went off and found requirement specifications for building secure little black boxes, essentially. It was a hardware design standard. And I had to essentially try to meet the spirit of all these design requirements, but do it in software instead of hardware, which had never been done before. And I had to come up with the design and I had to go before basically the C level C suite of managers on the InfoSec side of the house. And, you know, I’ve only been there for a few months and I’m meeting all these people that are much older suits, as we used to call them, upper level management that were primarily engineers, they’ve been there 20, 30 years. And I’m this young guy saying, Hey, we’re going to do something cool and different. And we’re kind of rewriting the rules. And I had to present that and they gave me the green light to proceed with the project. And so we had to go through the design project, build it, and then go back and get it a final approval after it had been built.
And of course, part of that was going through a security evaluation. There was a group that would kick the tires on it and try to find problems and holes and they had issues with it. And I responded to all the issues. And ultimately, the suite of C level managers begrudgingly said, Okay, we’ll let you feel it, but don’t ever do this again. And looking back to my knowledge, and I’ve asked a lot of people that I’m in touch with that used to work at NSA in that group, that was the first software based crypto system that NSA ever produced. And all it was, was taking a paper one time pad. And instead of printing the key on paper, we put it on a floppy disk and wrote a program to run it on a PC to do the encryption and the decryption and to delete or erase each page of key as it was being used.
And that was it. It wasn’t a network system. I would never claim that it was robust and secure if it was on a network device. But for the time, it was revolutionary. And I didn’t even know at the time how revolutionary it was. It’s looking back years later, like, wow, I think that was probably the first software based system, crypto system that NSA ever produced. So that’s, I hang my hat on that nobody’s ever heard of it. I ran into somebody a few years ago at a conference that was actually somebody that had used the system. So, you know, and they knew about the system, they remembered it. And in conversation, it came out that I was the one that had produced it. So that was kind of cool to just get feedback from somebody in the field that had actually used it and benefited from automation and very welcomed automation at the time.

[Nate] How did the NSA first get into the business of penetration testing? And why?

[Jeff] I started at NSA in 1986. If anybody’s ever seen the old movie War Games, I believe that came out in 83. So the idea of hacking computers, and in those days, it was, you know, mainframes primarily had been around since probably the 70s. And there’s probably arguments that it goes back even further. So I mentioned, you know, when I was building my device, my system, that it was going through an evaluation looking for security holes. I mean, that was very much a part of the defensive side that built little black boxes and built crypto systems and secure communications and data handling systems for our customers. And those systems had to be protected and those systems had to be secure for sometimes 10, 20, 30 years.
So there was whole groups that would do security evaluations. You know, somewhere in my career, and I won’t bore you with the details, but I hopped around jobs a little bit, but I ended up in an office, again, on the InfoSec side, that was called Fielded Systems Evaluations. And what this group was designed to do was basically do what we did on the offensive side of NSA, you know, in terms of breaking codes and intercepting communications and figuring out how to read them. Somebody had made the realization that very often the ways that we were able to compromise or gain access to our adversaries communications was by taking advantage of the fact that the systems weren’t always used correctly or implemented correctly. Default settings weren’t changed. The default key settings weren’t changed or key settings were, you know, like a one-time pad is only supposed to be used one time. That’s where it’s secure. But if you use the same page for 30 days and countless messages, you’re introducing cryptographic vulnerabilities that make it so that you can break it.
So this office had the focus of not only basically exclusively looking at our own stuff, because we produced a lot of our own stuff because we produced the greatest stuff ever. NSA prided itself on producing little black boxes where messages went in and gobbledygook code came out and it couldn’t be broken. But we asked the question as an organization, how do we know our customers in the field are using them correctly and not finding workarounds or shortcuts? Because a lot of times this equipment is used by 18, 19, 20 year old kids that are designed for the communications role and everybody likes to find shortcuts. So one of the groups within that office fielded systems evaluations was called network systems. And that’s where I kind of gravitated. And this was in the early 90s. And that’s where I got together with a bunch of guys and we started focusing on, you know, if we’re looking at network systems, we really need to learn the craft or the trade craft of becoming a hacker.
So we started learning about what it means to be a hacker and how to break into, in those days, it was primarily Unix operating systems. And we started learning all the tricks and the techniques and how to exploit the features of the Unix operating systems in those days. And in terms of networking, the protocols that were commonly used for networking devices together, we started learning all that. And there was, you know, we were just a bunch of guys and one little branch of one little division within InfoSec. But what was also happening concurrently at a higher level, you know, at a management level and even higher levels of the government was, you know, there was a recognition that there was this growing need to try to address networking security and more and more companies and organizations were jumping on the internet.
You know, the World Wide Web came out and everybody started getting on the internet. So the world was changing. So in terms of formally organizing what happened, and I forget the exact date, I want to say it was 94-ish, 95-ish time frame, there was a reorganization and upper level management decided to open up a Center of Excellence that would be focused on primarily network and internet security, what we would call today cyber security. And that organization was called the Systems and Network Attack Center. We were pulled, you know, our field and systems evaluation group was sort of sliced and diced and put into this new center of excellence. And one of the offices of that Center of Excellence was the guys that I worked with and we were primarily doing pentesting and we called it Vulnerability and Threat Assessment back then. And I don’t think we hadn’t certainly coined the term red teaming, but we were doing the ethical hacking, you know, breaking in testing the security of your systems and networks by breaking into them.
We got pulled into a different building in a different location and into our own office, whereas before we were part of a larger group, we got our own office space. And we decided to continue to build on learning about how to be a hacker and learning sort of the way of the hacker. We were trying to live the hacker culture to some degree. So one of the things we did was we came up with a nickname for our office. And the nickname for our office was the Pit. So the Pit somewhere along the line has become part of the folklore of NSA and pentesting, but really at its origin, it was a bunch of guys that were dedicated to learning about how to hack systems and break into systems that got reorganized and put into a group. And so, you know, despite the folklore of the Pit, what it really was, was a bunch of guys that got together that were doing hacking and learning how to do hacking skills and learning how to break into systems. And it was simply the office that we were in. We had desks, we had cubicles, we had a conference table. And that is it. And we had a lot of Mountain Dew, by the way.

[Nate] And how sure were you guys at the time during your most ambitious Red Teaming projects that what you were doing was legal?

[Jeff] It’s an interesting question because, you know, we understood because very quickly we had to get the lawyers involved, the general counsel involved in what we were doing, because it was hammered into us, like I said, that NSA doesn’t do what NSA does to to US citizens. And so there was this sort of very quick and obvious recognition that, you know, breaking into computers and networks, even though it was being done for good reasons, was technically a violation of our charter. What we came to learn was that there’s a way to do that legally or following the rules. And that involved a lot of bureaucracy, it involved a lot of management oversight, it involved a lot of people signing off on the permission to do these types of things, which in a bureaucracy could take weeks, if not months, because we would have to generate a plan and put it down on paper.
You know, customer X, whoever customer X might be, wants us to do a vulnerability and threat assessment. Here’s their environment. Here’s what we think we’re going to do. They wanted us to write down sort of our whole attack methodology, even though we didn’t know what we were going to encounter and, you know, sort of come up with the rules of engagement. You know, what’s what are the targets? What are the goals? Write that down and then it would have to go off for weeks and months to get all sorts of levels of management and the lawyers to review it and sign off on it and approve it on a management chain to the deputy director of the information security group. So our frustration was very often, you know, we knew that what we wanted to do in terms of breaking into something would take very little time just because of the nature of how the networks and the operating systems were configured back in those days, but we would sometimes have to wait weeks and months to actually execute our attacks against the targets.
I’ll give you two examples. One example is that all the things that we were learning about how to break into things, the techniques and the exploits, if you will, very often we were pulling them down off the internet. We were just doing research. We didn’t have Google back then. There was other search engines. We were doing the research online primarily trying to find out the features and the unpublished features primarily of a lot of the protocols and services that were involved in networking and learning how to break into systems. So when we would touch something because we were NSA, our lawyers and our management wanted to label everything top secret because we touched it, which was a constant source of frustration because a lot of what we were doing was what would be considered open source and, you know, everybody knew about it. But as soon as we touched it and what they were trying to protect was NSA had any association or used it or had this technique, they wanted it labeled top secret. So that was one example.
The second example was in the early days when we were trying to explain primarily to our management and then also to the general counsel that it’s not so simple to like have a customer X wanting us to try to break in and be able to tell them ahead of time exactly what we were going to do. Penetration testing, there’s a methodology involved that starts with reconnaissance. You have to figure out what you’re up against. You don’t know what your targets are until you acquire the targets. So the very first thing that we would do was we would want to issue the ping commands and just see if there was maybe a class C IP address range. We would do a ping sweep to see how many systems were on the IP addresses, the range that we were given. The lawyers, when they first heard about that, they wanted to classify the ping as a top secret active attack because it elicited a response from the target.
And that was some of the archaic rules that they went by. But we were arguing, well, no, the ping command, you know, comes with the operating system. It’s a networking tool. You know, it’s something that network administrators and system administrators use to administer the network and identify devices so they can troubleshoot them. It comes with the operating system, but they wanted it labeled. They wanted it classified, top secret. They wanted it labeled as an active attack. And that’s the kind of, and I’m not pointing that out to say that they were dumb, even though we might’ve thought that at the time, but that was the mindset of the management at the time coming from a black box world, not necessarily embracing this new technology of networking and software and the way things were different in terms of the internet and everything was designed for information sharing.
So a lot of what I ended up doing in my role in the Pit became educating the management and the general counsel on and sort of demystifying and just breaking down that we’ll know this is what these things are. These are functions. These are applications. These are services. They’re part of the operating system. This is all pretty normal stuff. And we just know how to use them. It may be in a different way, or we know features of these different tools and services that are readily available that help us along our path. And then there’s this whole other category of things. Once you get somewhere, what do you do in terms of exploit and actually trying to break things or exploit things.
So I ended up doing what I used to call tool time. I would meet regularly with the general counsel and would just kind of walk through the sort of network security 101 and hacking 101. This is the methodology. This is the tools that you use. This is the techniques, but very much focused on methodology rather than specific things that we do in terms of using a particular exploits or after using a particular technique. It was very much sort of, this is the methodology of how we approach. Here’s a target. Here’s a company that says, you’re trying to break into us.

[Nate] My final question is, how did the NSA evolve from your first years working there through the Gulf War in the 90s up until you left in 96? Did anything change about the nature of the organization, particularly in the domain of info ops?

[Jeff] At least in the context of when I was there, you know, it was certainly a time of transition. There was change going on whether they liked it or not. When I first started there, you weren’t allowed to say that you worked at NSA. By the time I left, you were allowed to say that. You know, there was a time, I believe when it was, I was still there. It might have been after I left that the highway that’s right outside of Fort Meade put up signs, exit signs to the ramps that went to the entrance to NSA. And it said NSA entrance exit here. And that freaked out a lot of old timers that lived in the anonymity.
The whole idea that InfoSec had evolved from ComSec communication security to information security to information assurance just in the 10 years that I was there and the implications of what changing the names meant. The whole idea that NSA from on a business side as a producer of ComSec communication and crypto equipment was struggling to keep up with the pace of technology and wanting to continue to build, as Minihan said, hardware solutions where we were migrating to a software based and internet based world really struggled to keep up. Doesn’t mean there weren’t people that saw the light and wanted to move things forward, but in any large bureaucracy trying to change sort of the the foundations of how things are done, it is hard to do. It’s hard to steer a large ship with a very small rudder type of thing.
So change was very small. And I learned from the briefing that I watched the panel discussion of eligible receiver just getting a little bit of a vision of what they were talking about at the highest levels of management from the director and above, you know, from the joint forces about the vision that was going on at that level, based on where I was sitting. And I was just a very small player in a very large institution, you know, even being a member of the Pit and our view of the world and what we thought was important was very different from what I was learning that the higher level people were talking about.
So NSA, I think was in a lot of ways, at least on the infosec side, was fatally ill and had a very, very long, painful and arduous death. Not that I wished death upon that, that part of the organization, but it seemed to be inevitable, I guess. And there’s still people that argue that it shouldn’t happen the way it did, that there should still be that function within NSA, but that’s not where we are today. You know, the function of cybersecurity is more of a joint operation. It’s moved outside of NSA.

[Nate] Great. Have we left anything out?

[Jeff] Only the stuff I can’t talk about.

[Nate] Okay, great. In that case, Jeff, thank you for speaking with me.

[Jeff] Sure. Appreciate it. I hope, again, my goal is not to change people’s minds about NSA. If you want to believe they’re the bad guy and big brother, that’s fine. I just try to bring a different perspective based on my experience to get maybe people to do a little bit more research, dig a little bit deeper, or at least try to understand the various points of view and different forces that are in play that result in what we perceive as what NSA does or doesn’t do. There’s almost always more to the story than what you’re reading about in whatever you’re looking at.