The NSA has many means at its disposal. But how does it use these means, and for what goals? Discover the (literally) secret history of the NSA's cyber activity. Travel with us to Russia and back, learn the origins of FISA, and find out more about the government agency so secret it was once dubbed "No Such Agency".

Hosted By

Ran Levi

Born in Israel in 1975, Ran studied Electrical Engineering at the Technion Institute of Technology, and worked as an electronics engineer and programmer for several High Tech companies in Israel.
In 2007, created the popular Israeli podcast, Making History, with over 10 million downloads as of Aug. 2017.
Author of 3 books (all in Hebrew): Perpetuum Mobile: About the history of Perpetual Motion Machines; The Little University of Science: A book about all of Science (well, the important bits, anyway) in bite-sized chunks; Battle of Minds: About the history of computer malware.

Episode transcript:

[DRAMATIC READ] “Always eyes watching you and the voice enveloping you.  Asleep or awake, indoors or out of doors, in the bath or bed–no escape.  Nothing was your own except the few cubic centimeters in your skull.”

After Edward Snowden’s leaks of National Security Agency materials in 2013, sales of George Orwell’s famous novel 1984–about a tyrannical world government run by a secret, all-powerful organization watching everyone at all times–jumped 6,000 percent overnight.  This should give you a sense of popular opinion of the NSA in the world today.  

Hello and welcome to Malicious Life, I am Ran Levi.

The National Security Agency (NSA) is one of the most secretive intelligence agencies of the United States. For decades, the NSA has been wrapped in shadows, but in recent years more and more floodlights have been directed at it. Initially it was Edward Snowden’s leaked documents that revealed some of the organization’s methods, and later it was the WannaCry ransomware, which spread through an exploit called EnternalBlue, allegedly leaked from the NSA’s cyber weapons repository.

As with many American institutions, the story of the NSA can be traced back to World War II.  With advances in communications technologies, a race for intelligence began between the warring Ally and Axis powers, where the ability to intercept your enemy’s chatter became even more useful to war efforts than strong soldiers and big guns.  The U.S. learned its lesson and, after the war, took to centralizing and improving its cryptographic intelligence forces.

Seven years after the war, three years after the publication of 1984, the U.S. National Security Agency was created, at the bottom of the fourth page of a top secret memorandum from President Harry Truman.  The agency formed in secret would operate in secret for years thereafter: government officials even playfully gave the NSA its first nickname–“No Such Agency”–as it was even kept largely out of sight within Washington.

In Truman’s words, the “mission of the National Security Agency shall be to provide an effective, unified organization and control of the communications intelligence activities of the United States conducted against foreign governments”.  At least in the beginning, it would remain truthful to its stated intention: the NSA provided key signal intelligence during the Cold War, for example, which was fought just about entirely on intel.  They’d also had specific, pointed successes elsewhere, such as an unheeded prediction of the North Vietnamese attack on Saigon which, ultimately, sealed the Vietnam War.  The effectiveness of the NSA would come to save it later, when the agency first came under fire for spying on American citizens.

Over two decades after its formation, the public would finally come to learn of the NSA through the testimony of Lieutenant General Lew Allen, NSA Director at the time, before the Senate Select Committee to Study Governmental Operations with Respect to Intelligence Activities, known more commonly as the Church Committee.  Senator Frank Church, its namesake, opened with remarks that would set the tone not just of these hearings, but the conversation over the NSA for decades to come, when he said this:

“Just as the NSA is one of the largest, and least known of the intelligence agencies, it is also the most reticent. While it sweeps in messages from around the world, it gives out precious little information about itself. Even the legal basis for the activities of NSA is different from other intelligence agencies. No statute establishes the NSA or defines the permissible scope of its responsibilities. Rather, Executive directives make up the sole “charter” for the Agency. Furthermore. these directives fail to define precisely what constitutes the “technical and intelligence information” which the NSA is authorized to collect.”

Director Allen would go on to give a brief history of NSA operations, including how President Richard Nixon’s war on drugs led to their first instances of spying on Americans.  He claimed that “the one instance in which foreign messages were intercepted for specific watchlist purposes was the collection of some telephone calls passed over international communications facilities between the United States and South America,” but ultimately admitted, under pointed questioning, that “there isn’t any statute that prohibits [NSA] interception of domestic communications.”

Lew Allen also revealed a program specifically for spying on American anti-Vietnam War protest leaders, codenamed Project MINARET.  The idea was to track possible connections between influential Americans and foreign criminal entities, though many of the individuals included on these watchlists might not pass the average person’s smell test: among them, Martin Luther King Jr., Muhammad Ali, Jane Fonda, journalists from The New York Times and The Washington Post, and two U.S. Senators, including Frank Church.

So Congress was faced with a conundrum: Allen revealed both the effectiveness of the NSA–in providing crucial wartime intelligence, helping in the war on drugs, even stopping terrorist plots–and the questionable practices that resulted from its lack of oversight, mainly its spying on American citizens.

Congress proceeded to, basically, side with the NSA.  In 1978, three years after Allen’s testimony, they passed legislation that set up the FISA court system, which would systematize the NSA’s spying by providing warrants for wiretapping.  Since its founding, though, FISA has been criticized for being too loose with the reigns–in 2015, for example, a Justice Department document showed that FISA did not deny a single request all year.  Of course, without Edward Snowden’s 2013 NSA document leaks, we wouldn’t know any of this, as the court works in such secrecy that there’s little to no means for oversight.  One CNN report from 2013 pointed out, for example, that we don’t even quite know where the court literally is within its own building.

Those restrictions the NSA had in spying on American citizens were largely scrapped after 9/11 when, only a few weeks after the attack, the George W. Bush administration approved the then-top secret “President’s Surveillance Program”, or PSP.  The program allowed NSA access to communications records from American telephone companies, internet service providers and web services, according to a top-secret report by the NSA inspector general’s office, uncovered by the Washington Post in 2013.

Call detail records, email histories and internet usage data were compiled, en masse, under the PSP for years, with little consideration for legality or warrants.  This collection didn’t include the pure content of Americans’ communications, just the metadata itself (like who was doing what, where, and with whom).  That, of course, was the workaround the NSA leaned on, so that such actions would not obviously constitute unreasonable search or seizure made illegal by the 4th amendment.

The program was briefly discontinued in 2004, when senior members within the Justice Department (including former FBI Director James Comey, among others) led a sort of resistance against it.  These efforts didn’t kill the program, though–ultimately, it only led to a FISA court ruling that granted the NSA ability to, basically, keep doing what they were doing, only within a now legal framework.  The NSA’s power to spy on Americans without obvious foreign connection was expanded in 2007, and only in 2011, two years after Bush’s term ended, did this program, codenamed “Stellar Wind” become terminated.

Of course, all this NSA overreach wasn’t for naught–officials from within the organization have argued that in the decade following the events of 9/11, “over 50” terrorist plots worldwide have been foiled due to modern NSA data collection programs.  At least ten of those plots are said to have been aimed at United States targets, notably including the New York Stock Exchange and the New York City subway system.  While this “over 50” claim has met its detractors, it can also be argued that the mere existence of massive NSA spying complicates terrorism in the world generally.

Ultimately, though, if Lew Allen’s Church Committee testimony got the ball rolling on NSA suspicions, it was a 2006 USA Today report revealing the Bush-era NSA programs to the public that truly kicked off the severe public mistrust of the NSA we see today.  The stage was set, then, for the massive leak that turned the NSA into a household name…

Prism and UpStream were two of the NSA programs unveiled by Edward Snowden in 2013. Both aimed at collecting information via the Internet about US enemies from home and abroad. With Upstream, the NSA receives raw information from telecommunication providers such as AT & T: they analyze the information obtained, find phone numbers, IP addresses, and email addresses of people or organizations being monitored, and use the data for their own needs. With UpStream, the NSA has access to more than 75% of Internet traffic in the United States. Prism is a similar program that focuses on companies that provide Internet services such as Google, Facebook and the like.

Both programs, UpStream and PRISM, are relatively new. UpStream probably started before 2000 but expanded very much after 9/11. PRISM was created in 2007, following President Bush’s Patriot Act. But if we zoom out and review the history of the NSA, we can find the obvious roots of these two programs.

The Echelon program was created in the late 1960s: its purpose was to gather signals intelligence, or SIGINT, against the Soviet Union and its allies. According to press inquiries, Echelon’s people were eavesdropping on satellite communications, telephony, fax and similar types of electronic communications. Although, as might be expected, very little public information exists about Echelon, there is no doubt that it was a massive spy program. Collection and listening stations were established in many countries around the world, and it is estimated that the US government spent many miilions, if not billions of dollars, on Echelon. Snowden’s documents showed that the Echelon program continued well into the 21st century.

Knowing what we know today about UpStream and Prism, it is easy to identify the similarities between them and Echelon, both in goals and in the scale of activity. In other words, even without Snowden’s leak we could probably draw the outlines of these plans just from the NSA’s past patterns of activity. The interesting question is whether we can do the same for other NSA technologies and tools.

The Spaso House in Moscow has been the residence of US ambassadors to the Soviet Union since 1933. In 1952, Ambassador George F. Kennan lived there. These were the days of the Cold War, and American diplomats were very suspicious of their hosts. No one doubted that if they could do so, the Soviets would be more than happy to eavesdrop on the conversations of the American staff. Ambassador Kennan ordered repeated sweeps of all the rooms for hidden listening devices – but nothing was discovered. However, reports from British intelligence showed that the Soviets apparently managed to somehow listen to the conversations inside the house. How? That was the mystery.

In September 1952, Joseph Bezjian, a technician of the Department of State Security Service, was sent to conduct more extensive scans of the Spasso house. Bezjian suspected that the Soviets were somehow able to learn about the approaching inspections and removed the eavesdropping devices before the sweeps. So he pretended to be an ordinary guest who came to visit the ambassador and arranged things so that the technical equipment he needed would be sent separately ahead of his visit.

Bezjian asked the ambassador to read aloud text from some unclassified document, and began scanning the study using a portable radio receiver. He scanned the frequencies – until he suddenly heard the ambassador’s voice from his earphones. Soon he focused on the source of the broadcast: on one of the walls, above the ambassador’s desk, stood a big wooden plaque of the Great Seal of the United States. It was a gift from a delegation of the Soviet youth organization, The Young Pioneer, to the former American ambassador, almost seven years earlier. That gift turned out to be a Trojan horse. A tiny listening device was hidden behind the beak of the Great Seal’s bald eagle. Bezjian feared that the Soviets would try to steal the listening device to prevent it from falling into the hands of American electronics experts. That night he slept with the Great Seal under his pillow.

The next day, the Great Seal was sent to Washington – but it was only the beginning of the mystery. The listening device exposed by the Americans was different from anything they had encountered before. There was a microphone in it, and an antenna for transmission-but, and that was a big but – it had no battery, or any other connection to an external power source. How could the listening device be able to transmit signals without a power source? Did Soviet scientists succeed in breaking the laws of physics? The mystery was so great that the name given to the strange listening device was simply The Thing.

The Thing was the brainchild of a brilliant Soviet physicist named Leon Theremin. You may have heard of another famous invention of his: the musical instrument named after him.

The Theremin produces very unique sounds, and rose to prominence in the sixties mainly thanks to the song Good Vibrations by The Beach Boys. The Theremin is controlled by the manipulation of electromagnetic fields – a feature which attests to its genius, and the depth of Leon Theremin’s understanding of his field. It was this understanding that enabled him to plan and build, for his Soviet intelligence operators, a unique listening device.

At the heart of The Thing is a hollow metal cylinder called a Resonator. The dimensions of the cylinder were designed precisely so that electromagnetic waves at a certain frequency resonate within it, and amplitude increases as the waves are reflected off the metal walls. This phenomenon is similar to how sound waves are amplified within the resonance box of a Spanish guitar. The Thing’s amplified electromagnetic waves travel out of the cylinder, back into the antenna, and are transmitted from there into the air.

Now, at the edge of the cylinder is a thin sheet of metal called a diaphragm. The diaphragm is the microphone: the sound waves that hit it cause it to vibrate, and the slight vibrations change the electrical properties of the cylinder. Think of a swollen balloon pressed on its sides, and how the external pressure changes the density of the air inside the balloon. These slight changes affect the way in which the electromagnetic waves vibrate within the resonator by slightly changing the amplitude of the waves. In other words, the diaphragm movements modulate the electromagnetic waves emanating from the listening device. The receiver absorbing these radio waves is able to demodulate them, and retrieve back the information about the sound waves.

All the pieces of this electronic puzzle were known to the American researchers in advance: modulation of radio waves is the basis for AM and FM broadcasts, and the resonance of radio waves was a familiar phenomenon, especially in the radar world. But no one, until that moment, had found a way to make use of this idea in the secret world of listening devices.

The brilliance of Leon Theremin was in his use of resonance as a source of power for the The Thing. Outside the Spaso house, inside a large truck, Soviet operators directed a 300 MHz beam of radio waves at the Great Seal. 300 MHz was also the resonance frequency for The Thing, and these external radio waves were the power source of the listening device: they started the chain of events that ended with voice-modulated radio waves being transmitted from it. Bottom line: Theremin has found a way to build a battery-free listening device that can continue to operate practically forever if only 300 MHz of radio waves are projected on it. This technology was named Retro Reflection, and when American experts understood its operational principle, it became the basis for a wide range of listening devices created by the NSA for American spies and agents throughout the second half of the 20th century. Retro Reflection is also the basis for a variety of civilian technologies nowadays, such as RFID anti-theft chips in stores.

Fast forward to December 2013. The German magazine Der Spiegel published an article by security researcher Jacob Appelbaum, exposing a 50-page catalog written by a technical group within the NSA known as ANT or Advanced Network Technology. This catalog was created in 2008 for the Tailored Access Operations Division, or TAO, one of NSA’s elite cyber warfare units. It lists a host of hardware and software tools for spyware and bugging, from malware for  PC and mobile phones to fake GSM Base Stations. We don’t know for sure, but the ANT catalog is probably one of Edward Snowden’s leaked documents.

At least four of the 50 cyber espionage tools in the ANT catalog are based on Retro Reflection technology. For example, RAGEMASTER is the code name for a modified VGA video cable of the type that connects to desktop monitors. Inside the cable, on the red video line, there is a tiny resonator. By illuminating the cable with a focused radio beam, NSA spies can see and record whatever the user sees on his monitor.

SURLYSPAWN is the code name of another retro-reflector, this time a hidden chip embedded inside a keyboard. The chip receives information on key presses from the Data line of the keyboard cable, and transmits it out as modulations on the reflected radio waves. Another retroreflector, code named TAWDRYYARD, emits information about the transmitter’s location. In other words it seems that, like in the case of UpStream and Prism, here too the NSA adopted ideas and technologies from the organization’s technological history – and modified them to fit the era of cyber warfare.

But that does not mean the NSA does not continue to improve its methods of operation to deal with new challenges of the modern world. In the 1990s, for example, senior members of the organization realized that, along with the rise of the Internet, there was also a clear increase in the use of encryption technologies by many computer users. Free software such as PGP, or Pretty Good Privacy, has enabled non-specialists to implement strong encryption for email and instant messaging, and international standards such as SSL threatened to make large parts of the Internet communications infrastructure immune to NSA scrutiny.

That was the background to the Clinton administration’s announcement in 1993 that they’d be instituting a new initiative called Clipper Chip. The Clipper Chip, developed by the NSA, was supposed to be installed within any communications equipment produced in the United States, enabling the NSA to crack any encrypted communication going through it.

As one might imagine, this declaration aroused enormous opposition from Democrats and Republicans alike. High Tech industry leaders argued that such intentional inclusion of a security vulnerability would result in non-US technology companies not agreeing to use such equipment. Privacy advocates were furious at what they saw as a blow to the Fourth Amendment preventing unreasonable search and seizure.

The administration and the NSA tried to calm these claims. Each individual Clipper Chip has its own cryptographic key – that is, a long series of random numbers – with which the encryption can be cracked. This cryptographic key would presumably be in the hands of the NSA alone, and they would presumably be allowed to use it only if they could prove it necessary to do so.

Of course, this explanation did not convince the opposition: history has shown that the NSA has a tendency to expand its own powers from year to year, and government oversight of the organization has never been particularly tight. Another nail in the Clipper Chip’s coffin were two critical security weaknesses in the chip that an information security researcher exposed; weaknesses theoretically allowing a malicious third party to crack encrypted communications. In the end, the Clipper Chip was a failure: almost all the manufacturers refused to install it, and the program was shut down in 1996.

But the NSA’s need for free access to encrypted information has not disappeared.

In 2004, the NIST, National Institute of Standards and Technology, published a draft for a new encryption standard called Dual EC_DRBG. This standard defined an encryption protocol that  communications equipment manufacturers could implement in their devices in a reliable, secure, and error-free manner.

A few months later, in January 2005, appeared the first voices – from inside the NIST itself – who claimed that the Dual EC_DRBG standard might have a hidden weakness. And not just a weakness – but a backdoor that was intentionally placed in it by an unknown party. Encryption experts tend to take such concerns seriously – especially when it comes to an official standard that the US government has endorsed. In 2007, at a professional conference called CRYPTO, two researchers, Dan Shumow and Niels Ferguson, presented unequivocal proof of the existence of such a backdoor and demonstrated how it can be used to break the encryption.

What is the weakness hidden in Dual EC_DRBG? Well, almost all encryption schemes require the production of random numbers. It is this randomness that prevents a potential attacker from finding patterns within the encrypted information stream and using them to crack the encryption. Remember the British Alan Turing, who cracked the Enigma code? Much of his success was attributed to the fact that the German radio operators opened their morning telegrams with ‘Guten Morgen’ and concluded them with ‘Heil Hitler’. It was this pattern that the British codebreakers needed to crack the Enigma encryption.

In the Dual EC_DRBG standard, randomness is the result of a particular mathematical equation called an Elliptic Curve. This equation has parameters – let’s call them P and Q – that control how the equation produces random numbers. Think of them as dials that the communications equipment manufacturer can change as they implement the encryption scheme into their devices, to create their own custom version of the encryption.

But the reality is that most equipment vendors don’t want to change anything in the encryption standard. Encryption is notoriously complex, and even a tiny error in implementing the encryption algorithm can render it useless. Therefore, engineers always prefer to use default values for parameters if there are any – and this is why the NIST includes in the published standard two default values for P and Q.

How did the NIST select these two particular values? The authors of the standard did not elaborate, and it was this omission that aroused the suspicion of security experts. Then, Shumow and Ferguson showed that these two parameters are closely related to another set of numbers – let’s call them the Skeleton Key – that, if known, can enable an attacker to predict the random numbers produced by the Elliptic Curve equation of the the Dual EC_DRBG. In other words, if your communication equipment uses the default values of P and Q, and someone holds the Skeleton Key to the same default values … Well, how shall we put it professionally? You’re screwed. Your encryption is worthless.

Of course, the suspicion immediately fell on the NSA. In fact, the NSA was the one who gave the NIST the dual EC_DRBG scheme years ago and suggested that it should be made into a standard. No one could say for certain that the NSA did have the Dual EC_DRBG Skeleton Key, but the Clipper Chip affair showed everyone that the spy agency was very interested in breaking into encrypted communication channels. And if there is anything the history of the NSA has taught us, it’s that the intelligence organization is willing to invest tremendous efforts and large sums of money to achieve its goals.

Indeed, the documents leaked by Snowden revealed what was already clear to anyone with eyes in his head. In December 2013, Reuters reported that the NSA had paid $10 million to RSA Security, a big security firm, to integrate the cryptographic encryption standard into its cryptographic software libraries. The NSA itself denied these allegations in an official statement, but it is hard to believe that simple philanthropy is behind the secret deal with RSA.

Indeed, additional information Snowden released revealed a secret NSA program called BULLRUN, which aims to subvert international encryption standards. Dual EC_DRBG, it turns out, is just the tip of the iceberg: the NSA invests more than a quarter of a billion dollars each year to influence technology companies to allow it to peer through encrypted information via their products and services. If it’s any consolation, the British too have their own similar program, codenamed Edgehill.

What does all this mean? Is the NSA’s vigorous activity good or bad for us? There are those who see it as an unforgivable violation of privacy. Others will say that this is an essential part of the fight against terrorism and protecting the safety of American citizens. Either way, it is likely that we will discover in the future that the NSA’s long arms have reached much farther than most people imagine. Who knows, we might find it useful someday.

Resources:

http://www.nytimes.com/1994/06/12/magazine/battle-of-the-clipper-chip.html

http://www.nytimes.com/2005/12/16/politics/bush-lets-us-spy-on-callers-without-courts.html

http://projects.washingtonpost.com/top-secret-america/articles/a-hidden-world-growing-beyond-control/print/

https://www.theguardian.com/commentisfree/2013/jun/19/fisa-court-oversight-process-secrecy

https://intelligence.house.gov/uploadedfiles/setting_the_record_straight_2017_clean.pdf

https://www.washingtonpost.com/investigations/presidents-surveillance-program-worked-with-private-sector-to-collect-data-after-sept-11-2001/2013/06/27/2c7a7e74-df57-11e2-b2d4-ea6d8f477a01_story.html?utm_term=.8e65fd03bdc2

https://www.theverge.com/2013/7/17/4517480/nsa-spying-prism-surveillance-cheat-sheet

https://arstechnica.com/information-technology/2015/08/the-nsa-playset-espionage-tools-for-the-rest-of-us/

https://arstechnica.com/information-technology/2013/12/inside-the-nsas-leaked-catalog-of-surveillance-magic/

http://www.spiegel.de/international/world/catalog-reveals-nsa-has-back-doors-for-numerous-devices-a-940994.html

https://upload.wikimedia.org/wikipedia/commons/2/2a/NSA_RAGEMASTER.jpg

https://upload.wikimedia.org/wikipedia/commons/8/8f/NSA_TAWDRYYARD.jpg

https://www.schneier.com/blog/archives/2014/02/surlyspawn_nsa.html

https://www.theguardian.com/world/2013/sep/05/nsa-gchq-encryption-codes-security

http://www.eetimes.com/document.asp?doc_id=1274748

http://counterespionage.com/the-great-seal-bug-part-1.html

http://encyclopedia2.thefreedictionary.com/Cavity+Resonator

https://motherboard.vice.com/en_us/article/bmjpj3/michael-ossmann-and-the-nsa-playset

http://www.nytimes.com/2013/09/06/us/nsa-foils-much-internet-encryption.html?hp&_r=0

https://en.wikipedia.org/wiki/Bullrun_(decryption_program)

https://en.wikipedia.org/wiki/Clipper_chip

http://www.nytimes.com/2013/09/06/us/nsa-foils-much-internet-encryption.html?hp&_r=0

https://en.wikipedia.org/wiki/Dual_EC_DRBG

http://projectbullrun.org/dual-ec/description.html

https://gist.github.com/0xabad1dea/8101758

https://www.wired.com/2007/11/securitymatters-1115/

http://www.matthewaid.com/post/58904880659/nsa-surveillance-programs-cover-75-of-internet

https://www.google.com/url?q=http://nsarchive2.gwu.edu//NSAEBB/NSAEBB260/index.htm&sa=D&ust=1516113391701000&usg=AFQjCNEy2aoulcz97pRDfdut91JX3kFnnQ

https://www.aarclibrary.org/publib/church/reports/vol5/pdf/ChurchV5_1_Allen.pdf

http://foreignpolicy.com/2013/09/25/secret-cold-war-documents-reveal-nsa-spied-on-senators/

https://www.youtube.com/watch?v=XSM0pb3aJck

https://apps.washingtonpost.com/g/page/world/national-security-agency-inspector-general-draft-report/277/

http://www.nytimes.com/2005/12/16/politics/bush-lets-us-spy-on-callers-without-courts.html

https://www.newyorker.com/news/john-cassidy/n-s-a-latest-the-secret-history-of-domestic-surveillance

https://www.theguardian.com/world/2013/jun/27/nsa-data-mining-authorised-obama

http://abcnews.go.com/Politics/nsa-director-50-potential-terrorist-attacks-thwarted-controversial/story?id=19428148

https://www.nbcnews.com/news/other/nsa-program-stopped-no-terror-attacks-says-white-house-panel-f2D11783588

https://usatoday30.usatoday.com/news/washington/2006-05-10-nsa-qna_x.htm

http://projects.washingtonpost.com/top-secret-america/articles/a-hidden-world-growing-beyond-control/print/

https://www.theguardian.com/commentisfree/2013/jun/19/fisa-court-oversight-process-secrecy

https://intelligence.house.gov/uploadedfiles/setting_the_record_straight_2017_clean.pdf

https://www.washingtonpost.com/investigations/presidents-surveillance-program-worked-with-private-sector-to-collect-data-after-sept-11-2001/2013/06/27/2c7a7e74-df57-11e2-b2d4-ea6d8f477a01_story.html?utm_term=.8e65fd03bdc2

http://www.cryptomuseum.com/covert/bugs/thing/files/DoS_history.pdf