XKEYSCORE

Audio readout of this entry.

Overview

XKEYSCORE is the NSA's principal SIGINT search-and-analysis software platform — the analytical-tooling layer above the collection-programme layer. The platform systematically indexes the collected SIGINT product (internet traffic, email, chat, web-browsing material, and adjacent operational categories) into a searchable database that NSA-and-partner analysts use to conduct targeted searches against the collected material.

The defining characteristic of XKEYSCORE is its natural-language query interface — the operational pattern in which analysts type natural-language queries and the platform returns the matching collected material. The Snowden-disclosed documentation describes the platform as the "Google for SIGINT": XKEYSCORE functions for SIGINT analysts as a consumer-internet search engine functions for ordinary users. Within the broader Five Eyes intelligence-collection-and-analysis pattern, XKEYSCORE is the most widely deployed and most heavily used analytical platform — the shared analytical infrastructure across the Five Eyes cohort.¹⁴

Origins and predecessor platforms

The XKEYSCORE programme was initiated within NSA in approximately 2003. The institutional context was the post-2001 expansion of internet-traffic SIGINT collection, which produced collection volumes that required a more capable analytical platform than the prior infrastructure could support. The pre-XKEYSCORE NSA infrastructure included PINWALE — the long-term content-retention database that continues to coexist with XKEYSCORE in the architecture. XKEYSCORE is the front-end search and analysis layer; PINWALE is the back-end long-term content store into which XKEYSCORE-collected material of analyst interest is forwarded for retention beyond the XKEYSCORE field-site window.

The post-2003 development of XKEYSCORE expanded across the Five Eyes cohort and into selected non-Five-Eyes partners. By the 2013 disclosure period the platform was the institutional analytical-platform standard across the broader cohort.

Five Eyes and partner deployment

The documented deployment pattern across the Five Eyes and adjacent partners comprises five principal partners.

NSA operates the principal deployment: approximately 700 servers across approximately 150 NSA field sites worldwide. The major sites include NSA Maryland (Fort Meade) institutional headquarters; NSA Hawaii (the Kunia Regional SIGINT Operations Center, succeeded operationally by the Hawaii Cryptologic Center near Wahiawa, opened January 2012); NSA Georgia (Fort Gordon); NSA Texas (San Antonio Lackland Air Force Base, Medina Annex); NSA/CSS Misawa (Misawa Air Base, Security Hill, Japan); and additional NSA sites at partner-service deployment locations.

GCHQ operates the paired UK deployment, principally at the Cheltenham institutional headquarters and at the Bude field site that hosts the Tempora collection facility.

BND and BfV — the Bundesnachrichtendienst (foreign intelligence) and the Bundesamt für Verfassungsschutz (domestic intelligence) — both operate documented German deployments of XKEYSCORE, as confirmed by the Tagesschau headline of 3 July 2014: BND und Verfassungsschutz setzen US-Spähprogramm ein. The German engagement with the platform was disclosed in the Der Spiegel cover story of 20 July 2013 by Laura Poitras, Marcel Rosenbach, and Holger Stark; the July 2014 Tagesschau / NDR reporting (Kampf, Appelbaum, Goetz) separately disclosed the Tor-detection fingerprint and an XKEYSCORE source-code fragment.

ASD — the Australian Signals Directorate — operates the documented Australian deployment.

GCSB — the New Zealand Government Communications Security Bureau — operates the documented New Zealand deployment.

The platform is the most broadly deployed analytical infrastructure across the Five Eyes cohort and a significant non-Five-Eyes partner deployment in the German case.²³

Natural-language query interface

The defining operational characteristic of XKEYSCORE is the natural-language query interface. The Snowden-disclosed documentation includes screen-shot examples of the query interface, documenting the pattern: analysts type natural-language queries — examples include "show me all the encryption usage in country X across the past 30 days," "show me all the email accounts that have ever logged in from this IP address," "show me everyone in country X who searched for the word Y in language Z" — and XKEYSCORE returns the matching collected material.

The operational consequence of the natural-language pattern is that the analyst skill threshold for effective use of the platform is significantly lower than the threshold for prior NSA analytical platforms, which had required analysts to understand a detailed database schema and construct formal-query-language queries. The natural-language pattern broadened the analyst cohort that could effectively use the platform.⁴

Retention windows

The documented retention pattern across the major XKEYSCORE field sites in the 2008–13 institutional period combined a three-day content-retention window with a thirty-day metadata-retention window. The major XKEYSCORE field sites retained the full content of collected internet traffic for approximately three days; metadata about the collected traffic for approximately thirty days; and significantly targeted material for substantially longer institutional retention.

The operational consequence of the three-day content-retention window is that the most actionable XKEYSCORE-collected material must be acted upon within the retention window. The institutional pattern that emerged is a routine search cadence in which analysts conduct queries across the most recently collected material to identify significant collection product before the three-day window expires.¹

Pre-categorised fingerprints

The platform's institutional pattern of pre-categorised "fingerprints" allows analysts to identify categorisable targets without the per-target query construction the natural-language pattern would otherwise require. The disclosed fingerprint categories have included cryptographic-protocol-usage fingerprints (the detection of Tor anonymisation network usage, PGP/GPG email-encryption usage, VPN protocol usage, OTR off-the-record chat encryption usage, and adjacent cryptographic patterns); language-and-region fingerprints; political-and-religious-affiliation fingerprints; and additional categorisable target categories.

The Tor-detection fingerprint was disclosed on 3 July 2014 in the Tagesschau and Norddeutscher Rundfunk reporting by Lena Kampf, Jacob Appelbaum, and John Goetz — NSA targets the privacy-conscious — based on an XKEYSCORE source-code fragment. The disclosure documented that any internet user connecting to the Tor network from a non-Five-Eyes jurisdiction was categorised as a significant target by the fingerprint pattern.³

Documented deployments

Confirmed NSA principal deployment. The documented NSA deployment of XKEYSCORE across the 2003-present period at approximately 700 servers across approximately 150 NSA institutional field sites worldwide is the defining institutional pattern.

Confirmed GCHQ deployment. The documented GCHQ deployment of XKEYSCORE at the Cheltenham headquarters and at the Bude field site (the Tempora collection facility) — the paired Five Eyes institutional deployment.

Confirmed BND and BfV deployment. The documented German BND and BfV deployment of XKEYSCORE — disclosed in the Der Spiegel reporting of 20 July 2013 (Poitras, Rosenbach, Stark) and confirmed by the July 2014 Tagesschau headline 'BND und Verfassungsschutz setzen US-Spähprogramm ein.'

Sources and further reading

1. Glenn Greenwald, XKeyscore: NSA tool collects 'nearly everything a user does on the internet', The Guardian, 31 July 2013 — the principal initial XKEYSCORE disclosure.
2. NSA-Programm XKeyscore: BND und Verfassungsschutz setzen US-Spähprogramm ein, Tagesschau, 3 July 2014 — the principal disclosure of the BND XKEYSCORE deployment.
3. Lena Kampf, Jacob Appelbaum, and John Goetz, NSA targets the privacy-conscious, Norddeutscher Rundfunk, 3 July 2014 — the principal XKEYSCORE source-code-fragment disclosure documenting the Tor-detection fingerprint pattern.
4. Morgan Marquis-Boire, Glenn Greenwald, and Micah Lee, XKEYSCORE: NSA's Google for the World's Private Communications, The Intercept, 1 July 2015 — the principal subsequent XKEYSCORE institutional reconstruction.
5. Glenn Greenwald, No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State, Metropolitan Books, 2014 — chapters on XKEYSCORE.
6. National Security Archive Snowden Documents Collection, George Washington University.
7. Bruce Schneier, Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World, W. W. Norton, 2015 — subsequent academic-and-policy treatment.
8. Privacy and Civil Liberties Oversight Board, Report on the Surveillance Program Operated Pursuant to Section 702 of the Foreign Intelligence Surveillance Act, 2 July 2014 — the principal post-disclosure institutional review of the Section 702 framework that feeds XKEYSCORE.
9. Susan Landau, Listening In: Cybersecurity in an Insecure Age, Yale University Press, 2017 — chapters on XKEYSCORE and the broader SIGINT analytical-platform landscape.
10. Bruce Schneier, NSA Targets the Privacy-Conscious for Surveillance, Schneier on Security blog, 3 July 2014 — Schneier's contemporaneous analysis of the disclosed XKEYSCORE Tor-detection source-code fragment.