Open source intelligence techniques and the Dark Web

We all know and use the Surface Web -that part of the Web that’s indexed and accessible to popular search engines like Google, Bing and Yahoo! But the Surface Web represents just a fraction of the Internet.

Underneath lies a wealth of information that most people aren’t aware even exists because it can’t be accessed by conventional browsers. This so called ‘Deep Web' is vast and is currently estimated to represent approximately 96 per cent of all content on the Web.

Other content that cannot be indexed by traditional search engines and therefore part of the Deep Web includes dynamic pages returned in response to a submitted database query, content that’s dynamically downloaded from Web servers via Flash or Ajax, web archive and peer-to-peer networks.

As well as the Deep Web is another section called the Dark Web. The Dark Web is only accessible using specialist software, the most common of which is called Tor (originally the acronym for ‘The Onion Router' The anonymity delivered by Tor makes it an attractive tool for Internet users engaged in illegal online activities looking to carry out illegal activities online.

Often described as the Wild West of the Web, the ‘Dark Web' is the shady world where illegal drugs and weapons dealers, and hackers ply their trade.

The Dark Web

In recent years the Dark Web hit the headlines and brought public attention to content on the Web that’s intentionally hidden from Web browsers.

In 2013 the FBI used its knowledge of the Dark Web and Open Source Intelligence techniques to shut down the notorious Silk Road online black market site and arrest its founder, Ross Ulbricht. Over the course of two and half years, Silk Road had hosted over a million sales of 11,000 types of narcotics.

The site’s seizure brought the existence of the Dark Web into the mainstream and raised serious concerns about cyber security for those in Government, Law Enforcement and the Private Sector.

In December 2014 a study by Gareth Owen from the University of Portsmouth found the most common type of content requested by those visiting hidden services via Tor was child pornography, followed by black markets. Many whistle blowing sites -like WikiLeaks - maintain a presence as well as political discussion forums. Cloned websites, scam sites and other fraud related services were also prolific.

For this reason, having an integrated understanding of the Dark Web is of primary importance for any private company or public sector body looking to ensure it has ‘belt and braces' data security protocols in place.

A source of valuable intelligence

Techniques like Open Source Intelligence (OSINT) gathering and a proper understanding of the Dark Web is the first step in combating the Internet’s dark places. With an understanding of how to use open source encrypted anonymity services safely, organisations can explore OSINT sources - which include web-based communities, user-generated content, social-networking sites, wikis, blogs and news sources to investigate potential threats or analyse relevant information for business purposes.

Whether that’s using Deep and Dark web sites and directories to support intelligence gathering for investigation purposes, manage incidents or to combat cyber crime.