Its been a while since Rapid7's Open Data was closed off to anyone for the taking. The data it contained was rich for OSINT and vulnerability research purposes. I found OpenINTEL and wondered how it compared, whether it could be as useful. In my case, it was.

First we will look at the structure of the data focusing on forward DNS. Then we will unpack approaches to downloading and processing the data to store it in a meaningful way as well as provide tips for those who are limited with storage space. I’ll also share off-the-shelf tools, scripts, and commands so others can replicate the process.

Next we will explore how the data could be used to help pentesters and bug bounty hunters. The data as it stands spans a range of 10 years when you combine the various top lists together. This is a pretty long timeline for some domains, which could be helpful. For instance, consider a target fronted with WAF, you could look at the data to try find the origin IP. I used bug bounty targets as a test and was able to find data for about 250 targets. From that list of 250, I found that I could access the application directly for around 60 targets rather than through the WAF.

The data could also be used to perform reverse lookups. While there are many such services online, reverse lookups on certain record types is harder to come by, for instance TXT records type. TXT records often contain ownership verification values, and administrators sometimes use the same tenant or subscription across multiple domains. With the ability to perform reverse TXT lookups, this can reveal related assets and broaden the potential attack surface.

We will also unpack CrUX and how it can be helpful to regionally significant targets. The data in top lists is somewhat skewed since it focuses primarily on the global top one million sites as ranked by Cisco Umbrella, Tranco, etc. As a result, smaller or regionally significant targets may be underrepresented. CrUX, a recently added source could be useful here since it groups data by region and highlights locally popular sites, possibly making the data within OpenINTEL more relevant to your specific context.

I will conclude the talk with key takeaways and my opinion of OpenINTEL and whether its useful. Then I will provide the relevant links and move to Q&A.

The takeaways for my talk are:
* Historical data sets continue to be useful.
* OSINT shouldn't be limited to services built for it — we should keep exploring new sources.