Project SHADOWSTAR: A Data Driven Approach to Network Block Enumeration (Part 2)

by | Aug 5, 2021

TL; DR

In Part 1, we discussed network blocks, Internet registries, and standard network block enumeration methods for penetration tests and red teams. In this post, we’ll walk through the release of the SHADOWSTAR tool and explore how it can help teams perform network block enumeration significantly faster and more thoroughly.

All the source code and documentation for SHADOWSTAR can be found on our GitHub: https://github.com/SecurityRiskAdvisors/SHADOWSTAR.[1]

 

Introduction to SHADOWSTAR

Recall that in Part 1, we discussed Regional Internet Registry (RIR) data and how it can be used to help identify target network blocks. We also discussed a lesser-known data source called Internet Routing Registry (IRR) data and why it is useful as a complementary data source to RIR data. At the time of this blog post, there are several tools for querying RIR data; however, we were unaware of any that query IRR data. Hence SHADOWSTAR was born.

The objectives of the SHADOWSTAR tool are as follows:

  1. Provide the best possible interface for discovering network blocks belonging to an organization.
  2. Provide a secure and simplified interface that can be shared with a team of operators.
  3. Automatically perform database updates and abstract away the complications from ingesting RIR/IRR data simultaneously.

 

 

SHADOWSTAR Architecture

Below is a simple block diagram which shows the key components of the SHADOWSTAR architecture. There is a web application hosted in S3 that is connected to a REST API which calls the Athena service and returns results based on keyword searches.

For auto-updating there is an AWS Fargate task which is invoked at a regular interval by a CloudWatch rule. This task periodically updates the data from the various RIR/IRR sources enumerated in Part 1.

You’ll need to wait about an hour for the collection and parsing to place before the system will have any data that can be queried. Once that’s done, you’re ready to go!

The first thing you’ll notice about SHADOWSTAR relative to other enumeration methods is that it is extremely fast. This is due to its offline data-driven approach, which uses the Athena service and custom data parser to project the disparate data into a simple unified data model:

SHADOWSTAR produced 36,390 network blocks in less than 10 seconds

 

CIDR Reduction and the Overlapping Network Blocks problem

One curious feature of SHADOWSTAR is the checkbox that says “CIDR Reduce Upon Save” this is deserving of some explanation.

Notice that from above, when we export our results using the save button, the number of effective blocks we see is not 36,390. There are only 5,338! So where did our blocks go!?

SHADOWSTAR data after it has been reduced

The quick answer is that SHADOWSTAR is intelligent enough to know when blocks overlap, and it performs an algorithm called CIDR Reduction. We essentially optimize the result set down to the minimal spanning set of CIDR blocks that encompass the same logical span as the original set. The following diagram may be helpful to explain what’s happening:

Example of overlapping network blocks

Given a setup like the one above, SHADOWSTAR would select only the “3.128.0.0/15” network block since it encompasses all the others and thus it is pointless to keep the child blocks.

This inference is almost always valid because these overlapping blocks were collected from the same keyword search. Meaning that it is extremely likely that you will only need the largest one. However, if you prefer to keep the entire raw result set, you can simply uncheck the box.

Note: since we are consuming IRR data in conjunction with RIR data, this is a very common problem and moreover it is not desirable to have a non-reduced set of CIDR blocks since this leads to confusion and often duplication of work.

 

The Power of IRR Data

We spoke in part 1 about our relative success with using IRR data and we think it’s worth highlighting an example. Consider a company like Spotify, Inc. Let’s observe what happens if we select only IRR data sources for our keyword search:

Example of searching for Spotify blocks only in IRR data sources

Notice that there are 2 results (identical in this case), and they both came from the “LEVEL3” (CenturyLink) source. Recall that these objects are not actual network blocks, they are routes; However, as explained in Part 1, it is often possible to blur the lines between routes and network blocks to treat them as being one and the same.

Specifically, in this example, the results are highly interesting examined within the RIR responsible for managing them (RIPE-NCC). You see this:

Network block WHOIS information from RIPE-NCC.

With the image above, we can see that there is no mention of Spotify anywhere! Our hypothesis currently is that this block has been sub-allocated to Spotify by what appears to be an ISP based out of Sweden[2] and the IRR route refers to this sub-allocation.

OK, cool. But how do we know that this is related to Spotify? As mentioned in Part 1, validating IRR data is critical because the information is not authoritative. We can try several methods to gather evidence to support or disprove our hypothesis.

A personal favorite of ours is to use tools like Shodan[3], Censys[4], and ZoomEye[5] to check if there are live hosts within that range that indicate a positive association to the organization in question. In this case, there is one interesting host within this range (or at least there was one in 2018):

The host “pub0-2.itvpn00.ash.spotify.net” appears to be hosting an SSL VPN back in 2018. This seems to lend evidence to our hypothesis that this IRR route object can be considered a CIDR block for Spotify. Of course, your milage may vary here, but we’ve had great success in identifying unique systems and services from this IRR data. As always, confirm with clients before assuming a particular block belongs to them.

 

Conclusions

In our first post we covered a lot of content related to network block enumeration. In this post we covered how we adapted our process at SRA to improve our visibility into RIRs we have historically not been able to operationalize, showed the practical benefits of including IRR data in with RIR data, and released our tool that can help you level up your recon game.

We hope this post was informative and that you have been convinced to give SHADOWSTAR a try – we think that if you try it, you’ll never want to do network block enumeration any other way. Cheers!

 

References

  1. https://github.com/SecurityRiskAdvisors/SHADOWSTAR
  2. https://ip-osteraker.se/
  3. https://www.shodan.io
  4. https://censys.io/
  5. https://www.zoomeye.org

 

Peter Crampton
Scientist | Archive

Peter primarily specializes in reconnaissance and OSINT methodologies as well as adaptive Red Team engagements. He regularly works on web application assessments, purple teams, and red team engagements. Peter is familiar with several common tools such as Burp Suite, Cobalt Strike, and Metasploit.

Peter has worked with many companies across industries such as pharmaceutical, banking, financial institutions, and telecommunications.

Peter holds a degree in Computer Science from Drexel University and has worked in the InfoSec field for 7 years. He has experience working with government entities such as the DoD, DISA, and private contractors like Lockheed Martin.