Arguably the biggest SpiderFoot release in years, SpiderFoot 4.0 introduces the concept of writing your own correlation rules in YAML, plus integration with a number of open source security tools.
SpiderFoot’s goal is to automate OSINT collection and analysis to the greatest extent possible. Since its inception, SpiderFoot has heavily focused on automating OSINT collection and entity extraction, but the automation of common analysis tasks — beyond some reporting and visualisations — has been left entirely to the user. The meant that the strength of SpiderFoot’s data collection capabilities has sometimes been its weakness since with so much data collected, users have often needed to export it and use other tools to weed out data of interest.
We started tackling this analysis gap with the launch of SpiderFoot HX in 2019 through the introduction of the “Correlations” feature. This feature was represented by some 30 “correlation rules” that ran with each scan, analyzing data and presenting results reflecting SpiderFoot’s opinionated view on what may be important or interesting. Here are a few of those rules as examples:
- Hosts/IPs reported as malicious by multiple data sources
- Outlier web servers (can be an indication of shadow IT)
- Databases exposed on the Internet
- Open ports revealing software versions
- and many more.
With the release of SpiderFoot 4.0 we wanted to bring this capability from SpiderFoot HX to the community, but also re-imagine it at the same time so that the community might not simply run rules we provide, but also write their own correlation rules and contribute them back. We also hope that just as with modules, we see a long list of contributions made in the years ahead so that all may benefit.
Learn more about correlation rules here.
The good news is that SpiderFoot 4.0 will automatically upgrade your 3.x database to support correlations, so you don’t lose your configuration or historical scans. You can even run the correlation rules against historical scans:
$ ./sf.py -C <scan ID>
Of course, for any new scans, correlation rules will be run automatically and available in the UI and CLI. Try customizing the rule-set or adding your own (and don’t forget to contribute back!)
Open Source Tool Modules
SpiderFoot has integrated with a few popular open source tools such as DNSTwist, CMSeek, Whatweb, WAFW00F and Nmap for some time. With 4.0, we’ve introduced the following tools. Even better, you can build a docker container with all the tools built and included for you by using the Dockerfile.full file in the repo.
- nbtscan: Scans for open NETBIOS nameservers on your target’s network.
- Nuclei: Fast and customizable vulnerability scanner with a powerful templating framework for custom detections.
- onesixtyone: Fast scanner to find publicly exposed SNMP services.
- snallygaster: Finds file leaks and other security problems on HTTP servers.
- testssl.sh: Identify various TLS/SSL weaknesses, including Heartbleed, CRIME and ROBOT.
- TruffleHog: Searches through git repositories for high entropy strings and secrets, digging deep into commit history.
- Wappalyzer: Wappalyzer indentifies technologies on websites.
You can download SpiderFoot 4.0 here.