It’s been a long time in the making but SpiderFoot 3.0 is finally tagged and released. With the last official release (2.12) being almost two years ago and active development since then, you can imagine how much has changed, so let’s walk through the major changes.
Note that if you’ve been regularly pulling the master branch from the Github repo over the past few days or using the v3 branch over the past few months, some of this might not be new to you although some changes have been pushed as recent as yesterday, so be sure to git-pull and give it a another go.
Much broader set of targets
Out of all the targets SpiderFoot supports, those new in 3.0 since 2.12 are in bold:
- IPv4 addresses
- IPv6 addresses
- Domain names
- Phone numbers
- E-mail addresses
- Real names
When targeting names and usernames, it’s important to remember to place them in quotes, e.g.
"Frank Smith" and
"fsmith2000". Phone numbers must be in international format, prefixed with a
+ followed by the country code, e.g.
Rich command-line interface
Previously, SpiderFoot was controlled exclusively through a web interface but it’s now possible to also orchestrate scans through
sf.py itself via the command-line. This means you can do things like
python3 ./sf.py -m sfp_haveibeenpwned -s email@example.com to query HaveIBeenPwned? for an e-mail address. Take a look at
sf.py --help for more options.
Check out this tutorial to see an example of using the command-line to perform some DNS recon:
sfcli.py interface, which was introduced in SpiderFoot 2.10, still exists and is more for remotely controlling a SpiderFoot server but offers more functionality such as configuration management, deleting scans and so on.
Since Python 2.7 is now end-of-life, SpiderFoot 3.0 has been 100% converted to Python 3. This also means that many older dependencies, especially problematic ones like M2Crypto, have been completely replaced.
All you need to do now to install dependencies is
pip3 install -r requirements.txt And be sure to run SpiderFoot with
python3 in case your Linux distribution still uses Python 2.7 by default.
~170 modules and counting
Some dead data sources have been retired, many new ones have been added; with and without a need for API keys.
You can access the full module list in the repo’s README.
No Windows binary
You may have noticed that there’s no longer a place for downloading the Windows binary of SpiderFoot. That’s because py2exe, the tool used to build a Windows executable out of a Python application, hasn’t been touched since 2014.
Fortunately, it’s pretty easy to install and run SpiderFoot natively as Python on a Windows machine if you follow these simple instructions.
API key export/import
Managing API keys is probably the most annoying part of getting any OSINT tool fully set up. We’ve tried to make this less painful by providing an import/export feature in the web interface so that using API keys between installations or between SpiderFoot HX and the open source version is easier.
On that note, we are considering spinning up a service that would enable you to run scans without the need for any API keys, and you’d pay per scan. If you’d be interested in being a part of the beta, you can access the beta registration here.
JSON export format
SpiderFoot can now export data from the UI in JSON format, adding to CSV and GEXF.
Tons of bug fixes
A significant number of bugs have been fixed in this release, improving data quality and reducing errors. A new test suite is also gradually being introduced and will be developed over coming releases.
Want to contribute?
SpiderFoot is actively growing and always looking for contributors, especially when it comes to additional modules. Feel free to submit a pull-request to the repo, reach out to @spiderfoot on Twitter or e-mail us at firstname.lastname@example.org is you’re interested in making a contribution and looking for a place to start.