It’s that time again – another SpiderFoot release delivering a bunch of new modules, removing many modules for defunct data sources plus many performance improvements, under-the-hood code improvements and bug fixes as well.

Before reading on to all the major improvements, there is one big change to take note of: the spiderfoot.db file, which contains all scan data and configuration (including API keys) is no longer loaded from the spiderfoot folder, but instead from ~/.spiderfoot. You’ll know you’re impacted by this change if you get the following error message when trying to run SpiderFoot:

ERROR: spiderfoot.db file exists in /spiderfoot/.
SpiderFoot no longer supports loading the spiderfoot.db database from the application directory.
The database is now loaded from your home directory: /johnsmith/.spiderfoot/spiderfoot.db
This message will go away once you move or remove spiderfoot.db from /spiderfoot/.

This change has been made to make packaging easier in the future. Therefore, before you install and run SpiderFoot 3.5, simply do the following:

  1. Create a folder in your home directory called .spiderfoot (don’t forget the . in front). On Linux you can do this with mkdir ~/.spiderfoot
  2. Move the existing spiderfoot.db file from wherever your SpiderFoot installation is today, into that folder. For example, if your SpiderFoot installation is in /spiderfoot, simply run mv /spiderfoot/spiderfoot.db ~/.spiderfoot/

Now onto the good stuff…

Here’s the full list of what you’ll see in SpiderFoot 3.5:

  • 12 new modules.
  • In previous versions, IPv6 support was weak to non-existent, so we’ve updated many modules (e.g. Whois, AlienVault, Spyse and many more) to support IPv6 addresses and IPv6 netblocks
  • Excel export, so you can have one data type per tab in the export
  • Bech32 format support of Bitcoin addresses as targets and extraction from scraped web content
  • Split out reporting of VPN hosts, TOR nodes and Proxy servers as their own data types (previously they were reported as malicious)
  • Log file export and filtering in the UI
  • Added Dockerfile.full, which includes all tools that SpiderFoot supports today and more: CMSeeK, nmap, DNSTwist, wafw00f. You can still use Dockerfile if you want to build a slimmer image.
  • A significant number of bug fixes and data quality improvements across the core, unit tests and modules.

As usual a big thank you to @_bcoles, @thetechr0mancer and Krishnasis Mandal for all the work that helped make this release happen!

See the full list of changes since the 3.4 release here.

Download the SpiderFoot 3.5 open source release here.