Module re-structuring

This release significantly re-structured the modules so that they are more data source-centric. This gives you much more control over what data sources are used during a scan in addition to making SpiderFoot even more configurable. Here are the details:

  • sfp_dns was split up into multiple modules to break up the tasks of simple DNS look-ups, DNS record querying, brute-forcing and look-aside queries.
  • sfp_malcheck was split up so that each module represented one data source. There was also a lot of clean-up to tailor the modules to the different types of data they can consume, and data they returned. Ultimately this will result in better data quality and improved performance.
  • sfp_blacklists was split up in the same way, but for blacklists like SpamCop, SORBS, etc.
  • sfp_sharedip is now split up to be per-data source, which is, and
  • sfp_pastes is also broken up to be specific to each data source. was dropped because it’s no longer alive.
  • Some modules were renamed to reflect their data source rather than something generic, so that it’s more meaningful when setting up a scan.

New modules and updates

  • New: sfp_builtwith uses the BuiltWith Domain API to identify names, e-mail addresses, web technologies, phone numbers and more.
  • New: sfp_ahmia: Searches the search engine to .onion sites (sites on the TOR network / dark web).
  • New: sfp_torch: Searches the TORCH search engine for .onion sites. You need TOR enabled for this to work, because TORCH is on TOR itself.
  • New: sfp_fraudguard: Queries the API to identify potentially malicious IPs, hosts and domains in addition to collecting Geo-IP information.
  • New: sfp_bitcoin: Identifies Bitcoin addresses on scraped web content.
  • New: sfp_blockchain: Queries to find the balance of identified Bitcoin wallet addresses that might have been scraped from web pages being spidered.
  • Update: Palevo tracker removed from sfp_abusech, as it’s no longer alive.
  • Update: sfp_robtex (previously part of sfp_sharedip) now uses the API for idetifying co-hosted sites.

Enhancements / Bug fixes

  • CLI allows comments (prefixed by #)
  • Misc. minor bug fixes, performance improvements and tweaks.