This release significantly re-structured the modules so that they are more data source-centric. This gives you much more control over what data sources are used during a scan in addition to making SpiderFoot even more configurable. Here are the details:
- sfp_dns was split up into multiple modules to break up the tasks of simple DNS look-ups, DNS record querying, brute-forcing and look-aside queries.
- sfp_malcheck was split up so that each module represented one data source. There was also a lot of clean-up to tailor the modules to the different types of data they can consume, and data they returned. Ultimately this will result in better data quality and improved performance.
- sfp_blacklists was split up in the same way, but for blacklists like SpamCop, SORBS, etc.
- sfp_sharedip is now split up to be per-data source, which is HackerTarget.com, Bing.com and Robtex.com.
- sfp_pastes is also broken up to be specific to each data source. Notepadd.cc was dropped because it’s no longer alive.
- Some modules were renamed to reflect their data source rather than something generic, so that it’s more meaningful when setting up a scan.
New modules and updates
- New: sfp_builtwith uses the BuiltWith Domain API to identify names, e-mail addresses, web technologies, phone numbers and more.
- New: sfp_ahmia: Searches the Ahmia.fi search engine to .onion sites (sites on the TOR network / dark web).
- New: sfp_torch: Searches the TORCH search engine for .onion sites. You need TOR enabled for this to work, because TORCH is on TOR itself.
- New: sfp_fraudguard: Queries the Fraudguard.io API to identify potentially malicious IPs, hosts and domains in addition to collecting Geo-IP information.
- New: sfp_bitcoin: Identifies Bitcoin addresses on scraped web content.
- New: sfp_blockchain: Queries blockchain.info to find the balance of identified Bitcoin wallet addresses that might have been scraped from web pages being spidered.
- Update: Palevo tracker removed from sfp_abusech, as it’s no longer alive.
- Update: sfp_robtex (previously part of sfp_sharedip) now uses the robtex.com API for idetifying co-hosted sites.
Enhancements / Bug fixes
- CLI allows comments (prefixed by #)
- Misc. minor bug fixes, performance improvements and tweaks.