This week we pushed SpiderFoot HX v7 to production (zero downtime!) with a ton of improvements that further establish SpiderFoot HX as the OSINT automation platform for penetration tests, threat intelligence and attack surface management. This release has been in the works for months, based heavily on user feedback, so naturally there’s a lot to unpack.

Let’s walk through all the major changes.

Custom Modules beta launch

If you’ve been using the open source version of SpiderFoot for some time, you may have created custom modules for your own purposes. We’ve heard from a few folks over the years that it would be really useful if they could port those modules over to their SpiderFoot HX account and run them in the cloud, giving them more control over data collection and analysis methods.

In response to that feedback, we’ve launched the ability to import your own SpiderFoot modules (compatible with v3.3 of the open source version), which should require no code changes at all to run. While in beta, it isn’t possible to introduce third party binaries or libraries, but we’re aiming for that in the next release. This feature is now available to Enterprise customers.

Configure your module, upload the code and it will appear alongside all the usual SpiderFoot HX modules when you configure your next scan.

Firehose to Splunk, ElasticSearch and REST endpoints

If you’re a SOC trying to integrate OSINT into your operations, we recognize that being able to feed data into your Splunk, ElasticSearch or SOAR platform in real time is vital. This is especially useful when used together with Monitors, since you can monitor your attack surface and feed scan data into your platform as it is being discovered, enabling you to trigger your own responses from your own environment and tooling.

SpiderFoot HX now enables you to configure Splunk, ElasticSearch and REST endpoints and have data from scans fed to them in real time as SpiderFoot identifies them. With this you can:

  • Integrate with your SOC tooling (SIEM/SOAR) for alerting and automated response when a SpiderFoot HX Monitor discovers certain data
  • Feed data into your own local data warehouse in order to run your own custom queries
  • Trigger your own pager, email or other notification systems under certain conditions

This feature is now available to Enterprise customers.

Feed SpiderFoot data directly into your own security data platform as raw JSON, for your own analysis and reporting.

SpiderFoot HX API v2.0

SpiderFoot HX’s API has undergone a complete overhaul and re-launched as fully RESTful, with interactive documentation including examples for cURL, Python, Golang and more. We’ve also added many new endpoints giving you more control over how to query and control your SpiderFoot HX instance.

v1.0 of the API will be sunset at the end of this year (2021), so please migrate. If you need assistance doing so, please contact support.

Easy to navigate and search, plus example code for all the major languages.

Monitoring improvements

One of the major differentiating features of SpiderFoot HX over the open source version is its monitoring capability. In v7, we’ve made big changes giving you more control over what is monitored and how it is reported. Email notifications of changes now also include a visual report of the changes.

Overhauled dashboard for more meaningful reporting about changes.
New changes timeline makes it easier to spot areas of interest.
More granular control over what is monitored and reported.

Small but useful things

  • Configuring the name of export files is possible, as well as toggling whether screenshots are included and whether email notifications of export completion should be sent.
  • Exports are now available to Hobby plan users.
  • Screenshotting configuration is now a first-class citizen in the Configure menu (previously it was nestled within the Modules configuration section).
  • Many bug fixes, data quality improvements and small UI tweaks, not to mention a number of back-end stability and performance improvements.

Version 7 marks another milestone in the SpiderFoot HX journey, and there’s much more to come!