Machinae Security Intelligence Collector
Machinae is a tool for collecting intelligence from public sites/feeds about
various security-related pieces of data: IP addresses, domain names, URLs,
email addresses, file hashes and SSL fingerprints. It was inspired by
[Automater][1], another excellent tool for collecting information. The Machinae
project was born from wishing to improve Automater in 4 areas:
- Codebase - Bring Automater to python3 compatibility while making the code
more pythonic - Configuration - Use a more human readable configuration format (YAML)
- Inputs - Support JSON parsing out-of-the-box without the need to write
regular expressions, but still support regex scraping when needed - Outputs - Support additional output types, including JSON, while making
extraneous output optional
Installation
Machinae can be installed using pip3:
pip3 install machinae
Or, if you're feeling adventurous, can be installed directly from github:
pip3 install git+https://github.com/HurricaneLabs/machinae.git
You will need to have whatever dependencies are required on your system for
compiling Python modules (on Debian based systems, python3-dev), as well as
the libyaml development package (on Debian based systems, libyaml-dev).
You'll also want to grab the [latest configuration file][2] and place it in
/etc/machinae.yml.
Configuration File
Machinae supports a simple configuration merging system to allow you to make
adjustments to the configuration without modifying the machinae.yml we provide
you, making configuration updates a snap. This is done by finding a system-wide
default configuration (default /etc/machinae.yml), merging into that a
system-wide local configuration (/etc/machinae.local.yml) and finally a
per-user local configuration (~/.machinae.yml). The system-wide configuration
can also be located in the current working directory, can be set using the
MACHINAE_CONFIG environment variable, or of course by using the -c or
--config command line options. Configuration merging can be disabled by
passing the --nomerge option, which will cause Machinae to only load the
default system-wide configuration (or the one passed on the command line).
As an example of this, say you'd like to enable the Fortinet Category site,
which is disabled by default. You could modify /etc/machinae.yml, but these
changes would be overwritten by an update. Instead, you can put the following
in either /etc/machinae.local.yml or ~/.machinae.yml:
fortinet_classify:
default: true
Or, conversely, to disable a site, such as Virus Total pDNS:
vt_ip:
default: false
vt_domain:
default: false
Usage
Machinae usage is very similar to Automater:
usage: machinae [-h] [-c CONFIG] [--nomerge] [-d DELAY] [-f FILE] [-i INFILE] [-v]
[-o {D,J,N,S}] [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q]
[-s SITES] [-a AUTH] [-H HTTP_PROXY]
[--dump-config