So here’s a rundown of the software that we would add to a new system that is to be used to gather and store intelligence.
We’re starting our own system built on Ubuntu, however the programs are cross-platform so the list is applicable to any OS. None of the packages require a paid subscription, but most require registration of some kind.
We won’t delve into learning systems here, except to say that we would tend to install Jupyter Notebook editing tools - these are available as part of the other programs mentioned here.
I’ll be uploading and documenting a lot of the evidence that we’ve gathered and analysed using these tools over the coming months. For the time being however, I’m going to focus on the engineering environment and toolchain. I’ll also be documenting the process of creating and uploading the website, built on static files generated using gatsby.
File and source editors
One major task is file editing, specifically code files, and for this we use the same tools as a senior developer, and they are just basic editors with many extra features that we need, such as html/css/js, markdown, json, and so forth.
The text editor software that we use mainly are: vim, atom, and VS-Code. For actual development projects, Visual Studio and eclipse are used.
For images, GIMP and, on windows, paint.net. The video editors are OpenShot, which also installs InkScape and Blender, and on windows, VSDC Free Video Editor.
For live streaming, OBS Studio is used.
IQ Tooling
These are the tools used to collect, store and analyse our evidence.
AgentDVR is the system we use to monitor feeds from video sources such as CCTV, and it includes a wide range of complex tooling, including object and person detection, and the ability to connect to your own models on the local machine or remotely.
For gathering intel on websites and networks, there are thousands of tools. Spiderfoot is simple to use and has enough features to make it worthwhile. This is one tool where I like to use the bleeding edge from github master. Alternatively, it can run in docker, and there is also a free trial of the paid version.
Maltego Casefile is a proprietary product, with a community edition that is free of charge. It’s primarily a graphing tool, but it includes a wide range of plugin filters that enable datamining, network and domain analysis, and many more features.
For storage, we use a continual process of forking and merging, ensuring that vital data is replicated multiple times both for ease of retrieval and for protection from accidental or deliberate erasure.
Per-machine settings
For processor, always prefer Intel for reliability. Other parts vary per manufacturer/model and are evaluated in place and replaced where necessary.
No particular OS is preferred, however when installing from scratch Ubuntu is normally used until it becomes optimal to use something else. On windows, we install MSYS2 (which includes bash
and linux tools) by default and setup openssh to allow access over the network. For convenience we may use docker or VirtualBox, however guest machines are best handled using either qemu or lxc. A dedicated node would be managed using proxmox.
Nginx is installed by default, mainly for previewing pre-generated websites. Nginx is used on the server too in conjunction with php-fpm and mariadb. It’s possible to setup free-tier servers on amazon-ecs, google cloud, and others.
We typically install as many different web browsers as possible, but on a research machine, ensure that they logout after each session. As well as the standard bunch, we also use: Tor browser bundle for the convenience of having a random proxy available and for instant websites from your local just by editing a few files. Brave browser, for accessing NFT domains and for metamask wallet.
The same applies to crypto wallets, to a lesser extent. Each user should choose their own.
Each machine is given a new gmail, hotmail, and twitter address. Files may be transfered from a remote location using GoogleDrive or OneDrive, and locally using filezilla over sftp.
Finally, for completion we add bitcoin and liecoin core in case we need them.