Evaluation of Open-Source IoT Firmware Vulnerability Scanners on a Curated Image Corpus

This dissertation is a defensive-security measurement study: it answers, with declared metrics, the question how much ground truth do open-source IoT firmware vulnerability scanners actually detect, and at what cost? Two mature scanner pipelines — EMBA and FACT (Firmware Analysis and Comparison Toolkit) — are executed over a curated corpus of freely redistributable firmware images, and every finding is checked against independently constructed ground truth.

The corpus is deliberately lab-safe. It contains (a) IoTGoat, the OWASP deliberately insecure router image whose planted weaknesses are published in advance, and (b) official OpenWrt release images whose component versions (BusyBox, OpenSSL, dropbear, dnsmasq) can be pinned to known CVEs in the National Vulnerability Database. No third-party or live device is ever touched: all work happens inside the student's own virtual machine on images downloaded from public repositories.

Findings are normalised into a CWE-tagged results table and scored on five declared metrics: detection rate against ground truth, precision of reported findings, CWE category coverage, wall-clock runtime per image, and repeatability across a second scripted run. The output is a defensible, paper-style comparison with tables and charts that a viva panel can interrogate line by line — exactly the research-grade contribution expected of an M.Tech Project Work Phase-2 dissertation.

Firmware vulnerability scanners are adopted on trust: teams deploy EMBA or FACT because their documentation sounds convincing, without knowing what proportion of known weaknesses each tool actually surfaces on a realistic corpus, how many findings are false leads, or how repeatable the tools are. Tool papers measure themselves on private samples with private metric definitions, so results are not comparable, and an incorrect belief that "the scanner covered it" creates a dangerous, silent gap in a defensive programme.

There is a need for an independent, methodologically declared evaluation: a curated corpus with independently constructed ground truth, an identical execution protocol for each scanner, and metrics — detection rate, precision, CWE coverage, runtime, repeatability — defined before the experiment begins. This dissertation supplies that evaluation in a form any college lab can rebuild, and turns the result into actionable guidance for practitioners choosing a free firmware-audit pipeline.

Abstract

Embedded and IoT devices ship with Linux-based firmware that is rarely patched, so the firmware image itself is where most weaknesses live. Open-source scanners such as EMBA and FACT automate the extraction, component identification and weakness flagging of such images, but each tool's documentation reports its own capabilities on its own samples. There is little independent, metric-declared evidence about how the free tools compare. This dissertation measures exactly that on a curated, legally redistributable corpus, using ground truth that is constructed independently of both tools.

Introduction

Firmware analysis has become a standard part of product-security assessment in India too, as CERT-In advisories and the growing IoT device base push organisations to audit what they deploy. Academic work, beginning with the large-scale static analysis of Costin et al. (USENIX Security 2014), showed that firmware images are a rich source of weaknesses. Tool papers since then claim ever-better automation. What a practising engineer needs to know is narrower and measurable: on images whose weaknesses we already know, what fraction does each free scanner find, how many of its reports are noise, what does it cost in time, and does it behave the same way on a re-run?

Existing system and gap

• Vendor documentation and benchmark blog posts report tool capabilities on cherry-picked samples, with no shared metric definition.
• Published academic analyses (Costin et al.; the FirmAE emulation study, ACSAC 2020) focus on scale or emulation success, not on a head-to-head scanner comparison against declared ground truth.
• No published study combines detection rate, precision, CWE coverage, runtime and repeatability on a corpus that any Indian college lab can legally rebuild.

Proposed study

• Build a versioned corpus: IoTGoat (published planted-weakness advisory as ground truth) plus selected OpenWrt release images (ground truth derived from component versions mapped to NVD CVEs).
• Execute EMBA and FACT under an identical, scripted protocol in a fixed Ubuntu 22.04 virtual machine; add a manual binwalk-based baseline triage to show what a scanner adds over raw extraction.
• Normalise all findings to CWE categories, score the five declared metrics, and publish every table so the experiment can be repeated and argued with.

Feasibility

• Technical: both tools are open source (GPL-3.0) and install on a 4-core, 8 GB RAM virtual machine; the corpus is freely downloadable; Python suffices for the analysis layer.
• Economic: zero licence cost; the only requirement is a standard lab PC.
• Ethical/legal: only public, redistributable images are analysed, inside the student's own VM; no live device, network or third-party system is scanned — the study is purely defensive and lab-safe.

In scope

• Firmware images that are publicly and legally redistributable: IoTGoat (OWASP) and official OpenWrt release images; corpus fixed and versioned before any scanner run.
• Two scanner pipelines (EMBA, FACT) plus a manual binwalk baseline; static analysis and each tool's own built-in emulation features as shipped.
• Ground truth limited to weaknesses knowable without exploiting anything: the IoTGoat advisory and version-to-CVE mappings in NVD.
• Metrics: detection rate, precision, CWE coverage, runtime, repeatability (two scripted runs per scanner).
• All work inside the student's own VirtualBox VM on a non-networked or lab-local setup.

Out of scope

• Scanning, probing or exploiting any live third-party device, network or web service; no penetration testing of systems the student does not own.
• Building or modifying a scanner; this is a measurement study of shipped tools.
• Commercial tools with licences that forbid such publication.
• Fuzzing, exploit development and any active attack technique.

The study uses a quantitative, comparative experimental design with metrics declared before execution (pre-registration logic adapted to a dissertation timeline). Every design choice exists to make the comparison fair and the result arguable: identical environment, identical corpus, scripted runs, independent ground truth, and a repeatability pass.

Protocol skeleton (fixed in writing before runs): one Ubuntu 22.04 LTS VM, 4 vCPU / 8 GB RAM, snapshot restored before every scanner run; corpus frozen at manifest v1.0; each scanner executed twice per image by a shell script that timestamps stdout/stderr; findings exported and normalised to a single CSV schema (image, tool, finding, CWE, path, component, confidence).

Metric definitions: detection rate = confirmed ground-truth items appearing in a tool's findings ÷ total ground-truth items; precision = findings confirmed against ground truth or manual inspection ÷ total findings; CWE coverage = distinct CWE categories found ÷ categories present in ground truth; runtime = wall-clock per image; repeatability = agreement between run 1 and run 2. Validation: the metric engine itself is unit-checked on a synthetic 20-row results file. Ethics: the protocol documents that only redistributable public images in the student's own VM are analysed; a one-page ethical-conduct note is included in the report as required by the research-methodology course (22RMI16).

The dissertation is organised as a measurement pipeline: a frozen corpus feeds identical scanner executions, all outputs flow into one normalised results store, and a metric engine joins that store with independently built ground truth. Nothing in the pipeline needs network access beyond the initial corpus download.

flowchart TD
  A["Public repositories
(IoTGoat, OpenWrt downloads)"] --> B["Versioned corpus manifest
SHA-256 + component versions"]
  B --> C["Extraction and unpacking
(binwalk / tool-internal)"]
  C --> D["Scanner executions
EMBA, FACT, binwalk baseline"]
  D --> E["Normalised findings store
CWE-tagged CSV"]
  B --> F["Ground truth build
IoTGoat advisory + NVD CVE mapping"]
  E --> G["Metric engine
(Python: detection rate, precision, runtime)"]
  F --> G
  G --> H["Repeatability pass
second scripted run"]
  H --> I["Results tables, charts and dissertation report"]

The manifest is the spine: every image and every ground-truth item is traceable to a URL, a hash and a version pin, so the viva panel can verify any row of any table. The harness guarantees fairness — same VM snapshot, same invocation, same output directory per tool — while the metric engine is a pure-Python program small enough to show and explain line by line in the viva. This is the classic measurement-study shape: pipeline in, numbers out, every number reproducible.

• Add dynamic analysis coverage: run each tool's built-in emulation features and compare static-only versus emulated detection on the same corpus.
• Extend the corpus to GPL-published vendor firmware dumps and measure how detection degrades when ground truth must be built purely from NVD mapping.
• Track the tools across releases: rerun the frozen corpus against EMBA/FACT versions one year apart to measure scanner improvement rates.
• Package the metric engine and harness as a maintained open-source benchmark so successive student batches can grow the corpus and results table.