Secrets-Leak Scanner & Rotation Tracker for Git Repositories (Python/MySQL)

KeyRing: a Secrets-Leak Scanner & Rotation Tracker is an individual, defensive-security project built on a problem every student developer has already created for themselves: passwords, API keys, database URLs and tokens committed into Git repositories. Deleting the file does not help — the secret lives in history forever, in every clone of the repository, and leaked credentials are among the most common root causes of cloud-account misuse.

KeyRing does two things no off-the-shelf tool does together. First, it scans: a Python engine walks the full history of repositories you own (or are authorised to scan), extracts candidates from every diff, and matches them against signature rules — cloud access-key formats, private-key blocks, database connection strings, generic password = assignments — filtered by Shannon entropy to keep false positives low. The findings store never saves the secret itself: only a salted SHA-256 digest, a masked snippet and the commit locator. Second, it tracks: each confirmed finding becomes a rotation ticket — Open, Credential Revoked, Secret Rotated, Verified, Closed — with owner, evidence and timestamps, plus a pre-commit hook that blocks future leaks at commit time and a baseline mode so legacy findings get deadlines instead of alarms.

The bundle ships with demonstration repositories containing only documented example credentials and dummy values, so the entire workflow can be run and defended safely on your own laptop. Python, JavaScript and MySQL are exactly the languages and tools in the IGNOU MCA_NEW curriculum; Flask and Chart.js are labelled industry-standard extras on top.

When a credential ends up in a Git commit — a database password, a cloud access key, a payment-gateway secret — the usual student response is to delete the file and commit again. That fixes nothing: the secret remains in repository history, in every clone and fork, and credential leakage remains one of the most common precursors of account misuse. Manual grepping finds only what the searcher already suspects, working-tree-only scripts miss history entirely, and the raw output of even good open-source scanners is a wall of alerts with no path to closure.

The deeper problem is remediation, not detection: a finding only matters if the credential is revoked, replaced and verified, and there is usually no record that any of that happened. Examiners and employers alike ask the question student projects cannot answer — "you found three leaked keys in June; show me they are dead today". What is needed is a small, explainable toolchain that (1) scans full repository history with a low false-positive rate, (2) never stores the secrets it finds, (3) turns each confirmed finding into a tracked, time-stamped rotation workflow, and (4) prevents new leaks at commit time — all running safely on the developer's own machine and own repositories.

Abstract

Credentials committed to Git repositories remain recoverable in history forever, and leaked keys are a routine root cause of account misuse. This project builds KeyRing, a tool for the author's own repositories that (1) scans full Git history for credential-shaped strings with signature rules and an entropy filter, (2) stores findings without persisting the secret, (3) drives every confirmed leak through a tracked rotation workflow to verified closure, and (4) prevents new leaks with a pre-commit hook. It follows the Secure Software Development Framework's practice of building security into tooling and produces measurable results: findings inventory, false-positive rate, rotation closure statistics.

Introduction

Student repositories carry the same cargo as corporate ones: a database password in settings.py, a payment-gateway test key pasted during debugging, a private key committed "temporarily". Version control preserves every version of every file, so deleting a secret in the latest commit changes nothing about clones that already exist. Detection therefore has to work on history, and remediation must be tracked to closure — usually two separate commercial products a student can install but never explain. This project builds both.

Existing System

• Manual practice: grep, delete in a new commit, assume it is gone — history keeps it.
• Ad-hoc scripts check only the working tree, never history, and produce false-positive walls.
• Open-source scanners (gitleaks, TruffleHog) are excellent detectors but leave remediation — severity, rotation, proof of closure — to the user.
• Limitations: no history depth, no entropy filtering, no rotation workflow, no prevention, no audit trail.

Proposed System

• Scan: a history walker runs git log -p per repository and feeds added lines from the diffs to the rule engine.
• Detect: two-stage rules — signature regexes for known formats (cloud keys, private-key blocks, connection strings, token prefixes) and a generic assignment rule gated by Shannon entropy; every rule carries a severity.
• Store safely: MySQL keeps repo, commit, rule, a salted SHA-256 digest and a masked snippet — never the secret.
• Track: each confirmed finding opens a rotation ticket with owner and target date; statuses Open → Credential Revoked → Secret Rotated → Verified → Closed produce a days-open SLA report.
• Prevent: a pre-commit hook scans staged changes and rejects commits containing candidates; a .secretsignore allowlist covers fixtures.
• Baseline: the first scan of a legacy repository records existing findings with remediation deadlines; later scans fail only on new leaks.

Feasibility Study

• Technical: Python 3, Git and MySQL 8 are free; the scanner is a laptop-scale batch job needing no cloud services.
• Economic: zero licence cost; demo repositories carry documented example credentials only, so nothing real is at risk.
• Operational: one student scans all their own repositories in minutes; the ticket workflow replaces the "I rotated it somewhere last month" memory.
• Ethical and legal: scanning targets only repositories the author owns or is permitted to scan; the tool never contacts any external service, and found secrets are masked everywhere, including screenshots.

In scope

• Scanning Git repositories the author owns, or a college repository with written permission — local clones only.
• Detection coverage for cloud access-key formats, private-key blocks, database and service connection strings, common token prefixes and generic high-entropy assignments.
• A MySQL findings store that never persists a secret, plus a rotation ticket workflow with evidence tracking.
• A pre-commit hook, baseline mode and an HTML dashboard with CSV export.
• Demonstration repositories containing only documented example credentials and dummy values.

Out of scope (v1)

• Scanning any repository, host or account the author does not own or is not authorised to scan.
• Automated provider-side revocation API calls — rotation is tracked and evidenced manually in v1.
• Memory-scan, filesystem-scan or CI-service integration beyond the pre-commit hook.
• Brute-force or validation of found credentials against any service — found secrets are treated as compromised and rotated, never tested.

The project follows an incremental SDLC model over 16 weeks, matching the MCSP-232 expectation that the proposal (with DFD, ER and PERT/Gantt charts) is approved before development begins.

Testing approach: pytest unit tests per rule with fixture files using documented example credentials; a property test that no digest ever equals a digest of an unsalted fixture; integration tests that clone the demo repositories, run a full scan and assert exact finding counts; hook tests that attempt a leaky commit and assert it is rejected; and a performance run on a 60,000-commit repository to validate throughput. Every rule change re-runs the corpus, so false-positive claims are regression-tested rather than anecdotal.

KeyRing is a four-stage pipeline with a prevention loop, all local:

1. History walker — for each repository, the tool runs git log -p --no-renames and parses added lines per commit, recording commit hash, author date and file path; incremental scans remember the last scanned commit so re-scans are cheap.
2. Detection engine — added lines pass through the rule set: signature regexes first (specific formats with their severities), then the generic assignment rule, where a candidate must clear a Shannon-entropy threshold before it is reported. Matches are deduplicated by content digest so one secret leaked in fifty commits is one finding with fifty locators.
3. Findings store — MySQL persists repo, rule, severity, per-install-salted digest, masked snippet and locators. The salt lives in the tool's config file, so a leaked database dump cannot be reversed into secrets by matching public digests.
4. Workflow and prevention — the Flask dashboard drives the ticket state machine (Open → Credential Revoked → Secret Rotated → Verified → Closed), reports days-open per ticket, and the pre-commit hook re-uses the same engine on staged diffs, so detection and prevention can never drift apart.

flowchart LR
  R[("Repos you own (local clones)")] --> W[History walker]
  W -->|added lines| E[Detection engine]
  RS[("rules.yaml + entropy filter")] --> E
  E -->|dedupe by digest| F[(MySQL findings - masked only)]
  F --> T[Ticket workflow]
  T --> ST["Open - Revoked - Rotated - Verified - Closed"]
  T --> SLA[SLA and closure report]
  H[Pre-commit hook] -->|staged diff| E
  H -->|block commit| DEV[Developer]
  F --> D[Flask dashboard]

DFD level 2 — decomposition of the Scan process

flowchart TD
  U([Developer]) -->|repo path| P21[2.1 Walk commits]
  P21 -->|commit diffs| P22[2.2 Parse added lines]
  P22 -->|candidates| P23[2.3 Match signature rules]
  P23 -->|remaining candidates| P24[2.4 Entropy filter and dedupe]
  P24 -->|confirmed findings| P25[2.5 Mask and persist]
  P25 --> D1[(D1 Findings)]
  P23 --> D2[(D2 Rules)]
  P24 --> D2
  P25 -->|finding id| U

ER diagram (core tables)

erDiagram
  REPO ||--o{ COMMIT : contains
  COMMIT ||--o{ FINDING : "yielded"
  RULE ||--o{ FINDING : "matched"
  FINDING ||--|| TICKET : "has one"
  TICKET ||--o{ TICKET_EVENT : "is moved by"
  REPO {
    int id PK
    string name UK
    string path
    datetime baseline_at
  }
  COMMIT {
    string sha PK
    int repo_id FK
    datetime authored_at
    string author
  }
  RULE {
    string id PK
    string name
    string pattern_kind
    string severity
  }
  FINDING {
    bigint id PK
    int repo_id FK
    string rule_id FK
    string digest_sha256
    string masked_snippet
    string status
  }
  TICKET {
    bigint id PK
    bigint finding_id FK
    string owner
    date target_date
    string status
  }
  TICKET_EVENT {
    bigint id PK
    bigint ticket_id FK
    string from_status
    string to_status
    string evidence_link
    datetime moved_at
  }

The security-critical design decision is that the store is safe by construction: a full database dump reveals which findings existed and how they were handled, but not a single usable secret — the masking and salting make the evidence store strictly less sensitive than the repositories it watches.

• Provider checklists with evidence templates — guided, provider-specific rotation checklists (cloud consoles, database users, gateway dashboards) that generate the evidence link automatically.
• CI integration — the same engine as a pipeline job that fails builds on new findings, sharing the baseline database.
• Push protection — a server-side receive hook on a self-hosted Git server, blocking leaks before they enter history at all.
• Entropy threshold learning — per-repository calibration from historical findings to cut false positives further.
• Organisation mode — multi-repo scoring for a college lab or a small team, with per-owner ticket queues.
• SARIF output so the dashboard's findings can be consumed by code-review tools.