Privacy-Preserving Log Anonymisation Pipeline for a College Web Server (Python/MySQL)

PrivLog: a Privacy-Preserving Log Anonymisation Pipeline sits between a college web server and its log archive, and solves a problem every Indian campus now has: the logs the law requires you to keep are the same logs that hold your students' personal data.

Since CERT-In's Directions of 28 April 2022, organisations must retain computer logs for a rolling 180 days (and report incidents within six hours). Since the Digital Personal Data Protection Act, 2023, students are data principals whose roll numbers, email addresses and device identifiers in those logs deserve minimisation and protection. Most college servers simply keep raw logs — IP addresses, /results/<roll-number> URLs, email addresses in query strings — on a badly protected folder.

This project builds the missing layer: a Python pipeline that tails the web server's access log, detects identifiers (IPv4 addresses, emails, phone numbers, the college's roll-number pattern, high-cardinality user agents), and transforms them — keyed HMAC-SHA256 pseudonyms with the key held outside the database, IP-address truncation, user-agent generalisation — before anything is stored long-term. A verification gate re-scans the output for leaked identifiers and enforces a k-anonymity threshold on analyst exports. A Flask dashboard shows before/after utility metrics, proving the anonymised logs still answer operational questions (top pages, hourly load, error rates) while carrying no direct identifiers.

Python and MySQL come straight from the Anna University R2021 MCA curriculum (MC4103, MC4202); Flask and Chart.js are clearly-labelled industry-standard extras. Two students can complete it in one semester, with clean per-member module boundaries.

Indian colleges must keep web-server logs for 180 days under CERT-In's Directions of April 2022, yet the same logs embed students' personal data — device IP addresses, user agents precise enough to fingerprint a device, and roll numbers or email addresses embedded in URLs. Retained raw, the log archive becomes the single most sensitive file on campus: an unencrypted, months-long record of which student accessed which result or fee page, from where and when. The Digital Personal Data Protection Act, 2023 makes such casual retention of identifiable data hard to justify, while deleting logs early would break incident response and CERT-In's six-hour reporting expectation.

There is no tooling in a typical college for this: no inventory of what identifiers the logs contain, no separation between the audit purpose (incident investigation) and the analyst purpose (student projects, capacity planning), and no measurable guarantee that shared datasets are actually de-identified. What is needed is a pipeline that anonymises at ingestion time, retains the operational value of the log trail, proves the transformation with automated verification, and gives the college two defensible profiles — audit-grade and analyst-grade — for its two distinct purposes.

Abstract

A college web server's access log is simultaneously a security record and a personal-data liability. CERT-In's Directions of 28 April 2022 require a rolling 180-day log retention, while the Digital Personal Data Protection Act, 2023 imposes purpose-limitation and storage-minimisation duties on anyone processing students' personal data. This project builds a pipeline that keeps the operational and forensic value of web-server logs while stripping the personal data out: it detects direct and quasi-identifiers, transforms them (keyed pseudonyms, truncation, generalisation), verifies the output with a leakage scan and a k-anonymity gate, and stores only anonymised events in MySQL, with a dashboard measuring surviving analytical utility.

Introduction

A typical affiliated-college results portal serves thousands of requests a day. Every request line records the device's IP address, a fingerprintable user-agent string, and URLs embedding roll numbers (/results/2021cs1042.pdf) or occasional email addresses. Retained raw for 180 days, it is an unencrypted dossier of who checked which result, from where and when. The college cannot stop logging — incident response depends on it — so the data must be transformed before storage.

Existing System

• Raw logs are rotated and kept on the server for months, readable by anyone with the account.
• No inventory of what the logs contain — roll numbers, emails, device identifiers.
• Project students asking for logs get a raw copy emailed.
• Deletion policies swing between "keep everything forever" and "delete after a week", neither compliant.
• Limitations: no identifier inventory, no separation of audit needs from analyst needs, no measurable privacy or utility guarantee.

Proposed System

• Ingest: tail the access log, parse Common Log Format lines into structured events in a short-lived staging table.
• Classify: pluggable detectors find IPv4 addresses, emails, phone numbers, the college's roll-number pattern and user-agent strings, recording type, position and count per run.
• Transform: keyed HMAC-SHA256 pseudonyms (key in a separate keystore file) for roll numbers and IPs; last-octet truncation; user agents generalised to browser-and-OS family; free-text identifiers redacted.
• Verify: a leakage gate re-runs every detector on the output and blocks the batch if anything survives; analyst exports additionally pass a k-anonymity check (k ≥ 5).
• Store and serve: only anonymised events land in MySQL; a Flask dashboard shows per-run findings and before/after utility metrics.
• Configure: two profiles — audit-grade (reversible only with the keystore) and analyst-grade (one-way, generalised, k-checked) — matching the two lawful purposes.

Feasibility Study

• Technical: Python 3, MySQL 8 and Flask are free and run on the college's existing server; the pipeline is a batch job needing no web-server changes.
• Economic: zero licence cost; the only new artefact is the permission-protected keystore file.
• Operational: the IT section keeps its 180-day trail anonymised, project students get safe datasets, and the dashboard replaces manual grepping.
• Ethical and legal: the bundle's development dataset is fully synthetic; real logs are processed only on the college's own server with written permission, and raw events are purged after the utility comparison.

In scope

• The college's own web server access log (Common Log Format / combined format) as the single source; file-based ingestion only.
• Detector and transformer coverage for IP addresses, emails, phone numbers, one configurable roll-number pattern and user agents.
• Two anonymisation profiles (audit-grade, analyst-grade) with the keystore file kept separately from the database.
• Verification gate (leakage scan and k-anonymity check), utility measurement, dashboard and CSV export.
• A synthetic log generator so the whole pipeline can be demonstrated without touching real student data.

Out of scope (v1)

• Network-level capture or DNS/proxy logs; only the web server's file log is processed.
• Re-identification attacks against other people's datasets — the study measures risk on our own outputs only.
• Formal differential-privacy noise; k-anonymity and generalisation are the v1 guarantees.
• Real-time streaming processing; the pipeline runs on rotation intervals (five-minute batch).

The project follows an incremental SDLC model with fortnightly increments, so a working pipeline exists from week 3 and each review adds capability. Both members develop against the same MySQL schema from week 2.

Testing approach: pytest unit tests per detector and transformer; a property-based test asserting that no detector ever matches on pipeline output, whatever the input; a utility regression comparing aggregate reports before and after anonymisation on the same synthetic day; and a performance run confirming one million log lines process in under ten minutes on the college's existing server.

PrivLog is a batch pipeline with a hard verification gate, deliberately placed so that raw events never reach long-term storage:

1. Ingestion — a tailer follows the access log on the college's own server and hands raw lines to the parser in batches.
2. Parsing & staging — Common Log Format lines are parsed into structured events and landed in a staging table that is purged after each run's utility comparison.
3. Classification — the detector framework scans every field; each match is recorded in a findings table (type, position, sample-hashed value) so the report can show exactly what the logs contained.
4. Transformation — keyed HMAC-SHA256 pseudonyms for roll numbers and IPs (key loaded from the keystore file, never stored in MySQL), last-octet truncation, user-agent generalisation to browser-and-OS family, redaction for emails and phones.
5. Verification gate — the leakage scan re-runs all detectors on transformed output; the k-anonymity check applies to analyst-grade exports; a failed check blocks the batch and raises a run error.
6. Storage & presentation — anonymised events land in MySQL; a Flask dashboard shows findings, before/after utility charts and run history; exports are CSV.

flowchart LR
  L["access.log on college web server"] --> T[Tailer]
  T --> P[Parser and stager]
  P --> S[(Staging table - purged per run)]
  S --> D[Detector framework]
  D --> F[(Findings)]
  D --> X[Transformation engine]
  K[("Keystore file (HMAC key)")] --> X
  X --> G{Verification gate}
  G -->|identifier found| B[Block batch and raise error]
  G -->|clean| A[(MySQL - anonymised events)]
  A --> U[Utility metrics]
  U --> DB2[(Dashboard - Flask)]
  A --> E["Analyst export (k-anonymity checked)"]

ER diagram (core tables)

erDiagram
  PIPELINE_RUN ||--o{ EVENT : produces
  PIPELINE_RUN ||--o{ FINDING : records
  PIPELINE_RUN ||--o{ METRIC : computes
  CONFIG_PROFILE ||--o{ PIPELINE_RUN : "configures"
  EVENT ||--o{ EXPORT_ROW : "feeds"
  EVENT {
    bigint id PK
    datetime ts
    string ip_prefix
    string ip_pseudonym
    string ua_family
    string path_pseudonymised
    int status
    string method
  }
  FINDING {
    bigint id PK
    bigint run_id FK
    string id_type
    string field
    int count
  }
  PIPELINE_RUN {
    bigint id PK
    datetime started_at
    string profile
    string status
    int lines_in
  }
  CONFIG_PROFILE {
    string name PK
    string pseudonym_mode
    bool k_check
    int k_value
  }
  METRIC {
    bigint id PK
    bigint run_id FK
    string metric_name
    decimal raw_value
    decimal anon_value
  }
  EXPORT_ROW {
    bigint id PK
    bigint event_id FK
    string export_id
  }

The key design decision is purpose separation: audit-grade pseudonyms are reversible only by whoever holds the keystore (for incident response), while analyst-grade data is one-way, generalised and k-checked. One pipeline, two lawful purposes, nothing shared by accident.

• Streaming mode with a line-by-line gate for very high-traffic servers, keeping the same detector and transformer interfaces.
• Differential-privacy option for aggregate dashboards, complementing k-anonymity on exports.
• Additional log sources — DNS query logs and proxy logs through the same pipeline, widening campus coverage.
• Retention automation — scheduled 180-day purge jobs with signed deletion certificates for the college's compliance file.
• Re-identification risk scoring — publish a quantified risk estimate (prosecutor/journalist models) alongside every analyst export.
• A deployment guide for other departments, so the pipeline becomes shared campus infrastructure rather than a one-semester project.