Forecasting Tamil Nadu's Daily Peak Electricity Demand: SARIMA versus Gradient Boosting on Grid-India Data

Forecasting Tamil Nadu's Daily Peak Electricity Demand is an advanced data-science dissertation that builds and rigorously compares two families of forecasting models — classical seasonal ARIMA and gradient-boosted decision trees — on eleven years of publicly reported grid data for Tamil Nadu.

Every day, Grid-India (Power System Operation Corporation) publishes how much electricity each state actually consumed and how high its demand peaked. A research dataset scraped from those daily reports (Zenodo DOI 10.5281/zenodo.14983362, CC BY 4.0) provides daily energy met (MU), peak demand met (MW) and peak shortage (MW) for every Indian state from 2014 to 2024, bundled with matching daily weather variables such as temperature, dewpoint and solar radiation. This dissertation uses the Tamil Nadu series to answer a practical question: how far ahead can daily peak demand be forecast, and does a machine-learning model genuinely beat a well-tuned SARIMA baseline?

The study walks through the full data-science workflow: quality control of the raw series, decomposition of trend and multiple seasonalities (weekly, annual), feature engineering with lags and calendar variables, careful walk-forward backtesting at 1-, 7- and 14-day horizons, and error analysis of failure days (heat waves, holidays, cyclone outages). The deliverables — reproducible notebooks, a backtest report with MAPE/RMSE tables, and a written dissertation — make it a defensible M.Sc project for the Project Work & Viva-Voce slot, with no dependence on proprietary data or private code.

Tamil Nadu's grid operators must commit to generation schedules and day-ahead power purchases without knowing exactly what tomorrow's peak demand will be; under-forecasting forces expensive last-minute market purchases, while over-forecasting wastes contracted capacity. Published forecasting studies for Indian states rarely allow their claims to be checked: they use private or unshared data, evaluate with random train–test splits that leak future information into training, and seldom compare against a proper statistical baseline. There is no widely available, leakage-free, state-level evidence on how far ahead daily peak demand in Tamil Nadu can realistically be forecast and on whether machine-learning models genuinely outperform a well-specified SARIMA once evaluation is done honestly. This dissertation fills that gap with an openly licensed 11-year daily series, a pre-specified walk-forward protocol, and a failure-day analysis that tells operators when the models break, not just how often on average.

Abstract

State load dispatch centres must plan generation and day-ahead procurement against a demand they can only estimate. For Tamil Nadu — a state whose daily peak swings with monsoon temperatures, agricultural pumping and industrial load — even a small improvement in forecast accuracy translates into better procurement decisions and fewer emergency purchases. This dissertation develops daily peak-demand forecasting models for Tamil Nadu from an open, per-state dataset derived from Grid-India daily reports (2014–2024) and tests whether gradient boosting beats classical SARIMA at 1-, 7- and 14-day horizons under a walk-forward evaluation protocol.

Introduction

Electricity demand is one of the most studied forecasting problems in the time-series literature, yet state-level studies for India remain thin; most published Indian work uses either national series or single-city smart-meter data. Meanwhile, Grid-India has published daily state-wise supply statistics for over a decade, and a curated, quality-controlled version of those reports — including paired meteorological variables — is now openly licensed. That combination makes a reproducible state-level forecasting study possible for a master's student without access to utility SCADA systems.

Existing system / literature gap

Utility-grade short-term load forecasting research is dominated by (a) statistical models — ARIMA and its seasonal variants, exponential smoothing, and regression with weather terms — and (b) machine-learning models — random forests, gradient boosting and neural networks. Published comparisons disagree: ML models win on large, weather-sensitive series, while SARIMA remains competitive when the history is short or the evaluation is careless (random train–test splits that leak information). Indian state-level studies frequently suffer from exactly that flaw, and rarely release their data or code, so their claims cannot be checked. The gap this project fills: a documented, leakage-free, rolling-origin comparison on a public Tamil Nadu series, with the failure days analysed rather than hidden in an average error score.

Proposed study

• Data: the Zenodo dataset's Tamil_Nadu.csv — daily energy met (MU), peak met (MW), peak unmet (MW), and daily temperature, solar radiation and cloud cover, 2014–2024 — plus a calendar of Tamil Nadu public holidays.
• Models: seasonal-naive baseline; SARIMA with weekly and annual seasonality; random forest and gradient boosting (XGBoost) with lag, rolling-mean, holiday and weather features.
• Evaluation: rolling-origin walk-forward backtesting (retrain windows expanding yearly), 1-, 7- and 14-day horizons, scored with MAPE, RMSE and skill score against the seasonal-naive baseline, with error decomposition for the worst 20 forecast days.
• Deliverables: reproducible Jupyter/R Markdown notebooks, a results chapter with forecast-versus-actual plots, an error taxonomy (heat-wave days, monsoon transitions, holidays), and the dissertation report for the viva-voce.

Feasibility

• Technical: the dataset is a few megabytes of CSV; all models run on a standard laptop using statsmodels and scikit-learn/XGBoost. Nothing needs a GPU or a paid API.
• Economic: zero cost — the data is CC BY 4.0 licensed and every software component is open source.
• Operational: the study is self-contained secondary-data research; the only risk is minor gaps in the scraped series, which the quality-control work-package handles and documents.

In scope

• Single state (Tamil Nadu), single target variable for headline results: daily peak demand met (MW); daily energy met (MU) analysed as a secondary target.
• Public secondary data only: the Zenodo Grid-India-derived dataset (2014–2024) plus published holiday calendars.
• Model families: seasonal-naive baseline, SARIMA (statsmodels), random forest and XGBoost (scikit-learn/XGBoost).
• Horizons: 1, 7 and 14 days ahead; evaluation by rolling-origin walk-forward backtesting.
• Deliverables: reproducible notebooks, figures, dissertation, viva presentation.

Out of scope

• Intraday or hourly load forecasting (the public dataset is daily).
• Discom-level or feeder-level disaggregation; demand-response modelling; price forecasting.
• Deep-learning architectures (LSTM/transformer) — noted as future work; the comparison is deliberately classical-ML versus statistical to keep the experimental design clean.
• Rebuilding the dataset from raw Grid-India PDFs (the curated CC BY 4.0 version is used and cited).

Research design: quantitative, comparative time-series study with a pre-specified evaluation protocol (rolling-origin walk-forward backtesting). There is no human participant involvement, so no ethics clearance is required; the dataset is used under its CC BY 4.0 licence with attribution.

Instruments and analysis plan: Python (pandas, statsmodels, scikit-learn, XGBoost) in Jupyter notebooks with fixed random seeds; model comparison via a single scoring script so no result is computed twice differently. Sampling is a complete census of available days (~4,000), not a sample; every gap is documented rather than silently interpolated.

Although this is a research study rather than a software product, the workflow is still an engineered pipeline, and drawing it keeps the data flow honest — every arrow below is a notebook step that can be rerun from scratch.

flowchart TD
  A["Grid-India daily reports (source)"] --> B["Zenodo dataset 10.5281/zenodo.14983362"]
  B --> C["QC: missing days, duplicates, outliers"]
  C --> D["Clean Tamil Nadu daily table (2014–2024)"]
  D --> E["EDA: STL decomposition, ACF/PACF"]
  D --> F["Feature engineering: lags, rolling means, holiday and weather features"]
  E --> G["SARIMA (statsmodels)"]
  F --> H["Random forest + XGBoost (scikit-learn)"]
  F --> I["Seasonal-naive baseline"]
  G --> J["Walk-forward backtest: 1 / 7 / 14-day horizons"]
  H --> J
  I --> J
  J --> K["MAPE, RMSE, skill score tables"]
  J --> L["Failure-day taxonomy (heat wave, holiday, monsoon)"]
  K --> M["Dissertation results chapter + viva deck"]
  L --> M

Three design decisions are visible in the diagram. First, quality control sits before any modelling: the series was scraped from human-readable reports, so a documented QC log (which days are missing and why) protects every downstream number. Second, all three model families draw from the same cleaned table but not the same feature set — SARIMA uses only the demand history, while the tree models additionally use engineered lags and weather covariates — which is itself one of the study's comparisons. Third, evaluation is a single funnel: every model passes through the identical walk-forward harness, so differences in MAPE/RMSE reflect the models, not the harness. The pipeline is deliberately notebook-per-stage so the viva can demonstrate any stage in isolation.

• Add an LSTM/sequence model so the comparison spans statistical, tree-ensemble and deep-learning families on identical folds.
• Extend to all states in the dataset to ask a generalisation question: does the SARIMA-vs-boosting ranking hold across climatic zones?
• Disaggregate the study to intraday horizons when hourly data becomes openly available, and link shortage days (peak unmet) to forecast error.
• Build a small decision-support view that converts a 14-day peak forecast into day-ahead procurement hints, tested against historical market prices.