Evaluating Training Effectiveness with the Kirkpatrick Four-Level Model at a Pune IT Services Firm (SIP, SPSS)

Evaluating Training Effectiveness with the Kirkpatrick Four-Level Model is a Summer Internship Project carried out in the Learning & Development (L&D) function of a modelled, mid-sized IT services firm in Pune (fictional name: Deccanware Technologies, stated as fictional in the report). Indian companies spend heavily on training — onboarding batches, client-communication workshops, first-line leadership programmes — yet most evaluations stop at the happy-sheet: a reaction form filled in the last ten minutes of the session.

This study climbs the full Kirkpatrick chain of evidence for three real programmes — technical onboarding, client communication, and first-line leadership — across about 220 employees:

• Level 1 (Reaction): standardised five-item feedback sheets after every batch.
• Level 2 (Learning): parallel-form knowledge tests before and after each programme, scored out of 20.
• Level 3 (Behaviour): a six-behaviour supervisor checklist rated 8–10 weeks after training, when the learning should be visible at the desk.
• Level 4 (Results): proxy business indicators (defect leakage, first-pass resolution, client escalations) plus six manager interviews explaining what the numbers can and cannot prove.

The data is analysed in IBM SPSS Statistics: paired t-tests for pre–post learning gains, one-way ANOVA across programmes, Pearson correlation between reaction and behaviour scores, and multiple regression predicting behaviour change. A Tableau Public scorecard shows each programme's four-level chain so the L&D head can see, batch by batch, where the chain breaks. The deliverable is an evidence-based redesign memo, not another stack of feedback forms.

The L&D function of the modelled firm Deccanware Technologies evaluates its training calendar with end-of-session reaction forms and attendance data. This practice has three consequences. First, the firm cannot tell whether its technical onboarding, client-communication and leadership programmes produce measurable learning, because no pre- and post-training assessment exists. Second, it cannot show behaviour change on the floor, because no supervisor observation happens after the classroom ends — so weak programmes survive purely on good feedback scores. Third, the annual budget discussion collapses into an argument about satisfaction percentages, with no chain of evidence connecting training to defects, escalations or team metrics.

Published Kirkpatrick studies in India rarely measure all four levels inside one firm or test whether the levels actually correlate, so the firm has no benchmark design to copy. There is a need for a structured, statistics-backed evaluation that measures reaction, learning, behaviour and results for each programme, tests the relationships between levels, and tells the L&D head exactly where each programme's evidence chain breaks — before the next budget cycle.

Abstract

This SIP evaluates whether three training programmes at a modelled Pune IT services firm actually produce learning, behaviour change and business-relevant results, using the Kirkpatrick four-level framework as the measurement backbone. Pre–post tests, reaction sheets, supervisor ratings and manager interviews are combined and analysed in SPSS to locate where each programme's evidence chain holds and where it breaks.

Introduction

Deccanware Technologies (a modelled firm with ~1,400 employees) runs a continuous training calendar: a six-week technical onboarding for fresh joiners, a two-day client-communication workshop, and a twelve-week first-line leadership programme for team leads. The annual L&D budget is defended every April with two artefacts: attendance sheets and average feedback-form scores. Neither answers the question the CFO actually asks — did anyone's work change?

Kirkpatrick's four levels — Reaction, Learning, Behaviour and Results — remain the most widely taught framework for answering exactly that question (Kirkpatrick & Kirkpatrick, 2006; kirkpatrickpartners.com). The model's real value is not the four boxes but the chain of evidence: positive reactions do not guarantee learning, learning does not guarantee application, and application does not guarantee results. This SIP measures each link for the three programmes.

Existing system / literature gap

• The firm's current evaluation is a Level-1-only practice: five feedback items, averaged into a percentage, with no learning or behaviour measurement.
• Indian published studies on Kirkpatrick tend to evaluate a single programme or a single level; few measure all four levels for one organisation in one study, and fewer still test statistically whether reaction scores actually predict anything downstream.
• Skills-policy literature (NSDC, Skill India Digital) frames skilling at the national level but leaves firms without a low-cost, repeatable internal measurement design.

The proposed study

A single-group pre–post design across three programmes: Level-1 reaction sheets (n ≈ 214), parallel-form pre/post knowledge tests (≈ 180 complete pairs), supervisor behaviour checklists 8–10 weeks later (≈ 150 employees), and modelled Level-4 proxy indicators plus six manager interviews. SPSS analysis: paired t-tests (learning gain), ANOVA (programme differences), Pearson correlation (reaction vs learning gain; learning gain vs behaviour change) and multiple regression on the behaviour-change score. Findings feed a programme-wise redesign memo for the L&D head.

Feasibility

• Technical: every instrument is paper or spreadsheet based; SPSS (GE-IL-11) and Excel (GE-IL-04) are part of the SPPU curriculum; Tableau Public is free.
• Economic: no licensing or external survey cost; printing and travel only (under ₹1,000).
• Operational: instruments ride on sessions that already happen; supervisors spend 6–8 minutes per checklist; no employee-identifying data leaves the firm, and reports use employee IDs only.

In scope

• Three programmes at the modelled firm: technical onboarding (fresh joiners), client-communication workshop, and first-line leadership programme.
• Employees trained during one eight-week observation window (~220 trainees) and their supervisors.
• All four Kirkpatrick levels: reaction sheets, parallel-form pre/post tests, supervisor behaviour checklists, and Level-4 proxy indicators from the firm's modelled quality data plus six manager interviews.
• Statistical analysis in SPSS of the single-group pre–post design described in the methodology.

Out of scope

• Phillips's ROI (fifth-level) computation — cost data is not disclosed to the intern; the framework is discussed as future work.
• Programmes outside the three named ones (compliance e-learning, external certifications).
• A control group or random assignment — the firm will not withhold training for research purposes; this threat to validity is discussed openly instead.
• Individual performance appraisal data, compensation, or any personally identifying information beyond anonymised employee IDs.

Research design: quantitative-dominant mixed-methods study using a single-group pre–post design inside an eight-week internship window, with the Kirkpatrick four levels as the measurement framework and the firm as a single-case organisation.

Population and sampling: census approach — every employee completing any of the three programmes during the window (target ≈ 220) is invited; participation is voluntary. For Level 3, the direct supervisor of each trained employee completes one checklist. Six purposively selected managers (delivery, quality, L&D) are interviewed for the Level-4 triangulation.

Instruments: (1) a standardised five-item reaction sheet (relevance, clarity, facilitator, materials, confidence to apply), replacing the firm's ad-hoc form; (2) parallel-form knowledge tests (20 marks, MCQ + one scenario), Form A before training and Form B after, built from the same blueprint to avoid memory carry-over; (3) a six-behaviour supervisor checklist (e.g., runs peer code review, uses structured escalation, holds stand-ups) on a 5-point frequency scale; (4) a semi-structured manager interview guide.

Hypotheses: H1: post-test mean exceeds pre-test mean per programme (paired t-test). H2: reaction scores differ across programmes (ANOVA). H3: learning gain correlates positively with behaviour-change score (Pearson). H4: behaviour-change score is predicted by programme, tenure, reaction score and learning gain (multiple regression, α = 0.05).

Ethics: consent from the firm's HR head; employee IDs instead of names; supervisors rate behaviour, not personality; interview quotes anonymised; no data used for appraisal decisions.

The evaluation is a chain-of-evidence pipeline: each Kirkpatrick level feeds the next, and the analysis tests whether the links actually hold for this firm. Manager interviews sit outside the quantitative chain and are used to explain the Level-4 numbers.

flowchart TD
  A["Training calendar: onboarding, communication, leadership"] --> B["Level 1: reaction sheet after every batch"]
  A --> C["Level 2: pre-test Form A and post-test Form B"]
  B --> D["Excel: match records by employee ID"]
  C --> D
  D --> E["SPSS: paired t-test for learning gains"]
  E --> F["Level 3: supervisor checklist after 8 to 10 weeks"]
  F --> G["SPSS: ANOVA, correlation, regression on behaviour change"]
  G --> H["Level 4: proxy quality indicators"]
  H --> I["Manager interviews: 6 thematic codes"]
  I --> J["Where does each chain break?"]
  G --> J
  J --> K["Programme redesign memo for L&D head"]

Measurement framework

flowchart LR
  L1["L1 Reaction: 5-item sheet"] --> L2["L2 Learning: test gain"]
  L2 --> L3["L3 Behaviour: supervisor rating"]
  L3 --> L4["L4 Results: defect leakage, escalations"]
  M["Manager interviews"] --> L4

Design notes. Matching pre-test, post-test and checklist records by employee ID is the critical data-hygiene step, done in Excel with a lookup before any SPSS test. Parallel forms reduce test–retest memory effects; the single-group design cannot rule out history and maturation, which is declared as a validity threat and mitigated by the 8–10-week gap and the interview triangulation. Internal consistency (Cronbach's alpha ≥ 0.7) is verified for every multi-item scale in the pilot.

• Phillips ROI level: with disclosed cost data, add the fifth-level ROI computation (benefits vs programme cost) for the onboarding programme.
• Control-group design: rotate the programme across two batches a quarter apart so one batch serves as an untreated comparison, strengthening causal claims.
• Behaviour decay tracking: repeat the supervisor checklist at 6 and 12 months to measure how long behaviour change lasts.
• Digital evaluation: move the reaction sheet and checklist onto the firm's HRMS so every batch is evaluated by design rather than by intern effort.
• Cross-firm benchmarking: replicate the design in two other Pune IT firms to build a Pune-level comparison of L&D maturity.