Sentiment Analysis of Telugu–English Code-Mixed Social-Media Text: Multilingual Transformers vs BiLSTM Baselines

Telugu speakers on social media rarely write in one language or one script. A typical comment under a Telugu film trailer or a Hyderabad news video mixes Romanised Telugu, English words and occasionally Telugu script — "movie asalu bagaledu, waste of time" or "BGM keka 🔥". Standard sentiment tools, trained on monolingual English or on Telugu-script text, handle such input poorly, and publicly available labelled Telugu–English code-mixed data is scarce compared with Tamil, Malayalam or Hindi.

This dissertation does two things. First, it constructs a sentiment corpus of public Telugu–English code-mixed comments with written annotation guidelines, two independent annotators and inter-annotator agreement measured by Cohen's kappa. Second, it benchmarks classical and neural approaches on that corpus: TF-IDF with a linear SVM, a BiLSTM with FastText subword embeddings, and fine-tuned pretrained multilingual transformers — multilingual BERT, XLM-RoBERTa and MuRIL — using macro-F1 over repeated runs, followed by a structured error analysis by code-mixing level, script, negation and sarcasm.

The work applies the Deep Learning and NLP electives of the JNTUH R22 M.Tech programme and is implemented in Python with PyTorch and scikit-learn; Hugging Face Transformers is used as an industry-standard extra. Ethics are built in: only public comments are collected through official interfaces, user handles are removed, and no personal data is stored. Results are produced by the student's own experiments — this bundle provides the protocol and empty results tables.

Abstract

Code-mixed text — language alternation within a sentence — is the norm in Indian social media, but sentiment resources for Telugu–English remain limited. This work builds an annotated corpus of Telugu–English code-mixed comments (target 12,000 items, three sentiment classes plus a not-Telugu filter label) with documented guidelines and measured inter-annotator agreement, and compares five models: TF-IDF + linear SVM, BiLSTM with FastText embeddings, multilingual BERT, XLM-RoBERTa base and MuRIL. Models are evaluated with macro-F1 over five seeds on a fixed test split, with statistical significance testing and error analysis stratified by code-mixing index and script.

Introduction

Pretrained multilingual encoders have changed multilingual NLP, but most are pretrained on native-script text, while Telugu users frequently type in Roman script with inconsistent spellings ("bagundi", "baagundhi", "bgndi"). Whether such encoders transfer to Romanised, code-mixed Telugu — and whether an Indian-language-focused encoder like MuRIL has an advantage — is an empirical question this dissertation answers on a carefully constructed corpus.

Literature Gap

• Shared tasks and datasets for Dravidian code-mixed sentiment (e.g. DravidianCodeMix) cover Tamil, Malayalam and Kannada; Telugu–English resources are fewer and smaller.
• Many studies report accuracy without agreement statistics for their labels, which makes it hard to know the ceiling.
• Gap: a documented Telugu–English sentiment corpus with measured agreement, and a controlled benchmark of classical, recurrent and transformer models with transliteration-handling ablations.

Proposed Work

• Corpus construction pipeline: collection via official APIs, filtering, language/script tagging, anonymisation, annotation in Label Studio, adjudication.
• Transliteration strategies: raw text, Roman-normalised, and Telugu-script back-transliteration, compared as an ablation.
• Benchmark and error analysis with significance testing.

Feasibility

All tools are open-source; fine-tuning base-size encoders on about 10,000 short comments needs a single GPU with 12–16 GB memory (college lab GPU or a cloud notebook). Annotation of 12,000 items by two annotators is achievable in about five weeks at a few hundred items per day each.

Sentiment analysis tools used by businesses, film producers, political researchers and public-service departments in Telangana and Andhra Pradesh fail on the way people actually write online: Telugu and English mixed within a sentence, mostly in Roman script with non-standard spelling, plus emojis and slang. Monolingual models misread such text, and the lack of a documented, agreement-checked Telugu–English sentiment corpus makes it impossible to measure how well modern pretrained multilingual transformers actually perform compared with cheaper classical and recurrent baselines.

This dissertation addresses two questions: (1) Can a reliable Telugu–English code-mixed sentiment corpus be built from public social-media comments with substantial inter-annotator agreement? and (2) On that corpus, how do TF-IDF + SVM, BiLSTM with FastText, mBERT, XLM-R and MuRIL compare in macro-F1, and which linguistic phenomena — code-mixing level, script, negation, sarcasm — account for their errors?

In scope

• Public comments on Telugu film, sports and news content, collected through official APIs within their terms of service.
• Three sentiment classes (positive, negative, neutral) plus a not-Telugu label used only for filtering.
• Classical, recurrent and base-size transformer models; transliteration ablation; error analysis.
• Ethics: public data only, anonymisation, no attempt to identify or profile individuals.

Out of scope

• Hate-speech or offensive-language classification (separate task and ethics review).
• Large encoder variants and generative models; speech or image content.
• Releasing raw comment text where platform terms forbid it (only IDs and labels would be shared).

Research design: a two-part empirical study — (A) corpus construction with a reliability study, and (B) a controlled comparative experiment with repeated runs.

Annotation protocol

• Guidelines define each class with 10+ Telugu–English examples, rules for mixed sentiment (label the dominant sentiment; if balanced, neutral), sarcasm (label intended sentiment) and emojis.
• Each item labelled independently by two annotators; Cohen's kappa reported per batch. Target: κ ≥ 0.61 ("substantial" in the Landis–Koch scale); if a batch falls below, guidelines are revised and the batch re-labelled.
• Disagreements are adjudicated by a third person (the guide or a senior student) and logged.

Experimental protocol

• Stratified 70/15/15 split, fixed once; no near-duplicate comments across splits (MinHash deduplication).
• Transformers: learning rate 2e-5 to 5e-5, batch 16–32, up to 5 epochs, early stopping on validation macro-F1, max length 128.
• Metrics: macro-F1 (primary), per-class F1, accuracy; mean ± sd over 5 seeds; paired bootstrap or McNemar's test between the best transformer and best baseline.
• Hypotheses (not results): transformers exceed the best baseline by ≥ 3 macro-F1 points; MuRIL is at least as good as XLM-R on Romanised Telugu; errors concentrate in high code-mixing and sarcastic items.