YC

The problem with automating court data across fragmented government websites

Extracting daily cause-lists and orders from the High Court, DRT, and NCLT without human supervision.

System: Court Data Automation
Category: Pipelines · Scraping & Scheduling
Reading Time: 7 min read
Case Study→
Worked through: Playwright · Headless Chromium · PDF Extraction · Linux cron · SQLite Idempotency
01. WHY I BUILT IT

The Trigger & Underlying Problem

The Trigger: Litigation cases were active across multiple judicial bodies: the High Court, Debt Recovery Tribunals (DRT), Debt Recovery Appellate Tribunals (DRAT), and the National Company Law Tribunal (NCLT).

The Core Problem: Lawyers had to wake up before 6:00 AM every morning to check court websites, find out if their matter was listed, which courtroom it was in, and whether orders had been uploaded. Websites were slow, unannounced format changes were common, and missed hearings carried severe legal consequences.

Why Standard Solutions Were Insufficient: No commercial API covered all these specialized tribunals reliably with firm-specific case tracking.

02. PRACTICAL IMPACT

What It Solves in Practice

PRACTICAL PROBLEM

Automatically scrapes daily cause-lists, extracts courtroom allocations and item numbers, downloads new signed orders, and logs them to the firm database by 6:00 AM every morning.

BENEFICIARIES

Litigation attorneys, paralegals, and institutional clients tracking case milestones.

BEFORE THIS SYSTEM

Paralegals spent 2–3 hours each morning manually downloading dozens of PDF cause-lists, using Ctrl+F to find case numbers, and typing courtroom notes into spreadsheets.

03. IMPLEMENTATION

Architecture & Key Engineering Decisions

A scheduled background data pipeline running on a Linux VPS. Cron triggers the pipeline. Headless browser workers navigate the court portals, extract the latest rosters, run PDF text extraction, and match case numbers against the firm roster.

Key Decisions:
  • Used Playwright over simple HTTP requests because modern tribunal sites rely heavily on ASP.NET PostBack events and dynamic JavaScript session cookies.
  • Extracted text from cause-list PDFs using streaming token parsers to match case prefixes regardless of formatting quirks (e.g., "O.A./123/2024" vs "OA 123 of 2024").
  • Implemented an SQLite-backed idempotency tracking ledger to record which order files and cause-lists were already processed.
File: cause_list_parser.py
import re

# Normalizes messy court case numbering variations into a canonical hash
CASE_REGEX = re.compile(r"(?P<type>[A-Za-z]+)[\s/.-]*(?P<num>\d+)[\s/.-]*(?P<year>\d{2,4})")

def normalize_case_key(raw_text: str) -> str:
    match = CASE_REGEX.search(raw_text)
    if not match:
        return raw_text.strip().upper()
    ctype = match.group("type").upper()
    cnum = match.group("num").lstrip("0")
    cyear = match.group("year")
    if len(cyear) == 2:
        cyear = f"20{cyear}"
    return f"{ctype}:{cnum}:{cyear}"
04. CHALLENGES & SOLUTIONS

What Failed in Practice & How It Was Addressed

WHAT WENT WRONG
  • Court portals frequently timed out or reset TCP connections between 5:00 AM and 6:30 AM due to automated server backup jobs on the court side.
  • Court registries randomly published cause-lists as scanned bitmap images inside PDFs rather than searchable text, breaking standard string matching.
  • Session cookies expired unpredictably during multi-page pagination.
HOW I SOLVED IT
  • Built a retry coordinator with exponential backoff and jitter that detects socket resets and reschedules jobs at 15-minute intervals until successful confirmation.
  • Integrated a fallback OCR pipeline (Tesseract) that triggers automatically when standard PDF text stream extraction returns fewer than 50 printable characters.
  • Stored authenticated browser storage states to disk so subsequent page navigations reuse valid session contexts without re-initiating full handshakes.
05. REFLECTION & LESSONS

Takeaways & Future Iterations

What I Learned:
  • Government and legacy portals never behave predictably. You cannot write brittle scrapers that assume a 200 OK response; every network boundary must assume failure.
  • Idempotency is the single most important property of an automated data pipeline.
WHAT I WOULD CHANGE NOW:

I would package each court extractor as an event-driven AWS Lambda worker orchestrated with AWS Step Functions, rather than running all scrapers on a single Linux VPS.

← Back to all notesAbout the Engineer →