System: IntentIQ
Category: Experiments · Autonomous Agents & Browser Intelligence
Reading Time: 8 min read
Building IntentIQ: why static selectors break and how I decoupled intent from DOM paths
From a simple Playwright script to an autonomous workflow intelligence system.
Worked through: Python · Playwright · OpenAI API · BrowserUse · Blueprint Engine · Judge Rubric
01. WHY I BUILT IT
The Trigger & Underlying Problem
The Trigger: I was tired of watching browser automation test suites shatter every time a frontend engineer changed a CSS utility class or renamed a button label.
The Core Problem: Traditional automation scripts rely on rigid CSS selectors or brittle XPaths. When a website redesigns or classes re-hash in production, the automation breaks completely even though the human intent of the task hasn't changed.
Why Standard Solutions Were Insufficient: I wanted to build an automation system that understands semantic goals rather than hardcoded DOM coordinates.
02. PRACTICAL IMPACT
What It Solves in Practice
PRACTICAL PROBLEM
Enables workflows to survive frontend redesigns. IntentIQ stores what the user wants to accomplish, and if execution encounters an unexpected UI shift, an evaluation agent heals the path dynamically.
BENEFICIARIES
Anyone running high-volume web automation across third-party websites they do not control.
BEFORE THIS SYSTEM
Automation engineers spent half their week rewriting broken XPath selectors.
03. IMPLEMENTATION
Architecture & Key Engineering Decisions
A multi-component system: a Recorder captures user actions, a Blueprint Engine abstracts actions into semantic intents, Playwright executes deterministic paths, BrowserUse acts as an autonomous doctor when paths fail, and a Judge Engine scores execution quality.
Key Decisions:
- OpenAI API handles high-level intent reasoning, while headless Playwright handles deterministic, low-latency execution.
- Blueprint Engine stores semantic action goals (e.g. "submit login form with credentials") instead of DOM paths.
- Judge Engine scores state transitions against a strict rubric before marking an interaction as successful.
04. CHALLENGES & SOLUTIONS
What Failed in Practice & How It Was Addressed
WHAT WENT WRONG
- Relying solely on LLMs for every interaction was prohibitively slow and expensive: a 5-step test took 45 seconds and cost $0.30.
- Knowing whether a step succeeded was much harder than executing it: green checkmarks were given even when a modal failed to open.
- Unexpected native OS popups and alert dialogs hung headless browser instances.
HOW I SOLVED IT
- Created a hybrid execution model: Playwright runs the deterministic blueprint at native speed (sub-second). The LLM is invoked only as a fallback doctor when Playwright throws a locator timeout.
- Built an Output Extraction and Judge Engine that extracts DOM state changes post-action and scores execution against expected semantic outcomes.
- Added native OS-level popup listeners and AppleScript handlers to dismiss system dialogs automatically.
05. REFLECTION & LESSONS
Takeaways & Future Iterations
What I Learned:
- Humans don't think in selectors; humans think in workflows. Decoupling intent from presentation is the key to resilient automation.
- Teaching software to survive in dynamic environments requires feedback loops, not just script replay.
WHAT I WOULD CHANGE NOW:
Today, I would compile blueprint execution into lightweight WebAssembly modules for client-side replay and reduce token consumption further with fine-tuned local models.