Sheeraz Ahmad K.

I build the infrastructure that tells you whether your AI is actually working. At my last engagement — a 1.5-year contract with a New York-based AI fintech startup — I owned quality end-to-end for their LLM-powered research platform. That meant designing and shipping: ∙ A multi-dimensional LLM-as-Judge evaluation pipeline that scored AI-generated reports across factual accuracy, citation quality, completeness, coherence, and relevance — replacing subjective manual review with automated, repeatable assessments. ∙ BigQuery-based telemetry and observability for tracking LLM output quality over time, enabling data-driven decisions on prompt changes and model upgrades. ∙ GCP-hosted automation (Cloud Functions, Cloud Scheduler, Pub/Sub) that ran evaluation jobs on every report generation cycle — no manual triggers, no missed regressions. ∙ A Playwright + TypeScript test infrastructure from scratch — E2E tests, API tests, and CI/CD pipelines via GitHub Actions — covering the full application stack. ∙ FlakyGuard, a custom Playwright Reporter (npm package) with LLM-powered flake classification, Allure integration, Slack alerts, and CI auto-quarantine. Before that: 4+ years of test automation and infrastructure work across companies like Verizon Media and Shutterstock, consistently shipping in TypeScript, Playwright, and CI/CD environments. What I’m looking for: Contract roles (full-time or part-time) where I can own AI/LLM quality, test infrastructure, or evaluation systems. I work US hours (EST overlap), communicate async-first, and ship independently with minimal supervision.