Serhii K.
LLM Agents & Evals - RLS, Auth, Audit Logs, PII/PHI for AI SaaS
Your AI feature works in the demo. Then a customer asks: can tenant A ever see tenant B's data? Who approved that AI decision? What happens when a document tells your agent to ignore its rules? I make sure you can answer with tests, not slides. Senior full-stack engineer, 10+ years. I ship LLM agents in TypeScript (Next.js) and Python (FastAPI) products and build the floor around them: - evals in CI that fail the PR when quality drops - traces, so an engineer sees why the agent did what it did - Postgres row-level security + tests that prove one tenant can't read another - auth/RBAC and agent tool permissions - the agent only gets what the user has - append-only audit log for AI decisions and admin actions - prompt-injection checks + PII/PHI redaction before data reaches the model Proof - my public-safe demo: an AI decision feature with 15 gold cases and a fail-closed floor. Break one expectation → CI goes red. Restore → green. Same scorers run in Braintrust, offline stays the merge gate. Caveat - the decision engine there is a deterministic baseline today. The LLM path is in progress. Shipped before: - a production LLM scheduling agent and a RAG pipeline at a healthcare AI startup, with evals, tracing, RBAC and audit logging - PII/PHI anonymization before data reaches the model, plus the data pipelines and enterprise integrations around it How I work - fixed scope first: - Audit - read-only, every finding as Problem → Evidence → Impact → Fix → Effort. From USD 2,500 - Hardening build - I fix it: eval gate, traces, RLS, audit log, auth/RBAC. From USD 4,000 - Hourly after that - USD 250/h Compliance: I've worked with PHI, and I get you ready for SOC 2 / HIPAA / GDPR questions - what to log, what must fail closed, who can see what. Readiness, not certification - I'm not an auditor. Not a fit: fine-tuning, model training, RLHF, generic web builds. Remote. Vancouver, Canada (Pacific time).