AI Evaluation Engineer · India · Remote worldwide
Ship AI you can trust.
I help teams measure whether their LLM apps, RAG systems and agents actually work — with eval sets, scoring and monitoring that catch failures before your users do.
16+
years in quality engineering
4
AI systems evaluated
2k
eval cases built
eval_run · rag-support-bot
● pass
faithfulness
0.71 → 0.94
answer_relevance
0.78 → 0.91
hallucination_rate
12% → 3%
412 cases · judge κ 0.82
example run
About
From breaking software to measuring intelligence
I spent 16 years in software quality engineering — building test strategies, automation frameworks and release gates. When teams started shipping LLM features, I saw the same problem in a new shape: nobody could say with confidence whether the product was getting better or worse.
Today I lead AI evaluation practices. I turn fuzzy goals like “helpful” and “accurate” into criteria you can score, build datasets that reflect how people really use your product, and wire evals into the release process — so quality becomes a number you can track, not a feeling.
- LLM evaluation
- Eval dataset design
- LLM-as-judge
- Rubric design
- RAG evaluation
- Agent evaluation
- Red-teaming
- Hallucination testing
- Evals in CI/CD
- Human-in-the-loop review
- Drift monitoring
- Test automation
Services
Where I can help
From a one-off audit to a full evaluation practice. Every engagement ends with something your team can run without me.
AI Evaluation Strategy & Audit
A review of how you test your AI today, the gaps that matter most, and a prioritised plan for closing them.
Eval Dataset & Rubric Design
Golden sets, adversarial and edge cases drawn from real usage, plus scoring rubrics your team and an LLM judge can apply consistently.
RAG & Retrieval Evaluation
Measure retrieval relevance, faithfulness and groundedness separately, so you know whether to fix the search or the generation.
Agent & Tool-Use Evaluation
Check that agents pick the right tools, pass the right arguments and finish multi-step tasks — not just that the final answer sounds right.
Safety, Hallucination & Red-Team Testing
Probe for fabricated facts, prompt injection, jailbreaks and policy violations before someone else finds them.
Continuous Evals & Production Monitoring
Regression evals in CI/CD that block bad releases, and production monitoring that flags drift as it happens.
Process
How I evaluate
The same loop works for a chatbot, a RAG search or a multi-step agent. What changes is what we measure.
01
Define success
Agree what “good” means for your use case, written as criteria you can actually score.
02
Build the eval set
Representative, adversarial and edge cases from real usage — versioned like code.
03
Score
Automated metrics, LLM-as-judge and targeted human review, calibrated against each other.
04
Iterate
Compare prompts, models and retrieval changes on the same yardstick, then ship the winner.
05
Monitor in production
Sample live traffic and catch drift or new failure modes before users report them.
Free consultation
30 minutes on your AI’s quality. Free.
No sales pitch. If I’m not the right fit, I’ll say so and point you somewhere better.
What we’ll cover
- Your AI use case and how it is tested today
- The failure modes most likely to hurt you
- Metrics and an eval approach that fit your stack
- Three concrete next steps you can start on this week
Who it’s for
Startups shipping their first LLM feature · Product teams unsure whether a model change helped · AI leads setting up an evaluation practice
1 Book a slot
2 We review your AI use case
3 You get an evaluation plan
Pick a time that suits you
Choose a slot and add a line about your AI use case. You’ll get a calendar invite with a video link straight away.
Selected work
Results, not vibes
A few examples of what measurable AI quality looks like in practice. Client names kept confidential.
RAG · Customer support
[Client type] support assistant
Problem Answers sounded confident but often cited the wrong policy document.
Approach Built a golden set from real tickets and scored retrieval and faithfulness separately to find where it broke.
faithfulness [0.XX] → [0.XX]
Agents · Operations
[Client type] workflow agent
Problem The agent passed demos but failed silently on real multi-step tasks.
Approach Added tool-call and trajectory checks to CI, with a regression suite built from production traces.
task success [XX]% → [XX]%
Model migration
[Client type] model switch
Problem The team wanted a cheaper model without quietly breaking quality.
Approach Ran side-by-side regression evals with an LLM judge calibrated against human labels.
[N] regressions caught before release
Eval stack
Tools I work with
Tool-agnostic by default — I fit into the stack you already have rather than adding another platform.
- promptfoo
- DeepEval
- Ragas
- LangSmith
- Langfuse
- Arize Phoenix
- OpenAI Evals
- Weights & Biases
- Python
- pytest
- Playwright
- GitHub Actions
Testimonials
What teams say
[A one- or two-sentence quote about the result you delivered — ideally with a number in it.]
[Name], [Role], [Company]
[A quote about how you worked with the team: clear, practical, easy to hand over.]
[Name], [Role], [Company]
[A quote from someone who took the free consultation and what they got out of it.]
[Name], [Role], [Company]
FAQ
Questions teams usually ask
Why can’t normal QA test an LLM?
Traditional tests expect one correct output. LLMs give different, often equally valid answers, and fail in new ways like hallucination, tone drift or unsafe replies. Evaluation replaces exact matches with criteria, scoring and statistics — but it still builds on solid QA discipline.
What is LLM-as-judge, and can it be trusted?
It means using a model to score outputs against a rubric. It scales well, but only after it is calibrated against human judgements and checked for bias. I treat it as one signal alongside metrics and human review, not the final word.
Do you need access to our model or data?
Not to get started. Most evaluation runs against your app’s API or logged outputs. For deeper work I use anonymised samples and am happy to sign an NDA first.
How long does an eval setup take?
A first useful eval suite usually takes one to three weeks, depending on how many use cases you have and how much labelled data already exists. You see results from the first week.
Is the consultation really free?
Yes — 30 minutes, no obligation. You leave with concrete next steps whether or not we work together.
Writing
Notes on AI evaluation
-
Hello world!
Welcome to WordPress. This is your first post. Edit or delete it, then start writing!
-
Why your LLM needs an eval set before launch
Demos prove that something can work. An eval set tells you how often it does — and whether your next change made…
-
LLM-as-judge: when to trust it
Using a model to grade another model scales beautifully — once you have checked that the judge agrees with the people who…
Contact
Let’s talk about your AI’s quality
Prefer writing to a call? Send a few lines about what you’re building. I reply within one working day.