Evaluation & Diagnostic AI Lead

🚨We’re Hiring: Evaluation & Diagnostic AI Lead
📍 Location: Kolkata
💼 Experience: Evaluation & Diagnostics Experience: 2+ years of proven experience creating evaluation test cases, running diagnostics, and guiding customers to auditable, secure and reliable AI.
Engineering & Implementation: 7+ years in software engineering and custom project implementation, preferably in regulated environments.
🔹The Opportunity:
As the capability lead, you will dedicate the core of your efforts to designing and building our evaluation and observability infrastructure on Arize AX. This includes setting up agent tracing, golden evaluation datasets, LLM-as-judge frameworks, and automated regression gates in CI/CD pipelines so no agent ships without verified performance. Beyond technical architecture, you will define the end-to-end evaluation strategy aligned with OWASP Top 10 standards for LLM applications, maintain auditable ADLC work instructions, and mentor engineering talent to build a self-sufficient Evaluation & Diagnostic capability within our India Capability Center (ICC).
🔹What We Are Looking For:
The ideal leader brings over seven years of software engineering and project delivery experience, paired with at least two years of specialized work in AI evaluation, tracing, and diagnostics. Strong fundamentals in Python, APIs, automated testing, and CI/CD are essential, along with hands-on mastery of observability tools such as Arize AX, Arize Phoenix, LangSmith, Langfuse, Ragas, DeepEval, promptfoo, or TruLens. Candidates with experience in AI red-teaming tools like Garak and PyRIT, PII detection, and Amazon Bedrock Model Evaluation and Guardrails will be strongly positioned for this role.
🔹What Sets You Apart:
Because our software operates in mission-critical laboratory settings, autonomous systems must align with GxP and 21 CFR Part 11 traceability standards. The strongest candidates will have tangible examples of production systems they have built, such as golden-dataset eva