S
Evaluation & Diagnostic AI Lead
External job listingat Spsconsultancyservices
Tasks Lead AI evaluation and diagnostic capabilities using Arize AX. Build evaluation frameworks, datasets, LLM-as-a-Judge evaluators and monitoring dashboards. Evaluate RAG applications...
External source - not verified4 hours agoOpen until: Feb 3, 2027
Salary
Not provided
Location
Kolkata, India
Employment type
Full time
Workplace
Not provided
Evaluation & Diagnostic AI Lead
Kolkata, India
Job description
Tasks
- Lead AI evaluation and diagnostic capabilities using Arize AX.
- Build evaluation frameworks, datasets, LLM-as-a-Judge evaluators and monitoring dashboards.
- Evaluate RAG applications and AI agent trajectories.
- Implement automated evaluation and regression gates within CI/CD pipelines.
- Test AI agents for prompt injection, jailbreaks and sensitive data leakage.
- Mentor and develop engineering teams within the India Capability Center.
Requirements
- 7+ years of experience in Software Engineering and custom project implementation.
- 2+ years of hands-on experience in AI Evaluation and Diagnostics.
- Strong proficiency in Python, APIs, Git, automated testing and CI/CD.
- Experience with AI evaluation tools such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, TruLens or OpenAI Evals.
- Knowledge of RAG evaluation, LLM observability and AI security.
- Experience with AWS Bedrock is an added advantage.
- Strong leadership, mentoring and stakeholder management skills.
How to apply
- Share your updated CV by email or DM.
Location
- Kolkata