L
AI Engineer – Agent Evaluation Specialist
External job listingat Linktms
AI Engineer – Agent Evaluation Specialist in Los Angeles, CA. C2C contract for 3+ months, possible extension. Hybrid.
External source - not verified3 days agoOpen until: Feb 4, 2027
Salary
Not provided
Location
Los Angeles, United States
Employment type
Full time
Workplace
Not provided
AI Engineer – Agent Evaluation Specialist
Los Angeles, United States
Job description
Tasks
- Design and implement LLM-as-a-Judge evaluation frameworks to measure agent response quality, accuracy, relevance, and reliability.
- Develop and maintain golden datasets in collaboration with Business Analysts and business stakeholders.
- Evaluate AI agent behavior including response quality, tool execution accuracy, reasoning workflows, task completion, reliability and consistency.
- Instrument AI agents using Langfuse for agent tracing, prompt/response tracking, evaluation scoring, performance monitoring and failure analysis.
- Build evaluation pipelines and metrics to continuously improve AI agent performance.
- Analyze agent failures and provide recommendations for optimization.
- Collaborate with AI engineers, data scientists, and product teams to improve enterprise GenAI solutions.
Requirements
- Hands-on experience with LLM-as-a-Judge evaluation frameworks
- Experience creating and managing Golden Datasets for AI evaluation
- Practical experience with Langfuse for AI observability and tracing
- Strong understanding of Agent Evaluation methodologies
- Experience evaluating LLM-powered agents and AI workflows
- Knowledge of Agentic AI concepts, tool calling, and agent behavior testing
Schedule
- Duration: 3+ Months (Possible Extension)
- C2C Contract
- Hybrid
How to apply
- Share qualified profiles by email.
Location
- Los Angeles, CA