AI Conversation QA Specialist
External job listingat Sibitalent
π We're Hiring: AI Conversation QA Specialist π Location: Mountain View (MTV), Oakland, CA or San Diego, CA πΌ Experience: 8+ Years π Education: Bachelor's Degree π’ Work Location: As...
Salary
Not provided
Location
Mountain View, United States
Employment type
Full time
Workplace
Not provided
AI Conversation QA Specialist
Mountain View, United States
Job description
π We're Hiring: AI Conversation QA Specialist
π Location: Mountain View (MTV), Oakland, CA or San Diego, CA
πΌ Experience: 8+ Years
π Education: Bachelor's Degree
π’ Work Location: As per client requirements
We are looking for an AI Conversation QA Specialist to evaluate AI-powered conversations, agentic workflows, and LLM outputs to improve response quality, accuracy, and overall user experience.
πΉ Key Responsibilities:
Evaluate AI conversations, agent traces, and identify quality issues.
Identify trends, tool-calling errors, intent issues, and member frustration.
Perform data labeling and generate synthetic datasets for AI evaluations.
Develop LLM-as-a-Judge evaluators, rubrics, and prompts.
Analyze AI model outputs and recommend improvements.
Query logs, analyze conversation traces, and identify recurring patterns.
Collaborate with AI Scientists, Product Managers, and Developers.
Perform quality audits, calibration, and data analysis.
πΉ Mandatory Skills & Qualifications:
Bachelor's degree in English, Communications, Linguistics, Business Analytics, Data Analytics, Computer Science, or related fields.
1β3 years of experience in Customer Support, Customer Service, Content Design QA, or similar roles.
Understanding of LLMs, AI conversations, and prompts.
Strong analytical and problem-solving skills.
Experience with Microsoft Excel or Google Sheets.
Knowledge of SQL and Python.
Experience with data filtering, Pivot Tables, charts, graphs, and basic statistical analysis.
Excellent communication, attention to detail, and organizational skills.
Ability to evaluate AI responses with accuracy, empathy, and an unbiased approach.
πΉ Preferred Qualifications:
Experience in financial institutions or regulated environments.
Familiarity with CRM and QA platforms.
Experience with AI evaluation, prompt engineering, and LLM-based tools.
Spanish fluency is a plus (not mandatory).
π― Ideal Candidate: Someone with a background in customer support, content QA, or data analytics who unders