AI Conversation QA Specialist

External job listingat Sibitalent

πŸš€ We're Hiring: AI Conversation QA Specialist πŸ“ Location: Mountain View (MTV), Oakland, CA or San Diego, CA πŸ’Ό Experience: 8+ Years πŸŽ“ Education: Bachelor's Degree 🏒 Work Location: As...

External source - not verified5 days agoOpen until: Feb 16, 2027

Salary

Not provided

Location

Mountain View, United States

Employment type

Full time

Workplace

Not provided

AI Conversation QA Specialist

Mountain View, United States

Job description

πŸš€ We're Hiring: AI Conversation QA Specialist
πŸ“ Location: Mountain View (MTV), Oakland, CA or San Diego, CA
πŸ’Ό Experience: 8+ Years
πŸŽ“ Education: Bachelor's Degree
🏒 Work Location: As per client requirements
We are looking for an AI Conversation QA Specialist to evaluate AI-powered conversations, agentic workflows, and LLM outputs to improve response quality, accuracy, and overall user experience.
πŸ”Ή Key Responsibilities:
Evaluate AI conversations, agent traces, and identify quality issues.
Identify trends, tool-calling errors, intent issues, and member frustration.
Perform data labeling and generate synthetic datasets for AI evaluations.
Develop LLM-as-a-Judge evaluators, rubrics, and prompts.
Analyze AI model outputs and recommend improvements.
Query logs, analyze conversation traces, and identify recurring patterns.
Collaborate with AI Scientists, Product Managers, and Developers.
Perform quality audits, calibration, and data analysis.
πŸ”Ή Mandatory Skills & Qualifications:
Bachelor's degree in English, Communications, Linguistics, Business Analytics, Data Analytics, Computer Science, or related fields.
1–3 years of experience in Customer Support, Customer Service, Content Design QA, or similar roles.
Understanding of LLMs, AI conversations, and prompts.
Strong analytical and problem-solving skills.
Experience with Microsoft Excel or Google Sheets.
Knowledge of SQL and Python.
Experience with data filtering, Pivot Tables, charts, graphs, and basic statistical analysis.
Excellent communication, attention to detail, and organizational skills.
Ability to evaluate AI responses with accuracy, empathy, and an unbiased approach.
πŸ”Ή Preferred Qualifications:
Experience in financial institutions or regulated environments.
Familiarity with CRM and QA platforms.
Experience with AI evaluation, prompt engineering, and LLM-based tools.
Spanish fluency is a plus (not mandatory).
🎯 Ideal Candidate: Someone with a background in customer support, content QA, or data analytics who unders

Similar jobs