AI Quality Analyst | $30/hr Remote

Position: AI Benchmark Quality ReviewerType: Contractor AssignmentCompensation: $30/hrLocation: RemoteCommitment: 40 hours/week, with at least 8 hours overlap during PST business hours
Role ResponsibilitiesValidate task quality by reviewing instructions, source materials, reference solutions, and evaluation criteria for consistency, completeness, and hidden requirements.Review AI agent performance by inspecting execution traces, tool calls, and generated deliverables to determine whether successes and failures are appropriately assessed.Audit grading logic by identifying brittle checks, incorrect expected answers, unsupported rubric criteria, and cases where valid alternative solutions are unfairly penalized.Investigate discrepancies and distinguish model limitations from task defects, grader errors, environment issues, and tool failures.
Requirements5+ years of relevant professional experience in technical QA, software testing, AI evaluation, data analysis, benchmark development, validation, or a related technical field.Comfortable reading Python, SQL, shell scripts, structured data, and execution logs to understand task setup and grading behavior.Strong analytical skills with practical experience checking calculations, reconciling conflicting evidence, and assessing the correctness and completeness of technical deliverables.Experience providing clear, specific, and reproducible technical feedback.Strong written English and attention to detail.
Application ProcessApply via Easy Apply.Check your email for next steps.