GenAI Evaluation & Data Infrastructure

About the role

Ampstek is looking for engineers to build an automated evaluation platform for a PII rewriting and sanitization service. The role involves developing a machine-interpretable Golden Dataset from human annotations and an automated regression framework to measure rewrite quality, privacy protection, and consistency.

The work focuses on GenAI and ML evaluation, data pipelines, test infrastructure, and workspace and document processing.

Responsibilities

• Design and implement a structured Golden Dataset representing PII ground-truth annotations.
• Build tooling to extract annotations from human-rated evaluation artifacts.
• Develop parsers and extraction pipelines for Google Docs, Google Sheets, Google Slides, Gmail and email, Google Calendar, Google Chat, PDF, and Microsoft Office files.
• Design protobuf and data schemas for machine-interpretable evaluation data.
• Build an automated evaluation and regression harness for the PII sanitization and rewrite service.
• Implement PII recall, PII precision, rewrite consistency, and entity-level and mention-level accuracy metrics.
• Build automated reporting and regression detection for service quality.
• Extend evaluation coverage across English, Spanish, Portuguese, Japanese, and Korean.
• Investigate opportunities to use GenAI and LLMs for evaluation, annotation validation, error analysis, and quality measurement where appropriate.
• Integrate the evaluation framework into existing engineering and test infrastructure.
• Partner with ML, privacy, data, and Workspace engineers to validate evaluation methodology and dataset quality.

Requirements

• Bachelor's degree or equivalent experience in Computer Science, Engineering, or a related technical field.
• Experience in GenAI evaluation, NLP/ML systems, data pipelines, or large-scale test infrastructure is particularly suited to this role.
• The source listing begins a further qualification with “3+ years of...” but does not provide the remainder.

Datos clave

• The role focuses on GenAI/ML evaluation, data pipelines, test infrastructure, and workspace and document processing.
• The work includes building an automated evaluation and regression framework for a PII sanitization and rewrite service.
• The role requires a bachelor's degree or equivalent experience in a related technical field.