Sr Data Engineer

We are looking for a Data Platform Engineer to build, operate, and scale a real-time data platform that transforms raw, high-throughput data into reliable datasets used for machine performance monitoring and optimization.
This position combines Data Engineering and Platform Engineering, with a strong focus on production data pipelines, real-time streaming, automation, observability, and platform reliability.
You will work closely with Physics, Analytics, and Engineering teams to transform scientific and exploratory datasets into scalable, reliable, production-ready solutions.
What You'll Do
Build, deploy, maintain, and optimize data pipelines and real-time streaming applications.Integrate new data sources and machine signals into the data platform.Transform scientific, laboratory, and exploratory data into reliable, documented, production-ready datasets.Partner with Physics, Analytics, and Engineering teams to translate technical and business requirements into scalable solutions.Manage platform operations, including upgrades, patching, infrastructure maintenance, capacity planning, and deprecations.Implement observability through logging, monitoring, dashboards, alerts, and usage tracking, primarily with Grafana.Automate source onboarding and operational workflows using Azure DevOps, Python, and Infrastructure as Code.Manage RBAC, permissions, and platform security.Monitor pipeline health and data quality.Troubleshoot production failures and latency issues.Participate in incident management and perform root cause analysis.
Required Qualifications
Advanced hands-on experience with Python and SQL for data engineering, automation, querying, scripting, and data transformations.Strong understanding of data modeling, data quality, ETL/ELT, pipeline architecture, and optimization.Hands-on experience with Apache Spark, PySpark, and Databricks.Experience working with Microsoft Azure.Production experience deploying and managing workloads on Kubernetes.Experience with Azure DevOps, including pipelines, repositories, CI/CD, and release management.Strong knowledge of Git, branching strategies, and code reviews.Experience building or supporting high-throughput, low-latency streaming data systems.Hands-on experience with Grafana for dashboards, monitoring, and alerts.Strong debugging, troubleshooting, incident management, and root cause analysis capabilities in production data environments.