Senior Data Engineer Azure Data FactoryDatabricksPySpark

Career Stage: Senior Associate L1 (+4-5 YoE) or Senior Associate L2 (+7 YoE)Job Title: Senior Data Engineer (Azure Data Factory/Databricks/PySpark)Location: Mexico, Colombia, Costa Rica, Argentina, Peru, Brazil.Industry: UHGTime Zone: EST
JOB OVERVIEWWe’re looking for a Senior Data Engineer with strong hands-on expertise in building scalable data pipelines and cloud-native data solutions on Azure. This role focuses on designing and implementing real production systems using Azure Data Factory, Azure Databricks, and modern big data technologies. You will work across distributed data platforms, integrating cloud and on-premises environments, and delivering robust, enterprise-grade data solutions aligned with industry best practices.
YOUR IMPACTDesign and build scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB)Develop PySpark-based transformation logic for large-scale data processing, including joins, aggregations, and window functionsArchitect and implement hybrid data integrations between cloud and on-premises systemsEnable secure connectivity from Databricks to on-prem databases using enterprise-grade patternsBuild and optimize data models across Cosmos DB and Snowflake for different workloadsImplement monitoring, logging, and error-handling mechanisms to ensure reliability and performanceCollaborate with cross-functional teams to define standards, patterns, and best practices for data engineering solutions
QUALIFICATIONSYour Skills and Experience (Must Have)Strong experience building data pipelines with Azure Data Factory and Azure DatabricksHands-on expertise in PySpark for distributed data processingAdvanced knowledge of SQL, including complex joins and performance optimizationExperience working with Cosmos DB and SnowflakeSolid programming skills in Python, Scala, or JavaExperience with Kafka or event-driven architecturesStrong understanding of data architecture and distributed systemsProven ability to deliver production-grade data solutions in complex environments
TECHNICAL REQUIREMENTSCandidates must demonstrate solid, hands-on knowledge across the following three areas:1. PySpark (Production-Level Coding)DataFrame transformations: select, filter, groupBy, agg, withColumnWindow functions: rank, row_number, lag, leadJoins: inner, left, broadcast joins and usage scenariosReading/writing data: Parquet, Delta, CSVUDFs: syntax, registration, and performance tradeoffsAbility to produce production-ready code without pseudo-code2. Azure Databricks (Architecture & Platform Expertise)Cluster types, auto-scaling, and cluster policiesNotebook orchestration, workflows, and job schedulingDelta Lake: ACID, schema evolution, time travel, OPTIMIZE, VACUUMUnity Catalog: governance, lineage, and access controlIntegration with Azure Data Factory3. Hybrid Data Architecture (On-Prem to Cloud Integration)JDBC/ODBC connectivity from DatabricksSecure credential management with Azure Key Vault and secret scopesNetwork architecture: VNet injection, private endpoints, Self-Hosted IREnd-to-end pipeline design from on-prem to cloudPerformance optimization and error handling in hybrid environments
SET YOURSELF APART WITH (NICE TO HAVE)Experience designing end-to-end data architectures in Azure ecosystemsKnowledge of Delta Lake features (ACID transactions, schema evolution, time travel)Experience with Databricks Workflows and orchestrationUnderstanding of data governance (Unity Catalog)Experience with hybrid cloud/on-prem integration patternsExposure to performance optimization at scale