Data Engineer

Title: Senior Data Engineer100% REMOTE – Latam About the RoleWe are looking for an experienced Senior Data Engineer to design, build, and maintain scalable data platforms and pipelines that support analytics, reporting, data science, and AI initiatives.The ideal candidate has strong hands-on experience with Databricks, Apache Spark, Python, SQL, and cloud data platforms, along with a solid understanding of data architecture, ETL/ELT, data quality, and performance optimization.Key ResponsibilitiesDesign, develop, and maintain highly scalable data pipelines and ETL/ELT workflows.Build and optimize data engineering solutions using Databricks and Apache Spark.Develop robust batch and streaming data pipelines using Python and SQL.Implement data processing and transformation using PySpark.Design and maintain data lake/lakehouse architectures using technologies such as Delta Lake.Develop reliable data ingestion pipelines from databases, APIs, files, and cloud-based sources.Build and maintain data models for analytics, reporting, and downstream applications.Implement data quality, validation, monitoring, and error-handling frameworks.Optimize Spark jobs, SQL queries, clusters, and data pipelines for performance and cost.Develop workflows and orchestration using Databricks Workflows, Airflow, or similar tools.Work with cloud data platforms such as AWS, Azure, or GCP.Implement CI/CD and automated deployment practices for data pipelines.Collaborate with data scientists, analysts, software engineers, architects, and business stakeholders.Participate in architecture and design discussions and provide technical leadership to junior engineers.Troubleshoot production data issues and ensure high availability and reliability of data pipelines.Establish and enforce data engineering best practices, coding standards, and documentation.Required Qualifications10+years of professional experience in data engineering.Strong hands-on experience with Databricks.Strong experience with Apache Spark / PySpark.Advanced SQL skills, including query optimization and complex transformations.Strong programming experience with Python.Experience designing and implementing ETL/ELT pipelines.Strong understanding of data lake and lakehouse architectures.Hands-on experience with Delta Lake / Delta tables.Experience working with relational and NoSQL databases.Experience with at least one major cloud platform: AWS, Azure, or GCP.Experience with workflow orchestration tools such as Airflow, Databricks Workflows, or Azure Data Factory.Experience with Git and CI/CD practices.Strong understanding of data modeling, data warehousing, and dimensional modeling.Experience with data quality, governance, security, and monitoring.Strong analytical, troubleshooting, and communication skills.Preferred QualificationsExperience with Azure Databricks and Microsoft Azure services.Experience with AWS Databricks, S3, Glue, EMR, or related services.Experience with Unity Catalog and Databricks governance.Experience implementing Medallion Architecture (Bronze, Silver, Gold).Experience with real-time/streaming technologies such as Kafka, Spark Structured Streaming, or Event Hubs.Experience with Snowflake, BigQuery, Redshift, Synapse, or similar data warehouses.Experience with dbt or other modern data transformation frameworks.Experience with Terraform or Infrastructure as Code.Knowledge of data governance, lineage, cataloging, and access-control frameworks.Experience supporting machine learning or AI data pipelines.Databricks certifications are a plus.Technical SkillsLanguages: Python, PySpark, SQLBig Data: Apache Spark, Spark SQLData Platform: Databricks, Delta Lake, LakehouseCloud: AWS / Azure / GCPOrchestration: Databricks Workflows, Airflow, ADFStreaming: Kafka, Spark Structured Streaming, Event HubsDatabases: PostgreSQL, SQL Server, MySQL, MongoDBWarehousing: Snowflake, Redshift, BigQuery, SynapseDevOps: Git, CI/CD, Docker, TerraformGovernance: Unity Catalog, Data Quality, Data Lineage, Security
Thanks & RegardsNitin KushwahaEros Technologies IncEmail: [email protected]