Data Engineer - Databricks
Aviso de fuente externaen Celebal Technologies
Celebal Technologies busca un Data Engineer con experiencia en Databricks, PySpark, SQL, Delta Lake y desarrollo de pipelines de datos escalables.
Salario
No especificado
Ubicación
Bogotá, Colombia
Tipo de empleo
Tiempo completo
Modalidad
No especificado
Data Engineer - Databricks
Bogotá, Colombia
Descripción del empleo
About the role
Celebal Technologies is looking for a Data Engineer with hands-on experience in data engineering and strong expertise in Databricks. The role focuses on building scalable data pipelines, data transformation, and working with modern cloud data platforms.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks.
- Develop data ingestion and transformation processes using PySpark and SQL.
- Work with Delta Lake and implement reliable data processing solutions.
- Build and optimize ETL/ELT pipelines for large-scale datasets.
- Implement data quality, validation, monitoring, and error-handling processes.
- Work with different data sources and integrate data into centralized data platforms.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Collaborate with Data Scientists, Analysts, and other engineering teams.
- Follow best practices for data security, governance, and pipeline development.
Requirements
- 3–5 years of experience in Data Engineering and strong hands-on experience with Databricks.
- Strong proficiency in PySpark and SQL, and experience with Delta Lake.
- Good understanding of ETL/ELT concepts and data pipeline development.
- Experience working with large datasets and distributed data processing.
- Good understanding of data warehousing and data modeling concepts.
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Strong problem-solving and analytical skills.
- Preferred: experience with Medallion Architecture (Bronze, Silver, Gold), Databricks Workflows/Jobs, Git, CI/CD practices, and Azure Data Lake, AWS S3, or similar cloud storage platforms; knowledge of Auto Loader, Structured Streaming, data governance, and security.
Datos clave
- El puesto requiere 3–5 años de experiencia en Data Engineering.
- Se requiere experiencia práctica con Databricks, PySpark y SQL.
- La experiencia con Delta Lake es un requisito.
Preguntas frecuentes
¿En qué consiste el puesto?
Se centra en diseñar, desarrollar y mantener pipelines de datos escalables con Databricks, además de procesos de ingesta y transformación.
¿Qué experiencia y conocimientos se requieren?
Se requieren 3–5 años de experiencia en Data Engineering, experiencia práctica con Databricks y dominio de PySpark y SQL. También se requiere experiencia con Delta Lake.
¿Qué conocimientos son preferidos?
Se prefieren conocimientos de Medallion Architecture, Databricks Workflows/Jobs, Auto Loader, Structured Streaming, Git y prácticas de CI/CD.