Lead Databricks/Spark Engineer

Tasks

• Analyze existing applications, frameworks, and data processing solutions.
• Support migration and modernization of existing data platforms to Databricks.
• Design, develop, and optimize large-scale Spark / PySpark workloads.
• Troubleshoot Spark performance issues and resolve complex migration challenges.
• Develop and maintain batch and streaming data processing pipelines.
• Identify opportunities to improve data platform architecture and performance.
• Collaborate with engineering teams to modernize existing frameworks and solutions.
• Apply strong debugging and problem-solving skills to overcome technical challenges.

Requirements

• 10+ years of overall experience in Data Engineering.
• Strong hands-on experience with Apache Spark / PySpark.
• Extensive experience with Databricks.
• Strong Python programming skills.
• Expertise in Spark performance tuning and troubleshooting.
• Experience with batch and streaming data processing.
• Hands-on experience in data platform modernization and migration.
• Strong debugging and problem-solving abilities.
• Good to Have: Scala programming experience; Spark Connect / Databricks Connect; MLflow experience; Exposure to ML platforms and ML pipelines; Framework or Platform Engineering experience; Strong understanding of Distributed Systems.

Schedule

• Duration: Long-Term Opportunity

How to apply

• Share your updated resume by email.

Location

• Columbus, GA 31901 / Atlanta, GA 30339