Lead Data Engineer Databricks DBT

Lead Data Engineer – Databricks & dbt
Location: Irvine, CaliforniaWork arrangement: On-site four days per weekSalary: $150,000 – $200,000 per year + bonus & benefitsEmployment: Full-time W2Vacancies: TwoInterview process: Two-stage virtual interview
Urgent requirementWe are supporting a leading investment management organisation in hiring two experienced Lead Data Engineers for a major data-platform migration and modernisation programme. This is an urgent requirement, with a streamlined two-stage virtual interview process.
This hands-on technical leadership role will focus on migrating legacy data warehouses and ETL platforms to a modern Databricks Lakehouse architecture. You will combine deep Databricks, dbt and Python expertise with solution design, migration planning and team leadership.
Key responsibilitiesDesign and develop scalable data pipelines using Databricks, PySpark, Python and SQL.Lead migrations from legacy data warehouses and ETL platforms to Databricks Lakehouse.Modernise existing ETL workloads into scalable ELT frameworks using dbt.Assess source environments and define migration strategies, roadmaps and execution plans.Build solutions using Delta Lake, Unity Catalog, Databricks Workflows and Structured Streaming.Implement Bronze, Silver and Gold Medallion Architecture patterns.Develop reusable dbt models, macros, tests and transformation frameworks.Establish data-quality, monitoring, reconciliation and lineage capabilities.Optimise Spark and Databricks workloads for performance and cost.Lead code reviews, maintain engineering standards and mentor other data engineers.Partner with architects, product owners, analysts and business stakeholders.
Required experienceAt least eight years of professional data-engineering experience.Strong hands-on expertise with Databricks, dbt, Python, PySpark and SQL.Proven experience leading enterprise data-platform migration or modernisation programmes.Experience migrating legacy data warehouses, ETL platforms or on-premises data environments.Strong understanding of Databricks Lakehouse and Medallion Architecture.Experience with Apache Airflow or comparable workflow orchestration technology.Knowledge of data modelling, data quality, testing, lineage and CI/CD practices.Previous experience within asset management, investment management or a closely related financial-services environment.The ability to lead technical delivery while remaining actively involved in engineering.
This role requires on-site attendance in Irvine, California, four days per week. Candidates should already be within a commutable distance or be prepared to relocate.
With two positions available and interviews commencing shortly, interested candidates are encouraged to apply immediately with an up-to-date résumé.