Databricks Architect

About the role

The Databricks Architect is responsible for designing, implementing, and optimizing scalable data engineering and analytics solutions on the Databricks Lakehouse Platform on AWS.

This role requires deep expertise in distributed data processing, Delta Lake–based architectures, and modern data engineering best practices.

Tasks

• Architect end-to-end Databricks Lakehouse solutions on AWS for ingestion, processing, storage, and consumption.
• Define and implement Delta Lake patterns including Medallion Architecture (Bronze/Silver/Gold).
• Lead design of scalable data pipelines using PySpark, Spark SQL, Workflows, and Delta Live Pipelines.
• Architect solutions for structured, semi-structured, and time-series workloads.
• Build robust ETL/ELT pipelines using Databricks Notebooks, Jobs, and Workflows.
• Implement streaming and incremental data processing as needed using Structured Streaming.
• Optimize Spark jobs with partitioning, caching, ZORDER, file compaction, and shuffle reduction.
• Implement CI/CD automation using Databricks Repos, GitLab/GitHub, and Infrastructure-as-Code.
• Architect Databricks solutions using AWS-native services including S3 (data storage), Glue Catalog (metadata governance), IAM (identity & access control), Lambda / API Gateway (low-latency serving mechanisms), and Kinesis (streaming ingestion).
• Ensure security, governance, and compliance via Unity Catalog, RBAC, and encryption standards.

Schedule

• Contract

How to apply

• Share resume.

Location

• LA/Irvine, CA (Onsite)