Databricks Architect

Were Hiring: Databricks Architect | AWS | Lakehouse Platform
📍 Location: Los Angeles, CA / Irvine, CA
💼 Experience: 12+ Years
⏳ Duration: Long-Term
🤝 Company: Persistent Systems
We are looking for an experienced Databricks Architect with deep expertise in AWS, Lakehouse architecture, distributed data processing, and scalable data engineering solutions.
The ideal candidate will be a hands-on technical leader who can design enterprise-grade data platforms, optimize performance and cost, and guide engineering teams in implementing modern data architectures.
🔹 Must-Have Technical Skills:
Databricks Lakehouse Architecture on AWS
Apache Spark, PySpark, Spark SQL, and SQL
Delta Lake, Medallion Architecture (Bronze/Silver/Gold)
Databricks Workflows, Jobs, Notebooks, and Delta Live Pipelines
AWS S3, Glue / Glue Catalog, IAM, Lambda, API Gateway, and Kinesis
Data partitioning, caching, Z-Ordering, file compaction, and Spark performance tuning
Unity Catalog, RBAC, data governance, and encryption
CI/CD, GitHub/GitLab, DevOps, and Infrastructure as Code (Terraform preferred)
🔹 Key Responsibilities:
✅ Design and implement scalable Databricks Lakehouse solutions on AWS.
✅ Build and optimize robust ETL/ELT pipelines using PySpark and Delta Lake.
✅ Architect low-latency data delivery and serving-layer solutions beyond Spark alone.
✅ Handle multi-terabyte structured, semi-structured, and time-series datasets.
✅ Optimize compute resources, caching, partitioning, cluster sizing, and cloud costs.
✅ Implement streaming and incremental data processing solutions.
✅ Establish CI/CD automation, data quality, lineage, monitoring, and governance standards.
✅ Lead architecture reviews, proof of concepts, and technical design discussions.
✅ Collaborate with data engineering, ML, BI, and business teams to drive platform adoption.
🔹 Preferred Qualifications:
Databricks certifications in Data Engineering or Machine Learning
Experience with MLflow, Feature Store, and MLOps pipelines
Experience in re