Senior Data Engineer

We’re Hiring: Senior Data Engineer – Databricks Migration
Location: Dallas, TX or El Segundo, CA
Employment Type: W-2 only
Duration: 1 year, with possible extension
Lorgan Tech Solutions, LLC is seeking a Senior Data Engineer to migrate legacy Scala Spark pipelines to PySpark and transition workloads to Databricks Serverless compute.
Strong BI design and data modeling expertise is mandatory, alongside hands-on Scala, PySpark, and Azure Databricks experience.
Key Responsibilities:
• Analyze existing Scala Spark jobs, dependencies, and Serverless compatibility.
• Convert Scala pipelines to PySpark using Copilot/Genie, with thorough manual review and validation.
• Preserve business logic, schema consistency, data quality, and performance.
• Resolve Serverless compatibility issues involving JAR/JDBC dependencies, DBFS paths, streaming triggers, and unsupported configurations.
• Transition classic job clusters to Serverless, updating libraries and parameters.
• Validate results through row-count checks, schema comparisons, and side-by-side output reconciliation.
• Benchmark performance and document migration decisions, known issues, and production cutover procedures.
Required Skills:
• Strong BI design and data modeling fundamentals.
• Hands-on expertise in Scala Spark and PySpark.
• Strong Azure Databricks knowledge, including Serverless compute.
• Experience refactoring and modernizing legacy Spark workloads.
• Strong testing, data validation, troubleshooting, and performance tuning skills.
Preferred Experience:
• Production experience with Databricks Serverless.
• Familiarity with Unity Catalog, Delta Live Tables, and Lakehouse architecture.
• Experience with AI-assisted development tools.
This opportunity is open only to candidates who can work on W-2. C2C arrangements are not available.
To Apply

Subject: Senior Data Engineer – Databricks Migration
Please include your current location, preferred project location, availability