This is my profileContact via CazVidhimanshu
Contact
Competencies
Demonstrated experience with Python.
Demonstrated experience with Pyspark.
Demonstrated experience with SQL.
Demonstrated experience with Azure Data Factory.
Demonstrated experience with Azure Data Lake Storage (ADLS).
Education
Languages
Summary
Professional profile
Experience
Software Engineer
Migrated Informatica ETL mappings and workflows to Azure Databricks (PySpark & Spark SQL), reducing dependency on legacy ETL tools.. Ingested data from Oracle and SQL Server (SSMS) into ADLS Gen2 using scalable Databricks ingestion frameworks.. Implemented Bronze–Silver–Gold data lake architecture for raw, cleansed, and business-ready datasets.. Converted Informatica transformations (Expression, Lookup, Joiner, Aggregator, Filter) into optimized PySpark logic.. Designed incremental and full load pipelines using watermark and timestamp-based strategies.. Built and managed Delta Lake tables with ACID transactions, schema evolution.. Implemented Unity Catalog for centralized governance, metadata management, and role-based access control.. Orchestrated Databricks notebooks using Azure Data Factory, handling dependencies, retries, and failure scenarios.. Performed source-to-target validation, data reconciliation, and data quality checks.. Optimized Spark jobs using partitioning, broadcast joins, caching, and cluster auto-scaling.. Collaborated with cross-functional teams to ensure successful migration, testing, and production deployment.
Software Engineer
Automated data transformation and ETL processes using Azure Databricks and managed the ETL pipeline with Azure Data Factory.. Created and improved data processing workflows with Azure Databricks, making them faster and more efficient.. Used Azure Databricks for big data processing and analysis of large datasets.. Built a centralized and scalable data storage solution with Azure Data Lake Storage Gen2 (ADLSGen2).. Creating complex stored procedure and use various Data optimization techniques to increase the Speed and reduce the time for higher proficiency in data migration.. Participated in sprint planning, Reviews, retrospectives, grooming sessions, and peer reviews.. Worked closely with Business analyst to turn business requirements into technical specifications.