Harsh

Senior Data Engineer

Contributed to the design and development of a Snowflake-based data warehousing solution for a leading U.S. bank as part of the core engineering team, helping reduce operating costs by 30% and accelerate customer onboarding timelines. Developed Python-based data transformation pipelines using Snowpark, extending existing SQL/DBT logic with programmatic UDFs and DataFrame operations to handle complex business rules. Leveraged Snowflake Cortex AI functions (SUMMARIZE, SENTIMENT, COMPLETE) to enable in-warehouse text analytics and ML-driven insights, reducing dependency on external ML infrastructure. Wrote and optimized complex queries and procedural logic using SnowSQL and Snow PL/SQL for stored procedures, transformations, and automation scripts. Led implementation of Data Vault modeling (Raw Vault strategy) to standardize enterprise data management, and delivered real-time and batch-based Snowflake views on top of the model for data analysts and business users. Deployed 50+ DBT models with complex SQL transformations, managing Snowflake SQL scripts and DBT models under Git version control, and established CI/CD pipelines using GitLab for automated testing and deployment. Designed and managed incremental CDC pipelines into Snowflake using DBT, and built an automated reconciliation testing framework to validate data parity between on-prem (Oracle) and cloud (Snowflake) sources, ensuring data quality and reliability. Integrated Snowflake with AWS services (S3, Lambda, SNS/SQS) for hybrid ingestion, and built a real-time ingestion pipeline using Glue, Kinesis, EC2, and Redshift for near real-time data availability. Created Snowflake tasks, stored procedures, Snowpipe, streams, and masking policies for secure automated pipelines, and optimized performance/cloud costs using materialized views, clustering, and query tuning. Spearheaded full-scale data migration from on-prem Oracle systems to Snowflake, refining complex SQL joins in DBT and migrating legacy transformation logic from Datastage to a modern DBT/Snowflake stack. Built a Control-M-based framework to trigger automated table refresh tasks with full audit logging, and delivered multiple Raise The Bar (RTB) support engagements across Hadoop, Spark, Python, and Databricks