Este é o meu perfilContato pela CazVidEste perfil está escrito em inglês
Sri
Senior Data Engineer
Contato
Competências
Demonstrated experience with Python.
Demonstrated experience with SQL.
Demonstrated experience with Java.
Demonstrated experience with Scala.
Demonstrated experience with PL/SQL.
Formação acadêmica
Resumo
Senior Data Engineer open to new opportunities
Experiência
Senior Data Engineer
Project: Modernizing and centralizing clinical data infrastructure for healthcare analytics and reporting. The platform integrates multiple clinical and enterprise source systems and processes high-volume record and event data across cloud analytics environments while maintaining HIPAA-compliant security and production reliability.. Design and develop real-time clinical data ingestion pipelines using Apache Kafka, PySpark, Azure Data Lake, and Delta Lake, processing high-volume clinical records and events through a Bronze/Silver/Gold Medallion architecture.. Extract and ingest clinical data from Epic EHR and SQL Server enterprise systems, standardizing source data for downstream cloud analytics.. Lead modernization of legacy cross-system workloads through Azure Data Factory and Databricks, converting existing processes into scalable, metadata-driven pipelines and improving processing efficiency.. Architect and enforce HIPAA-compliant data masking, tokenization, and role-based access controls across Snowflake and Delta Lake to protect sensitive PHI across processing layers.. Design and deliver REST API integration frameworks to securely exchange structured datasets between clinical and enterprise applications and cloud analytics platforms.. Optimize Snowflake workloads through query tuning, warehouse right-sizing, clustering strategies, and Snowpipe loading patterns to improve query response times.. Automate data platform deployments with Terraform, Jenkins, GitHub Actions, and Kubernetes, supporting CI/CD delivery across development, testing, and production.. Implement production monitoring with Prometheus and Grafana to strengthen pipeline observability and support production availability.
Senior Data Engineer
Project: Developed and enhanced a healthcare data platform for claims and member eligibility data across operational sources and a large-scale healthcare record base. The platform combined Lakehouse processing, Snowflake warehousing, real-time event processing, and automated data quality controls.. Led the build-out of scalable data processing layers using Databricks, Delta Lake, Apache Iceberg, and AWS S3 for large-scale claims and member eligibility batch workloads.. Built ingestion pipelines from Oracle and SQL Server transactional databases into the Lakehouse for downstream processing.. Spearheaded consolidation of PostgreSQL, MongoDB, and Cosmos DB data into Snowflake, creating consistent datasets for enterprise reporting and analytics across operational sources.. Developed real-time event processing with Kafka Streams to identify rule-based anomaly patterns and support downstream healthcare claims analytics.. Applied Apache Spark optimization techniques, including partition tuning, broadcast joins, caching, and resource allocation, improving processing efficiency and eliminating recurring data spill issues.. Integrated automated data validation frameworks with GitHub Actions and Jenkins, applying quality checks to production-bound datasets before downstream loading.
Data Engineer
Co-led the transition of enterprise analytics workloads from on-premises infrastructure to Azure, developing PySpark ingestion pipelines for structured and semi-structured data.. Introduced Delta Lake storage layers with schema evolution to support changing upstream data formats without disrupting downstream pipelines.. Built Kafka and Azure Data Factory pipelines for incremental loading and change data capture (CDC), supporting fresher data across analytical workloads.. Developed Snowflake dbt models using incremental strategies, automated tests, and documentation to deliver reliable datasets for Tableau and Power BI analytics.. Created reusable REST API integration workflows using Java and Spring-based microservices to standardize ingestion from enterprise SaaS applications.
Data Engineer
Developed and optimized batch ETL workloads with Apache Spark, PySpark, and Hadoop for high-volume historical transactional data.. Consolidated data from AWS S3, Amazon Redshift, Azure Blob Storage, and MongoDB into centralized, analytics-ready data marts.. Migrated legacy cron-based workflows to Apache Airflow DAGs with dependency management, automated validation, and failure alerting.. Evaluated and prototyped Apache Iceberg to address partition evolution challenges and improve query efficiency across large historical datasets.. Optimized Spark workloads through executor memory, shuffle partition, and caching strategies to resolve production memory and processing issues.
Junior Data Engineer
Developed SQL scripts, stored procedures, and Python-based transformation workflows to process operational data into MySQL databases.. Created automated validation checks for null values, duplicate records, and data integrity issues before downstream reporting.. Built foundational Power BI dashboards and ad hoc data views using Google BigQuery to support operational reporting.. Monitored AWS EC2 jobs, troubleshooting data loads and supporting Docker/Jenkins CI/CD workflows.