Aniket - Data Engineer at DATA UNIVERSE

Bengaluru, Karnataka, Indiaen

Developed Python scripts to automate data transfer processes, improving efficiency and accuracy.

Implemented automation for AWS data handling, reducing manual effort and processing time.

Published on CazVid

Summary

Recent Business Administration graduate with practical experience in data automation, scripting, and cloud data handling at DATA UNIVERSE. Eager to leverage technical skills in data engineering roles.

Key Achievements

Designed logging and data cleaning routines to ensure data integrity and traceability.

Skills

Python ProgrammingIntermediate

Proficient in scripting for data automation, integration, and process optimization.

Data ManagementIntermediate

Experience in handling data extraction, transformation, and loading (ETL) processes.

AWS ServicesBeginner

Basic understanding of AWS cloud services for data storage and processing.

Problem SolvingIntermediate

Ability to develop automated solutions to streamline data workflows and improve performance.

Experience

Data Engineer

DATA UNIVERSE

Make python script to take data from database and put in s3 bucket. Wrote a Python script (using PyCharm IDE) to automate connections to AWS services and handle file movement, cleaning, and logging.. Used Pandas to clean and transform raw data (removing, renaming columns) before uploading to S3 Final Bucket.. Used Python with Boto3 to interact with AWS (upload/download/check files). Check whether Data is ingested in Landing Bucket or not. Worked with Business to understand requirements and propose technical implementation approach.. Developed data ingestion pipelines for different source systems.. Work with testing team on building test cases and provide support on testing phase.. Used SQL in Redshift to clean, transform, and validate the loaded data for analytics and reporting.. Work with Business during User Acceptance testing.. I create glue job to copy files from Landing S3 Bucket and paste it into Final S3 Bucket. Create IAM role for Amazon Glue and give it full access of S3, Glue. Run the glue job manually. Check whether files got copied in Final Bucket or not. Create Lambda Function to Trigger Glue job. Create separate IAM role for Lambda and give it full access of S3, Glue, Lambda. Upload file in Landing Bucket and check Lambda Job got automatically invoked or not and check glue job got triggered or not.. Create Redshift Cluster.. Use Query editor or any SQL Workbench to connect with Redshift Cluster.. Create Table inside Redshift Cluster.. Use copy command to copy data from S3 Final Bucket to Redshift table.. Check all data got inserted or not.