Senior Data Engineer

Job Overview:We are seeking a highly skilled and motivated Data Engineer to join our innovative team. As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support our data platform initiatives. You will collaborate closely with cross-functional teams to ensure the availability, reliability, and performance of our data systems and solutions.
Roles & Responsibilities:
Data Pipeline Development:Build a Data Platform - design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data to produce a cloud cost chargeback dataset.Develop robust ETL (Extract, Transform, Load) processes to integrate data from diverse sources into our data ecosystem.Implement data validation and quality checks to ensure accuracy and consistency.
Data Modeling and Architecture:Design and maintain data models, schemas, and database structures to support analytical and operational use cases.Optimize data storage and retrieval mechanisms for performance and scalability.Evaluate and implement data storage solutions, including relational databases, NoSQL databases, data lakes, and cloud storage services.
Data Integration and API Development:Build and maintain integrations with internal and external data sources and APIs.Implement APIs and web services for data access and consumption.Ensure compatibility and interoperability between different systems and platforms.
Data Pipelines & Orchestration:Ability to build reliable, observable data workflows.Knowledge with cloud platform (Azure data factory, Synapse, AWS S3, BiqQuery, etc.)Scheduling, dependencies, retries, backfills and data validation.
Data Infrastructure Management:Configure and manage data infrastructure components, including databases, data warehouses, data lakes, and distributed computing frameworks.Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.Implement data security controls and access management policies to protect sensitive information.
Collaboration and Documentation:Collaborate with data analytics, and other stakeholders to understand data requirements and deliver tailored solutions.Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.Provide technical guidance and support to team members and stakeholders as needed.
Requirements: Bachelor’s degree in computer science, Engineering, Information Systems, or related field.Proven experience in data engineering, software development, or related roles.Proficiency in programming languages is commonly used in data engineering (e.g., Python, Java, Scala, etc.).Strong knowledge of database systems, data modeling techniques, and SQL proficiency. • Proficiency with ETL tools commonly used in data engineering (e.g., Databricks, Azure Data Factory).Experience with big data technologies and frameworks (e.g., Hadoop, Spark, Kafka, etc.). • Familiarity with cloud platforms and services (e.g., AWS, Azure, GCP, Snowflake, etc.).Excellent problem-solving skills and attention to detail.Experience with Cloudability Data Platform a plus6-8 years’ experience