As a Data Engineer, you will be involved in the entire development lifecycle, from brainstorming ideas to implementing scalable solutions that unlock data insights....Key Responsibilities:
• Design, build, and sustain efficient, scalable and performant Data Engineering Pipelines to ingest, sanitize, transform (ETL/ELT), and deliver high-volume, high-velocity data from diverse sources.
• Ensure reliable and consistent processing of versatile workloads of granularity such as Real Time, Near Real Time, Mini-batch, Batch and On-demand.
• Translate business requirements into technical specifications, design and implement solutions with clear documentation.
• Develop, optimize, and support production data workflows to ensure comprehensive and accurate datasets.
• With Software Engineering mindset/discipline, adopt best practices in writing modular code that is maintainable, scalable and performant.
• Use orchestration and scheduling tools (e.g., Airflow, GitLab Runners) to streamline workflows.
• Automate deployment and monitoring of data workflows using CI/CD best practices....Qualifications:
• Masters or Bachelors degree in Computer science or associated discipline with relevant industry experience(3+ Years) in a data engineering role.
• Strong Proficiency in writing and analyzing complex SQL, Python or any 4GL.
• Strong experience with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools (e.g., Terraform, CloudFormation).
• Strong hands-on experience with Cloud Data Warehouses like Snowflake, Redshift, BigQuery or other big data solutions
• Expertise & Experience with Data Lake and Open table formats technologies (e.g., Apache Iceberg)
• Expertise & Experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Beam, Trino).
• Experience with real-time/streaming data technologies (e.g., Kafka, Kinesis, Spark Streaming).
• Experience in using modern pipeline orchestration tools such as Airflow
• Knowledge in data warehousing concepts, data modelling and performance optimizatio...We will also consider employment qualified applicants with criminal records in accordance with applicable law.