Lead Data Engineer, Data Platform Team

Job Description

At Disney Streaming, data is critical to its operations and growth. The Services & Data Engineering team is responsible for data acquisition, real-time and stream processing, analysis, and distribution of data for operational and analytical uses. Our platform is built on public cloud and open source technologies and offers transparency into customer behavior and business performance.

The Data Platform team within Services & Data Engineering is hiring a Lead Data Engineer to build a secure data platform that powers our streaming products and can scale with the growth of our data and organization. You'll be tasked with raising the bar by innovating and adopting new patterns and technologies to keep your teams on the bleeding edge of development. You'll get to understand the challenges of facilitating a high-quality, flawless media streaming experience across mobile, connected devices, and web, all while using the latest technologies available from AWS and beyond.

At Disney Streaming we feel strongly that teams should own their own processes, decide their own technologies, and design solutions for the long term. If you're interested in working in a highly collaborative team environment like this, please get in touch - we'd love to hear from you!

Responsibilities:
• Work within a cross-functional team of engineers to build software for a large-scale data processing ecosystem supporting real-time and batch data that underpins all analytic and operational data flows
• Build tools and services to support data discovery, lineage, governance, and privacy compliance across the data platform
• Build and maintain a CICD process for infrastructure components to ensure high availability while remaining nimble
• Leadership and responsibility for technical architecture, reliability, resilience, and scalability of the Disney Streaming data platform
• Influence and drive software engineering and architecture best practices and standards within the team and wider organization, along with a culture of quality, innovation, and experimentation
• Work with other teams: evangelize the platform, best-practices, data driven decisions; identify new use cases and features and drive adoption

Basic Qualifications:
• 6+ years of software engineering, ideally Python and Scala
• 4+ years' experience engineering big-data solutions using technologies like Databricks, EMR, and Spark
• 3-5 years' experience deploying and running cloud-based data solutions (AWS, GCP, Azure) and familiar with tools such as CloudFormation, Kinesis, ECS, S3 (or equivalent)
• Knowledge and experience with distributed systems, their development, operations, observability; in-depth understanding of data partitioning and sharding techniques
• Experience with streaming data pipelines using tools such (Kinesis, Kafka, Spark, Storm, or Flink) and data warehouses (Redshift, Snowflake)
• Previous experience taking ownership of features/functionality and delivering projects end-to-end with high quality and robust architecture.
• Ability to dive deep into any technical component as well as understand and drive the overall systems architecture.
• You're a self-learner, independent, and have excellent problem-solving skills.
• You care deeply about craftsmanship in your software, and can work backwards from the customer experience.
• You have excellent written and verbal communication skills.

Preferred Qualifications:
• Experience building distributed systems with microservices and/or service-oriented architectures
• Familiarity with workflow scheduler tools such as Airflow
• Familiarity with metadata management, data lineage, and principles of data governance
• Experience building streaming data platforms, solving problems including event definition, evolution, processing, patterns and best practices, scalability, reliability, observability, validation etc.

Required Education

Bachelor's degree in Computer Science or related field or equivalent work

Jobcode: Reference SBJ-rbqkn6-18-221-146-223-42 in your application.