Senior Distributed Software Engineer - Big Data Infrastructure

Job Description

At Netflix, the big data platform is the core foundation in driving all of our product decisions that directly impact our customer experience. It is leveraged across the company to build data models for recommendations, analyze customer streaming experience, content take-rate, plus more. It is critical to the success of our business as we heavily rely on analyzing data to revolutionize internet TV!

We are the data infrastructure team that builds an ecosystem of microservices that expose the big data platform as a robust and highly available service to the rest of Netflix. We are constantly evolving our infrastructure. We are designing new microservices to take our architecture to the next level to keep pace with our increasing job execution and data needs.

Here is the vision of the data infrastructure team that is responsible for the orchestration layer.

Specifically, we are working on:
• Transitioning our data infrastructure into a more scalable and sustainable event driven architecture. For example, dropping a table in Metacat (our metadata service) would trigger an event to our S3 deletion service to clean up data. Finishing a job in Genie (our job management service) would trigger an event to our scheduler to start the next dependent job.
• Building an IFTTT service (called Microbots) to help us expand and scale on the usage pattern of event driven architecture.
• Elevating the architecture and functionalities of our workflow scheduling engine (called Meson) so we could continue to grow its functionality and scalability to match our growing use cases.
• Integrating Genie with our virtual execution environment Titus so it can evolve into a distributed big data job execution environment.
• Building our first version of Data access service to allow different ML applications to access data directly in our data warehouse.
• And many more ...
• If this aligns well with your passion, we would love to hear from you!

This would be your dream job if you enjoy:
• Working with a massive amount of data (100+ PB) that is growing rapidly.
• Understanding and solving real business needs at large scale by applying your software engineering and analytical problem solving skills.
• Architecting and building a robust, scalable, and highly available distributed infrastructure.
• Leading cross-functional initiatives and collaborating with engineers across teams.
• Sharing our experiences with the open source communities and contributing to Netflix OSS.

About you:
• You have 5+ years of experience in building large scale distributed applications.
• You are proficient in design and development of RESTful web services.
• You are an expert in Java. Python or Scala expertise is a plus.
• You have a BS/MS/PhD in Computer Science or related field.
• Most importantly, you are thrilled to help solve our big data challenges while revolutionizing internet TV!

To learn more about the team, here is our recent talk at re:Invent conference (in slides or video) and Strata that describes what we do in the big data space!

Jobcode: Reference SBJ-dyjq20-3-19-31-73-42 in your application.