Sr. Manager, Experimentation & Product Insights

Job Description

The Job

The Senior Manager of Product Insights & Experimentation will help evolve HBO Max Product testing capabilities that drive quantifiable growth in user engagement and retention through feature enhancements. This will include working with near-term capabilities while developing long-term organizational and process requirements to create a scalable Product experimentation pipeline. In addition, it will be essential to collaborate with the Product and Product Analytics teams to define and execute a meaningful test and learn agenda, comprised of both small and big loop tests that help to significantly grow the business.

The ideal candidate will have experience with A/B tests, working with testing platforms, offline evaluation and simulations, and can collaboratively work with product and engineering leaders to meaningfully translate business requirements into solutions. In addition, this team will help analyze user behavioral data to identify patterns, uncover opportunities, and create a common understanding of how people are interacting with the product.

The Daily

Leadership:
• Assess current capabilities and develop the roadmap for necessary growth/process improvement for Product Experimentation.
• Manage existing Product Experimentation resources and assess future hiring needs.

Experimentation:
• Lead analysis and execution of experiments in the areas of customer acquisition, streaming quality, application performance, and commerce.
• Share and promote experimentation best practices.
• Work with Product, Engineering and Product Analytics teams to ensure required event tracking is in place.

Analysis automation:
• Effectively scale analysis from a single A/B test to standardize metric definitions, statistical tests and results visualizations.
• Leverage 3rd party tools (e.g. Optimizely) and identify necessary internal development or external vendors to improve scale through self-serve models and automation.

Partnership:
• Partner closely with product managers, the experimentation platform team, data engineers to improve platform capabilities, drive development of new features and testing strategies.

Deep analysis:
• Work closely with the Product Analytics team to deliver reporting and analyses aimed at developing a deep understanding of in-app user engagement.
• Perform ad hoc analysis and build compelling presentations and propose data-driven decisions to shape retention strategies – supported by data from experiments.
• Synthesize large volumes of data and complex concepts into clear accessible learnings for a broad audience

The Essentials
• 8+ years of hands-on experience designing and executing A/B and MV tests. Experience with Optimizely preferred.
• 5+ years of experience using SQL.
• 5+ years of Python or other statistical programming language.
• 4+ years leadership experience (includes people management, mentoring interns, technical leadership, and project management).
• Deep understanding of statistical concepts like hypothesis testing, Bayesian decision-making, etc.
• Experience with bandit optimization, adaptive experimentation, and large-scale inference methods preferred.
• Knowledge of observational causal inference and design (regression adjustment, propensity matching) or quasi-experimental design preferred.
• Hands-on experience in advanced data analysis
• Strong communication skills, for both technical and non-technical audiences.
• Have a demonstrated history of translating complex data sets into clear and actionable narratives and communicate opportunities and challenges to stakeholders
• Experience and aptitude to deliver high-quality solutions on short timelines

Nice to Haves
• Experience thinking strategically to interpret market and consumer information, preferably about a subscription service or large e-commerce business.
• Familiarity with data exploration and data visualization tools like Tableau or Looker.
• Advanced degree in an analytical field or equivalent experience.

Jobcode: Reference SBJ-g6pp70-3-149-250-1-42 in your application.