Big Data Lead

Job Description

ONLINE INFRASTRUCTURE

What We Do

We enable Epic's online services teams to build, deploy, and manage services that are used by more than half a billion players around the world. Our mission is to provide world class tools and platforms to improve the experience of our developers and make it easier, faster, and safer to build, operate, and scale their applications. We operate at massive scale as one of the largest cloud computing users in the world.

What You'll Do

As a Lead Big Data Admin/Architect, you will help manage our big data platform infrastructure. You'll take responsibility for defining, running and optimizing our batch hardware running across several AWS EMR and traditional Hive/Spark clusters.

In this role, you will
• Support and augment our production and development Hadoop clusters
• Maintain and extend our Hive metastore as necessary
• Manage resource utilization across competing business priorities
• Monitor and tune cluster performance
• Identify opportunities to leverage components of the Hadoop ecosystems which are not yet part of our architecture
• Ensure a high level of security and auditing around our private and sensitive data
• Provide after-hours and weekend support for routine maintenance as well as troubleshooting outages
• Remain current on the latest Hadoop and distributed computing technologies

What we're looking for
• Deep knowledge of the low level workings of different big data execution engines: Spark, Tez, Hive, Trino/Presto, EMR
• Experience in performance tuning of auto scaling Hadoop clusters, Spark routines and Map/Reduce jobs
• Experience in security aspects of Hadoop and Linux such as File based ACLs, Kerberos, Ranger, and encryption
• Experience in independently performing root cause analysis of issues and presenting mitigation recommendations
• Experience in at least one scripting language (example bash, Perl etc.)
• Experience in debugging issues related to Java or Python applications
• Deep understanding of different storage/table formats and how they integrate with big data engines, e.g. Parquet, ORC, Iceberg
• Experience with any of the following is a plus: Snowflake, AWS RDS, MongoDB, any other RDBMS

Jobcode: Reference SBJ-g6ee9j-18-191-181-231-42 in your application.