Full Time Job

Senior Big Data Platform Architect

Epic Games

Cary, NC 10-19-2021
Apply @ Employer
  • Paid
  • Full Time
  • Entry (0-2 years) Experience
Job Description
Epic is seeking a Senior Big Data Platform Architect to help manage our big data batch platform infrastructure.

You'll take responsibility for defining, running and optimising our batch hardware running across several AWS EMR clusters.

• Supporting and augmenting our production and development Hadoop clusters
• Define, improve and bake our custom Hadoop implementation
• Maintain and extend our Hive metastore as necessary
• Managing resource utilization across competing business priorities
• Monitoring and tuning cluster performance
• Identifying opportunities to leverage components of the Hadoop ecosystems which are not yet part of our architecture
• Ensuring a high level of security and auditing around our private and sensitive data
• Providing after-hours and weekend support for routine maintenance as well as troubleshooting outages
• Remaining current on the latest Hadoop and distributed computing technologies

• Experience using Apache Bigtop
• Deep knowledge of the low level workings of different big data execution engines: Spark, Tez, Hive, Trino/Presto
• Experience in performance tuning of auto scaling Hadoop clusters, Spark routines and Map/Reduce jobs
• Experience in security aspects of Hadoop and Linux such as File based ACLs, Kerberos, Ranger, and encryption
• Experience in independently performing root cause analysis of issues and presenting mitigation recommendations
• Experience in at least one scripting language (example bash, Perl etc.)
• Experience in debugging issues related to Java or Python applications
• Deep understanding of different storage/table formats and how they integrate with big data engines, e.g. Parquet, ORC, Iceberg

Jobcode: Reference SBJ-rej1p0-52-23-219-12-42 in your application.

Company Profile
Epic Games

Founded in 1991, Epic Games is a leading interactive entertainment company and provider of 3D engine technology. Epic operates Fortnite, one of the world’s largest games with over 350 million accounts and 2.5 billion friend connections. Epic also develops Unreal Engine, which powers the world’s leading games and is also adopted across industries such as film and television, architecture, automotive, manufacturing, and simulation.