Senior Data Engineer

Job Description

DATA ENGINEERING
What We Do
Our mission is to provide a world-class platform that empowers the business to leverage data that will enhance, monitor, and support our products. We are responsible for data ingestion systems, processing pipelines, and various data stores all operating in the cloud. We operate at a petabyte scale, and support near real-time use cases as well as more traditional batch approaches.
What You'll Do
You will be responsible for designing, building, and maintaining our data infrastructure to ensure the reliability and efficiency of our data and systems used by our Ecosystem Security Core team. Your role will include building and maintaining data pipelines which stream, transform, and load data from various products and managing the AWS infrastructure for our security & analytics platforms. Additionally, you will work with engineers, product managers, and data scientists to design and implement robust and scalable data services that support the Ecosystem Security mission while ensuring our user's privacy. Your work will directly combat bad actors and keep our platform safe for all users.
In this role, you will
• Design and implement automated end-to-end streaming & ETL process to prepare data for machine learning and ad-hoc analysis, including data anonymization
• Manage and scale the tools and technologies that we use to label data, which run on AWS
• Devise database structure and technology for storing and efficiently accessing large data sets (millions of records) of different types (text, images, videos, etc.)
• Use and implement data extraction APIs
• Write and invoke custom SQL procedures
• Support data versioning strategies using automated tools, such as DVC
• Support devising strategies for labeling of new data by humans
What we're looking for
• Strong analytical background: BSc or MSc in Computer Science/Software Engineering or related subject - candidates without a degree are welcome as long as they have proven extensive hands-on experience
• Experience of ETL technical design, automated data quality testing, QA and documentation, data warehousing, and data modeling
• Experience with Python for interaction with Web Services (e.g. Rest and Postman)
• Experience with using AWS, Databricks, Snowflake, Elastic or other comparable large scale analytics platforms
• Experience monitoring and managing databases (we use Elasticsearch/MongoDB/PostgreSQL)
• Experience with SQL
• Experience with data versioning tools
• Experience developing and maintaining data automation infrastructure for streaming & ETL pipelines, such as Apache Airflow
• Experience with GraphDB would be a plus
• Demonstrated experience collaborating with product teams to understand how safety systems integrate with broader data systems

Note to Recruitment Agencies: Epic does not accept any unsolicited resumes or approaches from any unauthorized third party (including recruitment or placement agencies) (i.e., a third party with whom we do not have a negotiated and validly executed agreement). We will not pay any fees to any unauthorized third party. Further details on these matters can be found here.

Jobcode: Reference SBJ-6kk3vj-216-73-216-0-42 in your application.