Senior Data Engineer
Job Description
We are seeking motivated engineers to build, maintain and use our platforms and services to manage data at MLB. We are looking for those with experience with Infrastructure as code, automated tests as well as modern observability principles. The platforms we create and manage securely move, transform, and enrich data for both analytic consumption as well as the delivery of actionable intelligence throughout the MLB organization. These platforms and services create a single source of truth of well-curated data domains that MLB depends on.
You will have the opportunity to contribute to many different projects and transfer your ideas into solutions for some of the most meaningful data problems in MLB!
SOME OF OUR SERVICE CAPABILITIES:
● Connect to a wide variety of source and target systems
● Move data between systems
● Transform / Enrich data
● Create and manage MLB data products
● Organize disparate data into business domains
● Coordination services
EXAMPLES FROM OUR CURRENT AND FUTURE TECH STACK:
We use python modules like gcp libraries, tox, pytest, and panda regularly. In addition, we want to use frameworks like DBT and Meltano. Kafka and containers will be a regular part of our infrastructure. Some of the GCP services we use are for Storage, Containers, Secrets, Big Query and monitoring. It is important to understand how to use SQL to enrich and organize data optimally in GCP.
DEVOPS TOOLS:
We use tools like GitLab, Terraform, Ansible, Bash, and Docker Compose.
METHODOLOGIES:
We are an agile shop that believes in Infrastructure as Code (IaC) and crafting lite-weight asynchronous services with clean code that are tested in an automated way. We believe being very active on pull requests helps us win as a team!
YOUR BACKGROUND...
The ideal candidate should have 5+ years in software engineering delivering services with proficiency in Python and SQL. You have a solid understanding of IaC, SDLC, observability and how to build highly available services. We are looking for a self-sufficient engineer who can wear many hats and is confident as an individual contributor.
WHAT YOU'LL DO...
● Analyze, design, code, test, configure and modify software for our platform, integrations and services using various programming languages, technologies and development methodologies.
● Design, develop, test, debug and implement platforms, pipelines, solutions and/or software tools, and utilities for the purpose of assuring acceptable performance and service levels.
● Participate in the automated delivery of software using source control, IaC throughout the entire delivery model
● Ensures that implemented platforms, pipelines and solutions are optimally monitored, with relevant alerts, logging and tracing that guarantees the durability, availability and performance of our services.
● Organize Data into well-curated data domains designed for consumption and performance in GCP to provide MLB a governed single source of truth.
● Complete documentation that contributes value, including but not limited to testing, training and software delivery