RemoteLeaf Logo
Log in Sign up

Data Engineering Lead

3 weeks, 4 days ago
United States
Full-time
Lead
Data Engineer
DevOps and Infrastructure
Agile Apache Airflow Apache Spark Azure CI/CD Docker Git Machine Learning PostgreSQL Python SQL
Apply Now
MediaRadar

MediaRadar

MediaRadar: Revolutionizing ad sales with comprehensive advertising intelligence for media buyers and sellers.

Media
51-250
Founded 2006
$7M raised
View All Jobs 6

Description

  • Architect and implement complex end-to-end data pipelines using Azure Databricks and PySpark.
  • Design, build, and maintain a scalable Medallion Architecture (Bronze/Silver/Gold) for data processing and delivery.
  • Spend 70-80% of time coding and performing technical stewardship as a hands-on 'player-coach' for the team.
  • Optimize Apache Spark jobs, tune Databricks units, define cluster policies, and implement caching strategies to minimize compute costs and improve performance.
  • Proactively audit and refactor pipelines every 3-6 months to maintain effectiveness and reduce cloud costs.
  • Develop a proactive monitoring and alerts framework to achieve 99.9% reliability and mitigate system issues before they impact end users.
  • Build and maintain an end-to-end Data Validation Framework (e.g., Great Expectations) to enforce data accuracy and consistency.
  • Ensure data is available in the Gold layer within the required 24-hour turnaround time and minimize job failure rates.
  • Architect high-performance PostgreSQL schemas (indexing, partitioning) and optimize complex analytical queries.
  • Lead a lean team toward cross-trained agility: manage sprint cycles, conduct code reviews, enforce CI/CD best practices, translate business requirements into technical user stories, and collaborate with ML and offshore teams for model integration and knowledge transfer.

Requirements

  • 10+ years of experience in Data or Software Engineering with deep codebase involvement.
  • 3+ years as a Technical Lead managing agile teams.
  • Mandatory experience with Python.
  • Mandatory experience with PostgreSQL and pgvector.
  • Mandatory experience with Azure Databricks, PySpark, and Delta Lake.
  • Experience with Docker, Git, Azure DevOps, and CI/CD pipelines.
  • Proven ability to lead lean, high-impact teams while maintaining high individual output and advocating cross-training.
  • Experience scaling data processing through automation and performance/cost optimization.
  • Desired: experience with Apache Airflow (workflow orchestration) and familiarity with Azure Kubernetes Service (AKS).
  • Eligible to work remotely within the USA.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Lola Blankets

Data Platform Engineer

Lola Blankets 1-10 Textiles, Apparel & Luxury Goods

Lola Blankets is hiring a Data Platform Engineer to own its analytics platform and support engineering work across product, operations, integrations, and platform reliability.

Anywhere Full-time Mid Level Data Engineer
Apache Airflow Dagster dbt LLM Prefect Python Snowflake SQL TypeScript
4 hours, 17 minutes ago
Soname Solutions

Oracle Data Engineer (with German Language)

Soname Solutions 11-50 Internet Software & Services

Soname Solutions is seeking a Senior Data Warehouse Developer to support a German telecom client by designing, optimizing, and evolving its multi-layer data warehouse environment.

Georgia Poland Full-time Senior Database Administrator Data Engineer
Oracle PostgreSQL Power BI SQL
6 hours, 19 minutes ago
Hyphen Connect

Synthetic Data Engineer (AI Data/Training)

Hyphen Connect 1-10 staffing & recruiting

A Synthetic Data Engineer at the organization will design and manage domain-specific synthetic data pipelines that support data processing and model training workflows.

China Mid Level AI (Artificial Intelligence) Data Engineer
Apache Airflow Apache Spark
6 hours, 51 minutes ago
Alpaca

Senior Data Engineer

Alpaca 51-250 Capital Markets

Alpaca is seeking a Senior Data Platform Engineer to build and operate the data management layer for its global brokerage infrastructure as it scales across customers, jurisdictions, and high-volume financial event streams.

North America Latin America Full-time Senior Data Engineer
Apache Airflow dbt Docker GCP Helm Kafka Kubernetes Python SQL Terraform Trino
8 hours, 38 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers