RemoteLeaf Logo
Log in Sign up

Sr Data Engineer

3 weeks, 2 days ago
United States
Full-time
Senior
Data Engineer
Software Development
Apache Airflow Apache Spark Azure CI/CD Dagster Databricks dbt Git Python SQL
Apply Now
Protective Life

Protective Life

Protective Life Insurance Company offers financial security through insurance and investment products, serving people and protecting families for over a century.

Insurance
1K-5K
Founded 1907
View All Jobs 24

Description

  • Design, develop, and maintain Databricks pipelines using Python, SQL, Apache Spark, Delta Lake, dbt, and dlt.
  • Build reliable Bronze/Raw ingestion from APIs, databases, files, cloud storage, and SaaS platforms, including incremental loading, state management, and replay.
  • Develop Silver/Prep transformations for cleansing, validation, standardization, deduplication, conformance, and enrichment.
  • Create Gold/Prod data products, including dimensional models, facts, dimensions, aggregates, serving tables, and ML feature and training datasets.
  • Define and maintain data contracts, quality rules, freshness expectations, and backward-compatibility assessments.
  • Orchestrate assets in Dagster Cloud, including schedules, sensors, dependencies, backfills, deployments, and observability.
  • Apply Unity Catalog governance, access controls, lineage, and Azure Key Vault secret management.
  • Optimize Spark jobs and Delta Lake processing for performance and cost, including MERGE, schema evolution, time travel, and OPTIMIZE.
  • Troubleshoot production failures, data-quality issues, source changes, late or duplicate data, backfills, and recovery procedures; participate in on-call support.
  • Build CI/CD, monitoring, alerting, documentation, runbooks, and architecture standards while collaborating with technical and business stakeholders.

Requirements

  • Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience.
  • 3+ years of experience building and supporting production data pipelines in cloud data platforms.
  • Strong hands-on Python and SQL skills.
  • Experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake, MERGE, and incremental loading.
  • Understanding of medallion or multi-layer lakehouse architecture and ELT patterns.
  • Experience ingesting data from APIs, relational databases, files, or SaaS applications.
  • Working knowledge of dimensional modeling, slowly changing dimensions, data quality, and data contracts.
  • Experience with orchestration tools such as Dagster, Databricks Workflows, Airflow, or Azure Data Factory.
  • Experience with Git, pull requests, automated testing, CI/CD, and Azure DevOps or comparable tools.
  • Preferred: Databricks certification, Unity Catalog, dbt on Databricks, Python Delta Lake modeling, dlt/Airbyte/Meltano/Fivetran, and Dagster assets, checks, sensors, schedules, or branch deployments.

Benefits

  • Comprehensive health, dental, and vision insurance.
  • Mental health benefits and an employee assistance program.
  • Paid time off, paid parental leave, short-term disability, and a cultural observance day.
  • Healthcare account contributions, a pension plan, and a 401(k) with company matching.
  • ProHealth Rewards wellness platform with opportunities to earn cash rewards.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Aker Systems

Senior Data Platform Engineer

Aker Systems 51-250 IT Services

Aker Systems is hiring a Senior Data Platform Engineer to build and operate secure, scalable cloud-native data platforms for public-sector and utilities clients.

United Kingdom Full-time Senior Data Engineer
Agile Ansible Apache Airflow Apache Spark Argo CD AWS Azure Bash CI/CD Datadog dbt Docker Elasticsearch ELK Stack Git GitHub GitHub Actions GitLab GitLab CI Grafana Helm Java Jenkins Kafka Kubernetes OpenSearch Packer Prometheus Python SQL Terraform
21 hours, 23 minutes ago
Capco

Engenheiro de Dados Senior - Remoto

Capco 5K-10K Internet Software & Services

A Capco é uma consultoria global de tecnologia e negócios que atua nos setores financeiro e de energia, oferecendo oportunidades de carreira em um ambiente colaborativo, inclusivo e orientado à inovação.

Brazil Full-time Senior Data Engineer
21 hours, 23 minutes ago
Karat

Senior Data Engineer (India)

Karat 251-1K Internet Software & Services

Karat is hiring a Senior Data Engineer in Bengaluru to build reliable cloud-based data infrastructure, pipelines, and models that turn interview and performance data into actionable insights across the company.

India Full-time Senior Data Engineer
Apache Airflow AWS Docker Git PostgreSQL Python SQL Terraform
22 hours, 8 minutes ago
RevStar

Databricks Engineer

RevStar 51-250 Internet Software & Services

RevStar is seeking a remote, US-based Databricks Engineer to build and optimize cloud-agnostic Lakehouse data pipelines and MLOps frameworks for enterprise AI projects across AWS, Azure, and GCP.

United States Mexico Argentina Brazil Chile Colombia Venezuela Contract Mid Level Data Engineer
Agile Apache Airflow Apache Spark AWS Azure CI/CD Databricks Feature Engineering HIPAA Kafka MLflow MLOps Prefect Python SQL Terraform
1 day, 22 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers