RemoteLeaf Logo
Log in Sign up

Senior Data Engineer - Databricks

3 hours, 14 minutes ago
Slovakia, Spain, United Kingdom, Germany, Poland, Romania
Full-time
Senior
Data Engineer
Software Development
Apache Spark AWS Azure CI/CD Databricks Git PostgreSQL Python Secrets Management Serverless SQL SQL Server
Apply Now
Intetics

Intetics

Intetics is a top custom software development company with 28 years of experience, offering high-quality software applications and AI/ML integration. They excel in various industries and provide global talent solutions for exceptional project outcomes.

Internet Software & Services
1K-5K
Founded 1995
View All Jobs 81

Description

  • Own Databricks production support, including monitoring, alerting, and incident response across all production data flows.
  • Maintain SLA performance metrics for data pipeline delivery and report platform health to internal and external stakeholders.
  • Identify and implement pipeline optimizations to reduce compute costs, improve throughput, and shorten processing windows.
  • Migrate legacy ETL/ELT pipelines to Databricks and build automation to minimize manual intervention during transitions.
  • Support new customer onboarding by provisioning, validating, and hardening tenant data pipelines.
  • Design and build high-performance pipelines that ingest, transform, and serve ERP and CRM data at scale across Azure and AWS.
  • Own Delta Lake architecture, including schema design, partitioning, data quality enforcement, and incremental processing patterns.
  • Enforce data security best practices, including role-based access control, secrets management, and compliance requirements.
  • Implement data quality monitoring and observability for pipeline health and ML model inputs.
  • Apply and enforce multi-tenant data isolation patterns to ensure reliable and secure delivery for enterprise customers.
  • Partner with Enterprise Architecture to integrate data pipelines with the broader AI and analytics ecosystem.
  • Support on-call rotation and after-hours incident response across multiple time zones.
  • Maintain technical documentation, runbooks, and architectural decision records.
  • Apply CI/CD best practices, including version control, automated testing, and deployment tooling for data pipelines.

Requirements

  • 4+ years of data engineering experience.
  • At least 2 years of experience with Databricks or the Apache Spark ecosystem across Azure and/or AWS.
  • Proficiency in PySpark, SQL, and Python with experience building and operating production-grade pipelines under SLA constraints.
  • Hands-on experience with Delta Lake, including schema evolution, ACID transactions, optimize/vacuum lifecycle, and incremental and streaming processing patterns.
  • Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments.
  • Solid working knowledge of PostgreSQL, including query optimization and schema design.
  • Experience supporting and maintaining legacy ETL tooling such as SSIS, Informatica, or custom Python/SQL pipelines.
  • Experience supporting large-scale multi-tenant architectures with tenant isolation, per-tenant performance, and data privacy.
  • Proven ability to work collaboratively across data science, product, and infrastructure teams in a cross-functional environment.
  • Strong understanding of data governance, security, and compliance principles for enterprise data.
  • Experience operating Databricks workspaces across both Azure and AWS, including cost governance, cluster management, and cross-cloud data access, preferred.
  • Experience optimizing Databricks workloads in a serverless environment, preferred.
  • Experience with Microsoft SQL Server in a data engineering or ETL context, preferred.
  • Exposure to ML feature engineering or feature stores such as Databricks Feature Store or Feast, preferred.
  • Experience with customer onboarding automation or Infrastructure as Code patterns for provisioning tenant data pipelines, preferred.
  • Databricks Certified Data Engineer Associate or Professional certification, preferred.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Mindrift.ai: Be the “I” in AI

Senior Python Data Scraping Engineer (Freelance)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking a Senior Python Data Scraping Engineer for a freelance Tendem project to execute complex web data extraction and validation within a hybrid AI + human workflow.

United States Part-time Senior Backend Engineer Data Engineer
$94k-$94k
AJAX AWS Docker GitHub JavaScript JSON LLM Python Selenium
2 hours, 59 minutes ago
MediaRadar

Data Quality Engineer

MediaRadar 51-250 Media

MediaRadar is hiring a Data Quality Engineer to build and operate the automated tooling, monitoring, and dashboards that keep governed data accurate, measurable, and continuously validated across the company’s data pipelines and platforms.

India Full-time Mid Level Data Engineer QA Engineer
Python SQL
2 hours, 59 minutes ago
Mindrift.ai: Be the “I” in AI

Senior Python Data Scraping Engineer (Freelance)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is hiring a Senior Python Data Scraping Engineer for the Tendem project to support a hybrid AI + human data extraction workflow focused on accurate, structured web data delivery.

United States Part-time Senior Backend Engineer Data Engineer
$94k-$94k
AJAX AWS Docker GitHub JavaScript JSON LLM Python Selenium
2 hours, 59 minutes ago
Mindrift.ai: Be the “I” in AI

Senior Python Data Scraping Engineer (Freelance)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking a Senior Python Data Scraping Engineer for a part-time remote Tendem project to handle complex web data extraction and validation within a hybrid AI + human workflow.

Chile Colombia Peru Mexico Argentina Brazil Part-time Senior Backend Engineer Data Engineer
$52k-$52k
AJAX AWS Docker GitHub JavaScript JSON Python Selenium
3 hours, 14 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers