RemoteLeaf Logo
Log in Sign up

[Job-32448] Master Data Developer (Databricks), Colombia

4 days, 14 hours ago
Colombia
Full-time
Mid Level
Data Engineer
Software Development
Agile Apache Spark AWS Databricks Java Python Scala
Apply Now
CI&T

CI&T

CI&T is a global digital technology agency empowering agile growth for leading companies through advanced technologies with a team of 2000 experts worldwide.

Internet Software & Services
5001+
Founded 1995
View All Jobs 22

Description

  • Build, optimize, and maintain scalable data pipelines and ETL/ELT processes.
  • Implement data architectures including data lakes, warehouses, and integration layers.
  • Establish data quality checks, validation processes, and monitoring systems.
  • Develop and maintain integrations across data sources, third-party tools, and analytics platforms.
  • Collaborate with data, analytics, and business stakeholders to translate requirements into technical solutions.
  • Monitor and optimize pipeline performance, query efficiency, and storage costs.
  • Create and maintain technical documentation for data architectures, pipelines, and processes.
  • Implement data security best practices and support compliance with data privacy regulations.
  • Use AI tools and automation to improve data engineering workflows and productivity.

Requirements

  • Hands-on experience migrating Databricks environments using the databricks/migrate tool, including notebooks, jobs, clusters, secrets, and ACLs.
  • Experience migrating Unity Catalog to AWS Glue Catalog.
  • Experience migrating notebooks, jobs, DAG pipelines, and Delta Live Tables (DLT) pipelines.
  • Experience refactoring DBFS /mnt/ storage paths to S3-backed locations.
  • Experience migrating Delta Lake tables and raw files to Iceberg format on Amazon S3, including incremental synchronization.
  • Experience re-bootstraping Auto Loader or DLT streaming checkpoints and implementing event_id deduplication.
  • Experience validating migrations through row counts, checksums, sample queries, and ETL runtime comparisons.
  • Knowledge of AWS services including S3, Glue, Secrets Manager, IAM, Athena, Lake Formation, and EMR.
  • Proficiency with Spark SQL, PySpark, Java, Python, and Scala.
  • Experience working in Agile delivery pods, including ticket updates, standups, and code reviews.
  • Databricks Certified Consultant certification (Associate or Professional) preferred.
  • Experience migrating Databricks Runtime workloads to EMR on AWS preferred.

Benefits

  • Premium healthcare.
  • Meal voucher.
  • Maternity and parental leave.
  • Mobile services subsidy.
  • Sick pay and life insurance.
  • CI&T University learning opportunities.
  • Colombian holidays.
  • Paid vacation.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Via Logic

Cloud Platform and Data Engineer [U.S. Citizenship Required, Public Trust Eligibility]

Via Logic 1-10 Internet Software & Services

Via Logic is hiring a remote Cloud Platform and Data Engineer to support DHS programs by operating secure AWS infrastructure and Salesforce-connected applications, integrations, and data workflows.

United States Full-time Mid Level Data Engineer Platform Engineer
$100k-$120k
Agile Ansible AWS Bash Cybersecurity DevSecOps Docker Encryption Git Helm Java Jenkins Kubernetes Microservices Monday.com New Relic PostgreSQL Power BI Python Redis Salesforce Secrets Management SQL Tableau Terraform
14 hours, 16 minutes ago
C

Data Engineer - Web Scraping (3 months contract)

CXG

Join CXG as a remote Data Engineer - Web Scraping on a 3-month contract to maintain web-scraping operations and build data pipelines that support insights for global luxury brands.

India Vietnam Pakistan Taiwan Morocco Contract Junior Data Engineer
Apache Airflow Azure dbt Git Python SQL Terraform
1 day, 2 hours ago
Inductive

Data Engineer

Inductive 51-200 Health Care Providers & Services

Join InductiveHealth as a Data Engineer to modernize and operate secure, scalable public health data pipelines that support disease surveillance and decision-making for public health agencies.

United States Full-time Mid Level Data Engineer
$130k-$145k
Agile AWS Azure CI/CD Docker Elasticsearch Encryption GitHub GitHub Actions Grafana Kafka Kubernetes OpenSearch Python Secrets Management SQL Terraform
1 day, 14 hours ago
Ampd Energy

Senior Engineer - Platform & Data Infrastructure

Ampd Energy 51-200 Electrical Equipment

Ampd Energy is seeking a hands-on senior software engineer to scale and stabilize Enernet, its cloud platform for monitoring and managing connected battery energy storage systems on construction sites.

India Full-time Senior Data Engineer
AWS ClickHouse Go Grafana GraphQL InfluxDB Kafka NestJS Node.js REST API SQL Terraform TimescaleDB TypeScript Vue.js
1 day, 14 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers