RemoteLeaf Logo
Log in Sign up

[Job-32448] Master Data Developer (Databricks), Brazil

4 days, 14 hours ago
Brazil
Full-time
Senior
Data Engineer
Software Development
Agile Apache Spark AWS Databricks Java Python Scala
Apply Now
CI&T

CI&T

CI&T is a global digital technology agency empowering agile growth for leading companies through advanced technologies with a team of 2000 experts worldwide.

Internet Software & Services
5001+
Founded 1995
View All Jobs 22

Description

  • Build, optimize, and maintain scalable data pipelines and ETL/ELT processes.
  • Implement data architectures including data lakes, warehouses, and integration layers.
  • Establish data quality checks, validation processes, and monitoring systems.
  • Develop and maintain integrations across data sources, third-party tools, and analytics platforms.
  • Partner with Data & Analytics Managers and business stakeholders to translate needs into technical solutions.
  • Optimize pipeline performance, query efficiency, and storage costs.
  • Create and maintain technical documentation for data architectures, pipelines, and processes.
  • Apply data security best practices and data privacy compliance requirements.
  • Use AI tools and automation to improve data engineering workflows and productivity.

Requirements

  • Hands-on experience migrating Databricks environments using the databricks/migrate tool, including notebooks, jobs, clusters, secrets, and ACLs.
  • Experience migrating Unity Catalog to Glue Catalog.
  • Experience migrating notebooks, job DAG pipelines, and DLT pipelines.
  • Experience refactoring DBFS /mnt/ storage references to S3-backed locations.
  • Experience migrating Delta Lake tables and raw files to Iceberg format on S3, including incremental synchronization.
  • Experience re-bootstraping Auto Loader/DLT checkpoints and implementing event_id deduplication for streaming workloads.
  • Ability to validate migrations through row counts, checksums, sample queries, and ETL runtime comparisons.
  • Experience recreating secret scopes in AWS Secrets Manager.
  • Knowledge of AWS S3, Glue, Secrets Manager, IAM, Athena, Lake Formation, and EMR.
  • Proficiency in Spark SQL, PySpark, Java, Python, and Scala.
  • Experience working in Agile delivery pods, including ticket updates, standups, and code reviews.
  • Databricks Associate or Professional certification and DBR-to-EMR migration experience on AWS (preferred).

Benefits

  • Health and dental insurance.
  • Meal and food allowance.
  • Childcare assistance and extended paternity leave.
  • Profit Sharing and Results Participation (PLR).
  • Life insurance.
  • Gym and wellness partnerships through Wellhub (Gympass) and TotalPass.
  • Continuous learning through CI&T University, online learning partnerships, and a language-learning platform.
  • Physical, mental, and overall well-being platform, plus pregnancy and responsible-parenting support.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Via Logic

Cloud Platform and Data Engineer [U.S. Citizenship Required, Public Trust Eligibility]

Via Logic 1-10 Internet Software & Services

Via Logic is hiring a remote Cloud Platform and Data Engineer to support DHS programs by operating secure AWS infrastructure and Salesforce-connected applications, integrations, and data workflows.

United States Full-time Mid Level Data Engineer Platform Engineer
$100k-$120k
Agile Ansible AWS Bash Cybersecurity DevSecOps Docker Encryption Git Helm Java Jenkins Kubernetes Microservices Monday.com New Relic PostgreSQL Power BI Python Redis Salesforce Secrets Management SQL Tableau Terraform
14 hours, 16 minutes ago
C

Data Engineer - Web Scraping (3 months contract)

CXG

Join CXG as a remote Data Engineer - Web Scraping on a 3-month contract to maintain web-scraping operations and build data pipelines that support insights for global luxury brands.

India Vietnam Pakistan Taiwan Morocco Contract Junior Data Engineer
Apache Airflow Azure dbt Git Python SQL Terraform
1 day, 2 hours ago
Inductive

Data Engineer

Inductive 51-200 Health Care Providers & Services

Join InductiveHealth as a Data Engineer to modernize and operate secure, scalable public health data pipelines that support disease surveillance and decision-making for public health agencies.

United States Full-time Mid Level Data Engineer
$130k-$145k
Agile AWS Azure CI/CD Docker Elasticsearch Encryption GitHub GitHub Actions Grafana Kafka Kubernetes OpenSearch Python Secrets Management SQL Terraform
1 day, 14 hours ago
Ampd Energy

Senior Engineer - Platform & Data Infrastructure

Ampd Energy 51-200 Electrical Equipment

Ampd Energy is seeking a hands-on senior software engineer to scale and stabilize Enernet, its cloud platform for monitoring and managing connected battery energy storage systems on construction sites.

India Full-time Senior Data Engineer
AWS ClickHouse Go Grafana GraphQL InfluxDB Kafka NestJS Node.js REST API SQL Terraform TimescaleDB TypeScript Vue.js
1 day, 14 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers