RemoteLeaf Logo
Log in Sign up

Senior Software Engineer - Crawler

1 month, 3 weeks ago
United States
Full-time
Senior
Data Engineer
Software Development
Apache Airflow AWS GCP HTML Java Kafka Kubernetes LLM Python Snowflake Terraform Trino Vertex AI
Apply Now
ZoomInfo

ZoomInfo

ZoomInfo provides a comprehensive B2B database that offers accurate and actionable contact and company intelligence, enabling sales and marketing teams to enhance their campaigns and improve prospecting efforts through access to direct dial phone numbe...

Professional Services
1K-5K
Founded 2000
View All Jobs 7

Description

  • Design and implement scalable, fault-tolerant web crawling and extraction pipeline components.
  • Write production-grade code in Java and Python.
  • Build and operate large-scale ETL and ELT pipelines for data extraction and transformation.
  • Work with cloud infrastructure on GCP and AWS, primarily on GKE.
  • Improve observability, reliability, and operational excellence across the systems you own.
  • Partner with product and data science teams to deliver technical solutions.
  • Contribute to code reviews, documentation, and knowledge sharing.
  • Stay current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

Requirements

  • 5+ years of professional software engineering experience building production systems.
  • Strong computer science fundamentals in algorithms, data structures, concurrency, and distributed systems.
  • Proficiency in Java and/or Python.
  • Track record of owning features end-to-end from design through deployment and operation.
  • Experience designing and operating large-scale ETL/ELT pipelines.
  • Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
  • Experience with orchestration tools such as Apache Airflow.
  • Experience with streaming or event-driven systems such as Apache Kafka.
  • Production experience on GCP (preferred) or AWS; multi-cloud exposure is a plus.
  • Hands-on experience with Kubernetes (GKE/EKS) and familiarity with infrastructure-as-code tools such as Terraform.
  • Nice-to-have experience with web crawling at scale, including Scrapy or similar frameworks.
  • Nice-to-have familiarity with proxy infrastructure, rotation strategies, or anti-bot evasion techniques.
  • Nice-to-have experience extracting structured and unstructured web data from diverse site architectures.
  • Nice-to-have knowledge of SERP extraction and AI/LLM-based extraction approaches.
  • Experience working in a B2B data company or data-as-a-product environment is a strong plus.

Benefits

  • US base salary of $140,000 to $220,000.
  • Additional compensation may include bonus, commission, and equity.
  • Comprehensive benefits package.
  • Holistic mind, body, and lifestyle programs focused on well-being.
  • Remote work designation indicated by LI-REMOTE.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

RevStar

Databricks Engineer

RevStar 51-250 Internet Software & Services

RevStar is seeking a remote, US-based Databricks Engineer to build and optimize cloud-agnostic Lakehouse data pipelines and MLOps frameworks for enterprise AI projects across AWS, Azure, and GCP.

United States Mexico Argentina Brazil Chile Colombia Venezuela Contract Mid Level Data Engineer
Agile Apache Airflow Apache Spark AWS Azure CI/CD Databricks Feature Engineering HIPAA Kafka MLflow MLOps Prefect Python SQL Terraform
5 hours, 1 minute ago
Aspire Software

Senior Software Developer

Aspire Software 251-1K Internet Software & Services

Vocantas, part of Fluent Software Group, is seeking a Senior Software Developer to build secure, reliable workforce communication and scheduling products through hands-on full-stack development with a backend focus.

Canada Senior Software Engineer
$110k-$125k
Angular C# JavaScript Microservices .NET REST API TypeScript Vue.js
5 hours, 1 minute ago
Aether Aerospace

Software Engineer (Mid-Senior)

Aether Aerospace 501-1000 aviation & aerospace

Aether Aerospace is hiring a full-time Mid-Senior Software Engineer to develop C/C++ real-time software and Angular interfaces for U.S. Department of Defense programs, working remotely with 50% travel to Hollywood, Maryland.

United States Full-time Lead Software Engineer
$150k-$189k
Agile Angular C C++
5 hours, 1 minute ago
Coforge

Intelligent Automation Engineer

Coforge 10K-50K IT Services

Coforge is hiring a remote Intelligent Automation Engineer in Costa Rica to design, develop, and optimize scalable automation solutions using UiPath, Power Automate, and AI technologies.

Costa Rica Mid Level AI Engineer Software Engineer
Agile CI/CD Machine Learning Microservices PowerShell Python Scrum
1 day, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers