RemoteLeaf Logo
Log in Sign up

Senior Software Engineer - Crawler

1 month ago
United States
Full-time
Senior
Data Engineer
Software Development
Apache Airflow AWS GCP HTML Java Kafka Kubernetes LLM Python Snowflake Terraform Trino Vertex AI
Apply Now
ZoomInfo

ZoomInfo

ZoomInfo provides a comprehensive B2B database that offers accurate and actionable contact and company intelligence, enabling sales and marketing teams to enhance their campaigns and improve prospecting efforts through access to direct dial phone numbe...

Professional Services
1K-5K
Founded 2000
View All Jobs 8

Description

  • Design and implement scalable, fault-tolerant web crawling and extraction pipeline components.
  • Write production-grade code in Java and Python.
  • Build and operate large-scale ETL and ELT pipelines for data extraction and transformation.
  • Work with cloud infrastructure on GCP and AWS, primarily on GKE.
  • Improve observability, reliability, and operational excellence across the systems you own.
  • Partner with product and data science teams to deliver technical solutions.
  • Contribute to code reviews, documentation, and knowledge sharing.
  • Stay current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

Requirements

  • 5+ years of professional software engineering experience building production systems.
  • Strong computer science fundamentals in algorithms, data structures, concurrency, and distributed systems.
  • Proficiency in Java and/or Python.
  • Track record of owning features end-to-end from design through deployment and operation.
  • Experience designing and operating large-scale ETL/ELT pipelines.
  • Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
  • Experience with orchestration tools such as Apache Airflow.
  • Experience with streaming or event-driven systems such as Apache Kafka.
  • Production experience on GCP (preferred) or AWS; multi-cloud exposure is a plus.
  • Hands-on experience with Kubernetes (GKE/EKS) and familiarity with infrastructure-as-code tools such as Terraform.
  • Nice-to-have experience with web crawling at scale, including Scrapy or similar frameworks.
  • Nice-to-have familiarity with proxy infrastructure, rotation strategies, or anti-bot evasion techniques.
  • Nice-to-have experience extracting structured and unstructured web data from diverse site architectures.
  • Nice-to-have knowledge of SERP extraction and AI/LLM-based extraction approaches.
  • Experience working in a B2B data company or data-as-a-product environment is a strong plus.

Benefits

  • US base salary of $140,000 to $220,000.
  • Additional compensation may include bonus, commission, and equity.
  • Comprehensive benefits package.
  • Holistic mind, body, and lifestyle programs focused on well-being.
  • Remote work designation indicated by LI-REMOTE.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Affirm

Senior Software Engineer, Backend (Identity International)

Affirm 1K-5K Diversified Financial Services

Affirm is hiring a Senior Software Engineer for its remote UK Identity team to build and operate reliable identity, KYC, and user lifecycle systems supporting international expansion and compliant customer checkout experiences.

United Kingdom Full-time Senior Software Engineer
$152k-$213k
AWS Kotlin Kubernetes MySQL Python
28 minutes ago
Northbeam

Staff Software Engineer, Python

Northbeam 11-50 Internet Software & Services

Northbeam is hiring a Staff Backend-leaning Software Engineer to architect scalable data, experimentation, and AI systems for its remote-friendly marketing intelligence platform.

Canada Full-time Lead Software Engineer
$190k-$220k
Machine Learning Microservices Python React TypeScript
28 minutes ago
Northbeam

Software Engineer, Data Systems (Python)

Northbeam 11-50 Internet Software & Services

Northbeam is hiring a data-focused software engineer to build reliable integrations, pipelines, and data products powering its marketing intelligence platform for eCommerce brands.

United States Full-time Mid Level Software Engineer
$140k-$155k
Apache Airflow Dagster Docker GCP GraphQL Kubernetes Prefect Python REST API Secrets Management Shopify Snowflake SQL
43 minutes ago
Northbeam

Staff Software Engineer, Python

Northbeam 11-50 Internet Software & Services

Northbeam is seeking a Staff Software Engineer with a backend focus to architect scalable data, experimentation, and AI systems for its remote-friendly marketing intelligence platform.

United States Full-time Lead Software Engineer
$190k-$220k
Machine Learning Microservices Python React TypeScript
43 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers