RemoteLeaf Logo
Log in Sign up

Senior Software Engineer - Crawler

5 hours, 49 minutes ago
United States
Full-time
Senior
Data Engineer
Software Development
Apache Airflow AWS GCP HTML Java Kafka Kubernetes LLM Python Snowflake Terraform Trino Vertex AI
Apply Now
ZoomInfo

ZoomInfo

ZoomInfo provides a comprehensive B2B database that offers accurate and actionable contact and company intelligence, enabling sales and marketing teams to enhance their campaigns and improve prospecting efforts through access to direct dial phone numbe...

Professional Services
1K-5K
Founded 2000
View All Jobs 10

Description

  • Design and implement scalable, fault-tolerant web crawling and extraction pipeline components.
  • Write production-grade code in Java and Python.
  • Build and operate large-scale ETL and ELT pipelines for data extraction and transformation.
  • Work with cloud infrastructure on GCP and AWS, primarily on GKE.
  • Improve observability, reliability, and operational excellence across the systems you own.
  • Partner with product and data science teams to deliver technical solutions.
  • Contribute to code reviews, documentation, and knowledge sharing.
  • Stay current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

Requirements

  • 5+ years of professional software engineering experience building production systems.
  • Strong computer science fundamentals in algorithms, data structures, concurrency, and distributed systems.
  • Proficiency in Java and/or Python.
  • Track record of owning features end-to-end from design through deployment and operation.
  • Experience designing and operating large-scale ETL/ELT pipelines.
  • Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
  • Experience with orchestration tools such as Apache Airflow.
  • Experience with streaming or event-driven systems such as Apache Kafka.
  • Production experience on GCP (preferred) or AWS; multi-cloud exposure is a plus.
  • Hands-on experience with Kubernetes (GKE/EKS) and familiarity with infrastructure-as-code tools such as Terraform.
  • Nice-to-have experience with web crawling at scale, including Scrapy or similar frameworks.
  • Nice-to-have familiarity with proxy infrastructure, rotation strategies, or anti-bot evasion techniques.
  • Nice-to-have experience extracting structured and unstructured web data from diverse site architectures.
  • Nice-to-have knowledge of SERP extraction and AI/LLM-based extraction approaches.
  • Experience working in a B2B data company or data-as-a-product environment is a strong plus.

Benefits

  • US base salary of $140,000 to $220,000.
  • Additional compensation may include bonus, commission, and equity.
  • Comprehensive benefits package.
  • Holistic mind, body, and lifestyle programs focused on well-being.
  • Remote work designation indicated by LI-REMOTE.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Innodata

Data Platform Engineer (GCP)

Innodata 1K-5K IT Services

Innodata is hiring a hands-on GCP/Data Platform Engineer to support and improve large-scale production data platforms on Google Cloud, with a focus on reliability, troubleshooting, and operational enhancement.

Canada Mid Level Data Engineer
$64k-$120k
Apache Airflow CI/CD GCP Generative AI LLM Python SQL Terraform Vertex AI
5 hours, 34 minutes ago
Kunai

Senior Data Engineer

Kunai 51-250 Professional Services

Kunai is seeking a Senior Data Engineer to help modernize and scale financial services platforms by building data pipelines and systems for high-volume transactional and operational data.

United States Full-time Senior Data Engineer
Apache Spark AWS Cassandra CI/CD CloudFormation Docker Kubernetes Python Serverless Terraform
5 hours, 49 minutes ago
Fundraise Up

Senior Tools Engineer

Fundraise Up 51-250 Capital Markets

Fundraise Up is hiring a part-time Tools Engineer in Cyprus (remote) to build internal automations, bots, and AI assistants that streamline company operations across a global product organization.

Cyprus Part-time Senior AI Engineer Software Engineer
$0k-$0k
Bull ClickHouse Cloudflare Elasticsearch JavaScript Kafka Koa MongoDB NestJS Node.js PostgreSQL React Redis TypeScript Vercel Vue.js
5 hours, 49 minutes ago
Innodata

Application Reliability Engineer

Innodata 1K-5K IT Services

Innodata is hiring a hands-on Application Support Engineer to maintain and enhance business-critical enterprise applications running on Google App Engine and microservices across production and release environments.

United States Full-time Mid Level Site Reliability Engineer (SRE) Software Engineer
$60k-$100k
API Gateway Apigee CI/CD Docker Firestore GCP Generative AI GitHub Actions GitLab CI Go gRPC Java JavaScript Jenkins Kubernetes LLM Load Balancing Looker Microservices Node.js Python REST API SQL Tableau Terraform Vertex AI
6 hours, 4 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers