RemoteLeaf Logo
Log in Sign up

Research Crawling Engineer

2 months, 3 weeks ago
United States, Poland, Romania
Full-time
Senior
Data Engineer
Software Development
C++ Go HTTP Java JavaScript Machine Learning NLP Playwright Puppeteer Python Rust
Apply Now
MLabs

MLabs

MLabs is a Haskell, Rust, Blockchain, and AI consultancy specializing in mission-critical software development, cross-team collaboration, and cutting-edge value delivery for fintech, blockchain, and information technology sectors.

Internet Software & Services
11-50
Founded 2018
View All Jobs 42

Description

  • Construct and maintain large-scale web crawlers across diverse domains.
  • Design high-throughput, fault-tolerant systems for collecting data from millions to billions of URLs per day.
  • Navigate anti-bot systems, rate limits, and dynamic JavaScript-heavy websites.
  • Develop pipelines for data cleaning, deduplication, filtering, and normalization.
  • Build and maintain datasets structured for research and machine learning model training.
  • Monitor and optimize crawl performance, coverage, and data quality through rapid iteration.
  • Collaborate with research teams to align data collection efforts with modeling requirements.
  • Optimize infrastructure for cost-efficiency, low latency, and reliability.

Requirements

  • Extensive programming experience in Go, Rust, Python, Java, or C++.
  • Proven experience building web crawlers or large-scale data pipelines.
  • Solid understanding of HTTP, networking protocols, and browser behavior.
  • Familiarity with distributed systems and parallel processing techniques.
  • Experience handling large datasets, ideally at the terabyte to petabyte scale.
  • Demonstrated ability to debug and maintain systems in unstable or adversarial environments.
  • Experience with NLP pipelines or dataset curation for machine learning (preferred).
  • Familiarity with LLM pre-training data or retrieval systems (preferred).
  • Practical experience with headless browsers such as Playwright, Puppeteer, or Chrome DevTools Protocol (preferred).
  • Knowledge of proxy systems, IP rotation, and large-scale request orchestration (preferred).
  • Background in data quality evaluation or benchmarking (preferred).
  • Experience running workloads on cloud or bare-metal infrastructure (preferred).
  • Must have a 6-hour overlap with EST.

Benefits

  • Competitive compensation of $80K-$175K, commensurate with experience.
  • Comprehensive benefits package.
  • Equity included in the compensation package.
  • Fully remote work with flexibility and autonomy.
  • Opportunity to work on a web-scale crawler and knowledge graph at the forefront of AI data accessibility.
  • Lean, low-ego team environment focused on high output and professional growth.
  • Equal opportunity and accessibility commitments throughout the hiring process.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Dan.com - a GoDaddy brand

Data Engineer IV (6249)

Dan.com - a GoDaddy brand Internet Software & Services

itD is hiring a Data Engineer IV to build and optimize AWS-based data infrastructure and pipelines that support AI infrastructure and inference initiatives in a remote role.

United States Contract Lead Data Engineer
AWS
15 hours, 21 minutes ago
Xebia

Senior Data Engineer (GCP)

Xebia 1K-5K Internet Software & Services

Xebia is hiring a data engineer to support a BigQuery migration by building and maintaining scalable data pipelines for production reporting, analytics, and business intelligence solutions.

Romania Bulgaria Poland Europe Senior Data Engineer
Apache Airflow CI/CD Databricks dbt Git Python Snowflake SQL
15 hours, 21 minutes ago
Coupang

Staff Data Engineer (Coupang Pay)

Coupang 1K-5K Internet Software & Services

쿠팡페이 핀테크 데이터 플랫폼 팀의 Staff Data Engineer로서 실시간 결제·주문 데이터를 기반으로 데이터 파이프라인과 분석 인프라를 구축·고도화하는 역할입니다.

South Korea Full-time Lead Data Engineer
Apache Spark AWS ClickHouse Flink Hive Java Presto Python Scala SQL Tableau
15 hours, 37 minutes ago
Cresteo

Data Migration Engineer (SQL Server / TSQL)

Cresteo 51-200 information technology & services

Cresteo is hiring a Data Migration Engineer to own end-to-end SQL Server data migrations from legacy schemas into reconciled, production-ready data for US-based clients and international teams.

Latin America Full-time Senior Database Administrator Data Engineer
JSON SQL Server
15 hours, 37 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers