RemoteLeaf Logo
Log in Sign up

Senior Software Engineer (Agentic Search) - Crawler

4 weeks ago
United Kingdom, Netherlands
Full-time
Senior
AI Engineer
Software Development
Apache Spark Beam C++ DNS Flink Go HTTP Kafka Machine Learning MapReduce NATS RabbitMQ TLS
Apply Now
Nebius

Nebius

Nebius enables B2B companies to build local hyperscaling cloud platforms with cost-effective GPUs, InfiniBand network, and 50% less compute cost. They offer managed Kubernetes and a launch-ready business model for innovative cloud solutions.

Internet Software & Services
51-250
View All Jobs 153

Description

  • Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
  • Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
  • Develop crawl scheduling, prioritization, recrawl policies, and freshness strategies.
  • Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
  • Ensure reliable operation of crawling infrastructure under high-throughput conditions.
  • Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
  • Monitor resource usage, bandwidth consumption, and infrastructure cost.
  • Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements.
  • Enable safe experimentation with crawling strategies and content acquisition policies.

Requirements

  • 5+ years of experience building backend or distributed systems.
  • Strong Go or C++ expertise.
  • Experience with large-scale distributed systems, including 10k+ RPS, billions of URLs, or high-throughput pipelines.
  • Understanding of web protocols such as HTTP, DNS, and TLS, plus crawling, scraping, and content extraction.
  • Experience operating production systems and debugging failures in distributed environments.
  • Strong understanding of scalability, fault tolerance, and resource management.
  • Experience with web crawling is preferred.
  • Experience building streaming data pipelines and event-driven systems is preferred.
  • Experience with Kafka, Pulsar, NATS, RabbitMQ, or similar messaging platforms is preferred.
  • Experience designing distributed schedulers, queues, and asynchronous processing systems is preferred.
  • Experience with Spark, Flink, Beam, or MapReduce is preferred.
  • Experience in ad tech, social networks, search engines, or other large-scale content platforms is preferred.
  • Coding interviews are part of the hiring process.
  • Applicants must be authorized to work in the country where they apply and provide proof of employment eligibility.

Benefits

  • Competitive compensation.
  • Career growth and learning opportunities.
  • Flexibility and ownership.
  • Collaborative and innovative culture.
  • Opportunity to work on impactful AI projects.
  • International environment and talented teams.
  • Equal opportunity employer with an inclusive and diverse workplace.
  • Accommodations are available during the application process if needed.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Epic Games

Senior Backend Engineer, Developer Platform (Horde)

Epic Games 5K-10K Internet Software & Services

Epic Games is hiring a senior programmer for its Unreal Engine developer platform team to work on Horde, the internal platform that supports CI/CD, remote execution, build health, storage, and developer workflows.

United States Canada Full-time Senior Backend Engineer
$173k-$279k
AWS Azure C# CI/CD gRPC MongoDB .NET Redis
16 hours, 1 minute ago
Xebia

Principal Fullstack Architect (AI & Agentic Systems)

Xebia 1K-5K Internet Software & Services

Xebia is hiring a Senior AI Software Engineer for an insurance technology client to help scale an AI-native underwriting platform from MVP into a production-grade system.

Moldova Poland Romania Slovakia Bulgaria Sweden Czech Republic Hungary Lead AI Engineer Full-stack Engineer
AWS Azure Databricks GCP Generative AI LLM Microservices Python React Snowflake TypeScript
16 hours, 1 minute ago
Coupang

Senior Staff Back-end Engineer (eCommerce Engineering)

Coupang 1K-5K Internet Software & Services

Coupang is hiring a Senior Staff Backend Engineer to lead the architecture and technical strategy for its core e-commerce platform services across global markets.

South Korea Full-time Lead Backend Engineer
Apache Airflow Apache Spark AWS Cassandra Grafana Java Kafka Kotlin Kubernetes Microservices OpenTelemetry Prometheus Spring
16 hours, 16 minutes ago
Skyline Eco-Adventures

Autonomy Droid Perception SWE - Onboard Systems

Skyline Eco-Adventures 11-50 Hotels, Restaurants & Leisure

Zipline is hiring a Senior or Staff Perception Engineer to build and deploy real-time 3D perception systems for its autonomous drone delivery network, supporting onboard autonomy and planning for safety-critical backyard deliveries.

United States Full-time Senior AI Engineer Robotics Engineer
$200k-$280k
CNN Computer Vision Deep Learning Machine Learning
16 hours, 31 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers