Senior Software Engineer (Agentic Search) - Crawler

1 hour, 36 minutes ago
Full-time
Senior
Software Development
Nebius

Nebius

Nebius enables B2B companies to build local hyperscaling cloud platforms with cost-effective GPUs, InfiniBand network, and 50% less compute cost. They offer managed Kubernetes and a launch-ready business model for innovative cloud solutions.

Internet Software & Services
51-250

Description

  • Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
  • Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
  • Develop crawl scheduling, prioritization, recrawl policies, and freshness strategies.
  • Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
  • Ensure reliable operation of crawling infrastructure under high-throughput conditions.
  • Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
  • Monitor resource usage, bandwidth consumption, and infrastructure cost.
  • Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements.
  • Enable safe experimentation with crawling strategies and content acquisition policies.

Requirements

  • 5+ years of experience building backend or distributed systems.
  • Strong Go or C++ expertise.
  • Experience with large-scale distributed systems, including 10k+ RPS, billions of URLs, or high-throughput pipelines.
  • Understanding of web protocols such as HTTP, DNS, and TLS, plus crawling, scraping, and content extraction.
  • Experience operating production systems and debugging failures in distributed environments.
  • Strong understanding of scalability, fault tolerance, and resource management.
  • Experience with web crawling is preferred.
  • Experience building streaming data pipelines and event-driven systems is preferred.
  • Experience with Kafka, Pulsar, NATS, RabbitMQ, or similar messaging platforms is preferred.
  • Experience designing distributed schedulers, queues, and asynchronous processing systems is preferred.
  • Experience with Spark, Flink, Beam, or MapReduce is preferred.
  • Experience in ad tech, social networks, search engines, or other large-scale content platforms is preferred.
  • Coding interviews are part of the hiring process.
  • Applicants must be authorized to work in the country where they apply and provide proof of employment eligibility.

Benefits

  • Competitive compensation.
  • Career growth and learning opportunities.
  • Flexibility and ownership.
  • Collaborative and innovative culture.
  • Opportunity to work on impactful AI projects.
  • International environment and talented teams.
  • Equal opportunity employer with an inclusive and diverse workplace.
  • Accommodations are available during the application process if needed.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Senior Payments Engineer — Acquiring & Payment Processing

Sezzle 251-1K Diversified Financial Services

Sezzle is hiring a Senior Payments Engineer to own and scale its backend acquiring and payment-processing systems that support transaction accuracy, resilience, and money movement across processors and gateways.

AWS CI/CD Datadog Elasticsearch Git GitLab Go Kafka Kubernetes LLM Microservices MySQL New Relic PostgreSQL Prometheus Python RabbitMQ React React Native REST API SQL TypeScript
1 hour, 6 minutes ago

Application Engineer, Product Catalog

Google 100K+ Internet Software & Services

GFiber is hiring an Application Engineer to help design and maintain the systems and rules engines behind its product catalog for internet services across the U.S.

GCP GitHub Java Kotlin Microservices Redis Salesforce Spring Boot
1 hour, 6 minutes ago

Staff Enterprise AI Engineer

6sense 1K-5K IT Services

6sense is hiring a Staff Enterprise AI Engineer, IT Operations to design and scale secure, production-grade AI systems and workflow automation that improve internal efficiency and decision-making across the business.

AWS
1 hour, 21 minutes ago

Senior Payments Engineer — Acquiring & Payment Processing

Sezzle 251-1K Diversified Financial Services

Sezzle is hiring a Senior Payments Engineer to own and scale its backend payment-processing systems, partnering across teams to keep money movement accurate, resilient, and auditable.

AWS CI/CD Datadog Elasticsearch Git GitLab Go Kafka Kubernetes LLM Microservices MySQL New Relic PostgreSQL Prometheus Python RabbitMQ React React Native REST API SQL TypeScript
1 hour, 36 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers