R.K Digital is building an AI-powered data platform for the U.S. industry. Our mission is to collect, structure, and process data from hundreds of public records using intelligent automation.

We are looking for a Web Scraping Engineer to join our Data Engineering team Your responsibility is to build reliable, scalable data acquisition systems that power our AI pipeline.

You will work closely with our Lead Source Intelligence team, who document each data source and define the business requirements. Your role is to transform those requirements into production-ready scraping systems.

Responsibilities

• Develop and maintain web scrapers for public record.
• Build automated data acquisition workflows from structured and semi-structured sources.
• Analyze website structures and implement efficient extraction strategies.
• Handle pagination, authentication, JavaScript-rendered websites, CAPTCHAs, rate limiting, and anti-bot protections.
• Build browser automation using modern scraping frameworks.
• Integrate APIs whenever available instead of scraping.
• Design scalable scraping architectures capable of processing large volumes of data.
• Create reusable scraping modules that can be deployed across multiple sources.
• Monitor scraper health, failures, and website changes.
• Optimize scraping speed, reliability, and resource consumption.
• Ensure collected data matches RK Digital's business requirements and quality standards.
• Work closely with AI engineers to integrate scrapers into the company's automated data pipeline.
• Participate in designing the next generation of AI-driven data acquisition systems.

Technical Requirements

• Strong Python programming skills.
• Experience with Playwright.
• Experience with Selenium.
• Experience with BeautifulSoup and Requests.
• Knowledge of HTML, CSS selectors, and XPath.
• Experience working with REST APIs.
• Understanding of asynchronous programming.
• Experience with Git.
• Understanding of JSON, CSV, XML, and structured data.
• Experience deploying scraping applications.

Preferred Qualifications

• Experience bypassing anti-bot protections responsibly.
• Proxy management and request optimization.
• OCR and document extraction experience.
• Experience with PostgreSQL or other databases.
• Familiarity with Docker.
• Experience with cloud deployments.
• Experience integrating scraping pipelines with automation platforms such as n8n.
• Experience building large-scale data collection systems.