Preprint Open access
ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then clean it, so corpus quality is capped by the coarseness and accuracy of the rules. In this work, we propose ReScraper, a unified language mod …