Self-Repairing Data Scraping for Websites

Pre-processing and cleaning data from the web is challenging, as web pages are updated regularly. Whenever a monitored page changes its layout, existing processing pipelines break and must be adapted to the new design. We present an approach to scraping website data using Large Language Models (LLMs) to determine the location of the desired information and create a JavaScript path to the object in the Document Object Model of the HTML page. Our approach automatically detects when the path cannot be parsed anymore and repairs the path, continuously updating the scraping. Based on the example of the website kununu.com, our approach allows for consistent scraping and self-repair without overly impacting system performance. LLMs are only activated when an error in the pipeline is detected. In the future, we plan to expand this approach with multiple websites and data sources.