Hello, I have a recipe that did work on crawls of different URLs by the same company of the same type of web pages but the crawl now results in no data but I can still scrape the pages one at a time. Any idea why this is? Thank you.
Hi ger630,
Thanks for the post! If the recipe still works when you manually visit the page, the issue could be within the URL list that is being used in the Crawl. I recommend reviewing the URLs and try visiting them to make sure they are valid.
In addition, the Crawl settings could also be affecting the results. Sometimes sites need more loading time for the Crawl to be successful. Try increasing the wait time between scrapes.
Please let us know if you continue to have issues or any questions.
URLs were updated, did a new crawl from scratch, still not working, it’s weird. I can scrape each page on its own but it fails on the crawl.
Hi ger630,
Thanks for the update. That is strange. I’m not sure what the issue is. I’d recommend checking out our free Office Hours. We are offering them Monday through Thursday, Noon to 2, Pacific time. It’s an open learning session where you can ask questions and share screens to get detailed solutions. We will be able to better help you here since we can see the issue on your screen. You can join from here: How to start Scraping Data | Data Miner