Crawling with Pagination - Pagination not happening

Crawl Issues
#1

Hi - I am crawling with a recipe that uses pagination - but the crawl is not allowing the recipe to paginate.

What do I need to do?

Thanks!

#2

Hi There!

Thanks for posting. Below are a few steps that could help fix the issue.

  1. Confirm pagination is set up in the Crawl settings (Crawl Scrape Tab). You must indicate a max number of pages to paginate when setting up the Crawl.
  2. Make sure the pagination is working correctly at the recipe level by performing Next Page Automation in Page Scrape on a single URL.
  3. It could be a loading issue, some sites do not load the full page if the page is minimized or in the background. Keep the page open as your main window when testing the Crawl. You can also try adding a Page Action - Scrolling to the Recipe to assist with loading.