I am trying to scrap this Website, but every time new data loads, Data Miner would also pull duplicate rows. The site has no buttons but loads upon scrolling down. The Data Miner would continue to scrap duplicate unless we scrolled it down to load more items.
URL: Here
Unfortunately, this is a Password protected site. but the problem is discussed above.
Hi Hamid403,
Thanks for the post! It sounds like the site could be dynamically loading the data. Meaning that new data loads and the previous data is removed as you scroll down. This is done to help with site performance and we, unfortunately, don’t have a solution or feature for this issue quite yet.
At the moment Data Miner is only able to scrape once per page. And what has to happen to overcome this issue is Data Miner needs to scroll, scrape, scroll, scrape. What you’ll have to do is Run the recipe once, then manually scroll down. Then with Data Miner open, click the scrape button using the “Scrape and Append” option and then scroll again continuing this process. The button will accumulate the data so when you download the file it will have everything you scraped in a single CSV.
Please let me know if you have any questions