Crawl Scrape Not Working

Crawl Issues
#1

I’m using the free version of DataMiner to test it out before our company makes a decision on using it. I’m currently testing the crawl scrape feature, but I can’t get it to output the proper data. The URLs load and verify fine and the previewed data that it says it’s going to scrape before I actually do it is correct. But, when I actually click the scrape button, the results at the end are the same results from my list scrape recipe used for the URLs with one new column with the header of “Status” and every page that was scraped in the crawl has the status of “Skipped.” Other than that, it’s the exact same output from the list scrape recipe. Below is a sample of the log from the crawl scrape that I’m having issues with.

Page: https://www.linkedin.com/in/[username anonymized]/
8:42:57 AM Scrape Requested.
8:42:57 AM Scraping Data: LinkedIn Basic Info [list recipe to get URLs]
8:42:57 AM Scraped Page.
8:43:01 AM Stopped!
8:43:03 AM Not Started
8:43:03 AM Scrape Requested.
8:43:03 AM Scraping Data: Recipe 1 [crawl recipe]
8:43:03 AM Scraped Page.
8:43:04 AM Stopped!
8:44:27 AM Not Started
8:44:27 AM Scrape Requested.
8:44:27 AM Scraping Data: Recipe 1
8:44:27 AM Skipped Page.
8:44:27 AM Stopped!
8:44:27 AM Stopped!
8:44:28 AM Stopped!
8:44:28 AM Stopped!

#2

Hi Colin-MTM,

Thanks for posting. It sounds like there might be an issue with the URLs. The “skipped” status is typically reserved for invalid URLs. It’s possible the URLs are some how getting past our verification process, but still not working correctly.

I’d recommend sending an email to our support at support@dataminer.io. We can review the URLs and provide more info on the issue.

#3

Hey @Zach,

Thank you for the information. I have reached out to the support team about this issue two weeks ago (4/28) and haven’t heard back yet which is why I posted on this forum. I’m sending a follow-up email now.