How do I upload a list of URLs to scrape from?

#1

I’m trying to extract data from the URLs that I have in a CSV file which I’ve uploaded into DataMiner.

Here’s what the data looks like.

As you can see, the URLs are in column 2.

According to this, to scrape URLs, you need to go to Crawl Scrape.

I realize that this is a premium feature, and I’m a non-paying account owner at this point.

However, I’d like to scrape 2 URLs just to test it out.

Anyways, as you can see, I’m doing the Crawl Scrape, and since I’ve imported a CSV file, I’m selecting it, and I specified that the URLs are in Column 2.

I see this page that says “Select a recipe for scraping the URLs in your crawl.” However, I’m unable to create a new recipe.

When I click on the blue link at the bottom-right that says “+ Create a new recipe”, I’m prompted to another page where I try and create a “+New Recipe”. However, this doesn’t seem to connect from my uploaded CSV file, or this isn’t reading my CSV file.

As you can see here, I’m now simply trying to test to URLs instead of using my uploaded CSV file. This also is preventing me from going to the next page. Both of my URLs are valid, but I can’t advance from here.

Please help. Thanks in advance.

URL's Crawl Scrape not validating
#2

Hi ScrapingManiac,

Thanks for the post! Similar to justvervaart’s post, Data Miner sometimes has trouble connecting to the first url when setting up a Crawl. You can either reselect the column and click “Check it” again or close Data Miner and restart the Crawl setup. We are still looking into why this is happening and I apologize for the issue.

As for the recipe issue - Before setting up the Crawl, you will first need to build a new recipe that is designed to scrape the URLs. To do this, manually visit one of the URLs and then processed with the recipe writing process. You can learn how to write recipes from the following tutorials.

Please let us know if you continue to have issues or any questions.

#3

Hi ScrapingManiac,

I hope you very well

ScrapingManiac email address is not showing. I’ve also taken solo plans. There is no cure??

#4

Hi akashumak99,

If you are not getting results even though an email is visible, it may be a different issue. You can contact our support and we are happy to help further: support@dataminer.io

#5

I see a little email. But not to much 50 email showing out of 1000 rows