rosha

@tiburon thanks for the feature, i truly love it. i've added it to my blog with the JS snippet and it plays now on all the articles! https://jaywad.com/blogs/tips-for-a-productive-morning-routi...

I would wish to be able to customise the play button though.

it sounds great for me too. you must have headphone issues.

the idea of turning the web into a database colums/rows is hell of a great crazy idea, to be honest I was like wow, good luck with it.

I still believe that sometimes you might need some real anonymous way to crawl data to syndicate it, think of linkedin data, how are you going to insert the data when Linkedin blocks every single request you do? currently I use a paid service that has a crawler which I use for getting the data I need. https://proxycrawl.com/anonymous-crawler-asynchronous-scrapi...

Do you think Mixnode can help in getting to insert row data from difficult websites like Linkedin or Google?

I do not use tcp token, so I could share it for free, i use the api for javascript token which gives me dynamic content more.

I am currently using https://www.npmjs.com/package/proxycrawl web scraping system that offers anonymous scraping too. But they do not have a Go library yet. Does the system support anonymous scraping too like over a proxy? I'd love to use it if I can scrape URLs with proxycrawl API like with this example: https://api.proxycrawl.com/?token=DsFuFiigAZ2Wm6U1BPh7Zw&for...

Golang is a good choice too but in my experience its nothing compared to what you can do with Erlang Queue and Elixir. Regarding your question about proxycrawl, I do not know honestly, I tested the service for few days on some few millions per day and it was great too. I would say they are good for a very high volume, we are still using it, so that should be a good signal to try them.

I tried several different queue systems best version I got is using Erlang Queue, Elixir & Kafka on top for doing high concurrent crawler, the project was to develop a realtime Amazon product ASIN price monitoring system for our company as a challenger prototype. Our main problem was basically proxies, we stopped buying them as managing thousands of proxies is a huge effort that we did not want to take, also lack of data means our Hadoop clusters gets thirsty and machines stops learning properly. Currently we are using a third party https://proxycrawl.com on very high tiers > 10B with a great discount and we are happy to get that part solved. Other lessons learnt are like sometimes things fail and logs help a lot so you will need a highly available Logging and monitoring system.

Looks good thanks for sharing, you could also use an anonymous scraping API to scale your scraper up if you are crawling on high volumes for instance using this node package. https://www.npmjs.com/package/proxycrawl

Scraping on high volumes any serp is pain if your business relies on it and most services out there do not work on high volumes or they work and crazy expensive.

I have checked few solutions out there, I am using now proxycrawl. Developers of their api helped me get a very high volume of Serp data from different search engines like Yandex, google and yahoo and bing. I also use them for Javascript crawling as for our project we need lots of content which is rendered via javascript. I am amazed of how their API endpoint works. It is basically sending a URL to their API and you are good to start. Make sure to contact them for some sites as they do not allow you to crawl the world by default unless you prove your use case, they liked my product and that is how it got started. I've really having successful experience with it so I totally recommend, you basically communicate with developers who does lots of work to make it happen. As I am mainly in JS I asked for a Node JS package and they just built it open sourced. https://github.com/proxycrawl/proxycrawl-node

I recently had a similar idea which I ended up putting into real world, so I built a small search engine "UkookU" for used vehicles from scratch, it did not take me a long time to get the data and maintain it as I did not need to build any scrapers or care about sites blocking/abuse as I used a third party scrapping API called ProxyCrawl https://proxycrawl.com which allows me to make a proof of concept of the idea for free, so that shortened the time I needed to build the hadoop engine, etc.

I am thinking now recently to build a recruiting talent pool service, which is based on aggregated data from LinkedIn, Google, Bing, Yahoo, Facebook and many other sites and I am pitching something around it as I can get all the data with ProxyCrawl.

I am also thinking recently to do something about keyword ontologies, small markets around me using google/yandex data and offer it as a free helpful tool in the form of a mobile app. How do you normally get your data for your projects?

support says, the rate limit for LinkedIn can be increased on bigger packages, Do you recommend buying a bigger package? Thanks

thanks, I just talked to their support and got onboard very quickly, seems to work good for LinkedIn but only if support activates the token for you.