ziflex

Nope, just thought that ferrets are cool :)

Oh yes. The reason if this is that for now the language itself is DOM agnostic, it's just a port of an existing one. (https://docs.arangodb.com/3.4/AQL/) . So, the entire DOM thing is implemented by standard library which is pluggable. In the future, I might extend the language to make it less DOM agnostic by introducing new keywords for dealing with that. But for now you have to move document object around. Which is not that bad, because you may open as many page as you want in a single query.

Great! ^_^

This package is more like a runtime. There are plans to create a dedicated server, where you would be able to store your queries, schedule them and set up output streams like Spark or Flink. For now, it does not respect robots.txt. But it can be easily added.

Out of the box, there are not scaling mechanism yet, since the project is WIP. But, it's written in Go, which makes it pretty fast.

One idea of how you could scale it is to run cluster of instances of headless Chrome, put proxy/load balancer in front of it, and get Ferret a url to the cluster. It will treat it as a single instance of Chrome. The only problem, you would need to differentiate request from CDP (Chrome DevTools Protocol) client, and once a page is open, redirect all related requests to the same Chrome instance.

And you can open as many pages as you want in a single query (or as your memory allows you :) )

That's what you can do right now :)

https://github.com/MontFerret/ferret/blob/master/docs/exampl...

Document, returned form DOCUMENT() function, represents an open browser tab which allows you to do all interactions with the page.

PRs are welcome :) There is gonna be a separate project within the organization that would do all these things and even more. It's just beginning :)

Yes! This is one of the reasons why I wanted to be able making these changes without redeploying the whole thing!

It can! :)

Even more - it can interact with these pages! Here is an example of use of Google Search page: https://github.com/MontFerret/ferret/blob/master/docs/exampl...

Thank you very much for your valuable feedback and I'm glad that someone has finally got the idea :)

The idea is to create a high level abstraction that represents your web scraping logic. The project is still WIP. I will create a web server which will help you to store your queries, schedule them and set up output stream to other systems like Spark and Flink.

You are fine, I totally understand your scepticism. And you are right, there are definitely issues in communication.

First of all, I built it for myself. I needed a high level representation of scraping logic, which would run an isolated and safe environment. Second, I needed to be able easily scrape dynamic pages.

So, what I got is: - high level, declarative-is language, that hides all infrastructural details, which helps you to focus on the logic itself. that helps you to describe what you want without worring about underlying technology. Today, I'm using headless Chrome, tomorrow I will use something else, but the change should not affect your code. - full support of dynamic pages. You can get data from dynamically rendered page, emulate user's actions and etc. Heck, you can even write bots with it. - embeddable. now, I have only CLI, there are plans to write a web server where you can save your scripts, schedule them and set up output streams.

But the main idea is to provide high level declarative way of scraping the web. I'm not saying you can't do that with other tools. I'm just trying to come up with something more easy to work with.

Regarding examples, the project is still WIP, so as more complex features I get, more complex examples I get. Here is more or less complex, getting data from Google Search. It's not that difficult, but it showcases the core feature of work with dynamic pages.

https://github.com/MontFerret/ferret/blob/master/docs/exampl...

I could, if I knew Python pretty well :) But I've done it in the way I needed it to be done. I wanted to have an isolated and safe environment that would allow me to easily scrape the web without dealing with infrastructural code.

That's true. You can do it, of course. I'm not saying that this is the only way of doing it.

This is how it works under the hood. But everything is wired for you ;)

You definitely need to share. Web scraping is tedious. As more ideas we have, as more options we have to come up with a better solution for that.

That's true. The difference is how much efforts is needed to do that using API.

What it brings is just a higher abstraction of that API which lets you easily to get work done.

Yes, puppeteer and ferret use same technology under the hood - Chrome DevTools Protocol.

But, the purpose of the project is not to "be better than". No, the purpose is to let developers focus on the data itself, without dealing with technical details of underlying technologies. You can treat it as a higher abstraction on top of puppeteer / CDP.

It does not really matter whether it's written in JS or Go. main goal is to simplify data scraping and make it more declarative (even though, some people say it's not a declarative language :))

DOM is a representation of some data. Which means, you can extrapolate the data and then manipulate it. The language itself has nothing related to the DOM. All DOM operations are implemented via functions from standard library.

"Good artist copy, great artist steal" I'm trying to be a good artist trying to not invent a new brand language (I'm not that smart), so I just picked up (copied) an existing one that fits better for dealing with complex structures like trees. So it is AQL - ArangoDB Query Language. https://docs.arangodb.com/3.3/Manual/

If you have any suggestions how to improve the language - you are very welcome.

I'm sorry I do not fully understand what you mean. Imagine, that you need to grab some data from SoundCloud and, also imagine, they do not have a public API :) How would you do it without launching a browser?

The main purpose is to use scripts like SQL. Where you can write and modify your scripts for data without compilation. Plus, the project aims to simplify the process and hide technical complexity behind it. Moreover don't forget, that the system can work with dynamic pages which brings more complexity underneath.

And finally, you can use it as a library. It's totally embeddable.

Proxy support is not in place yet. I will add it in future releases. You are welcome to do a PR :)

Actually, it's not yet another language :) The syntax is taken from ArangoDB - AQL https://docs.arangodb.com/3.3/Manual/

One of the goals of the project is to hide technical details and complexities that follow modern web scraping, especially when you deal with dynamic web pages.

Well, I would say it's more declarative than imperative. There are few differences that make it less declarative - variables and ternary operator.

The main advantage of this over your approach with HtmlAgilityPack is that Ferret can handle dynamic web pages - those that are rendered with JS. And also, it can emulate user interactions. But anyway, thanks for your feedback :)