News for hackers. Stuff that matters.

sauain

would love to have your feedback on the python one too :)
I didn't try it, but I don't see a reason why not:

- RSS feed is transferred via HTTP. - BeatifulSoup can parse both HTML & XML.

(RSS uses XML format)

Crawlee is open source and free to use and we don't have any plans to monetize it in future. It will be always free to use.

We provide Apify platform to publish your scrapers as Actors for the developer community, and developers earn money through it. You can use Crawlee for Python as well :)

tldr; Crawlee is and always will be free to use and open sourced.

hey intev,

- Crawlee has out-of-the-box support for headless browser crawling (Playwright). You don't have to install any plugin or set up the middleware. - Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code. You don't have to care about what middleware, settings, and anything are or need to be changed, on the top that we also have templates which makes the learning curve much smaller. - Complete type hint coverage. Which is something Scrapy hasn't completed yet. - Based on standard Asyncio. Integrating Scrapy into a classic asyncio app requires integration of Twisted and asyncio. Which is possible, but not easy, and can result in troubles.

This is nice; we also use our [website content crawler(https://apify.com/apify/website-content-crawler) and [web scraper](https://apify.com/apify/web-scraper_ to scrape data from any website into JSON/CSV/etc. and then we use that data to feed LLMs.

interesting blog here: https://blog.apify.com/webscraping-ai-data-for-llms/

Re: on Monday April 15, 2024 @10:17PM (#40047755)
Apify's web scraping platform allows companies to mine data from websites and extract data for AI.
Re:Web Scraping Tutorial Using Crawlee on Wednesday April 10, 2024 @12:07AM (#39986860)
Build a HackerNews scraper using Crawlee
Re:Web scraping in 2024: breakthroughs and challenges ahead on Thursday February 22, 2024 @02:22AM (#39464194)
This blog explains the challenges and breakthroughs we have in 2024 in web scraping world.
Re:Daily.dev is like Reddit meets Stack Overflow on Thursday March 23, 2023 @11:41AM (#35276522)
Congratulations to the team!
Re:The Django Node.js alternative (NestJS) on Thursday March 23, 2023 @11:36AM (#35276440)
This blog explores why NestJS is a good alternative to Django.
Amplication, an open source devtool, launched a campaign, inviting first time contributors to contribute to their repository and get all the support they need for the PR and also some cool swag like a limited edition shirt and pack of stickers
Re:Amplication in TechCrunch's Top OS Startups of 2022 on Wednesday February 08, 2023 @10:46AM (#34709650)
Amplication has been ranked 23rd fastest growing startup of 2022 by TechCrunch.
Re:Amplication Launched Support for MongoDB on Friday February 03, 2023 @04:12AM (#34638994)
Amplication, a low-code tool, which can make production ready backend in minutes, now support MongoDB as well. It is third DB support after MySQL and PostgreSQL. This time through a new plugin for MongoDB.
I am using this product from last one year, this version take this product to next level.

The backend development will never be same

Re:Choose NestJS as your back end framework on Friday July 08, 2022 @03:43AM (#32024012)
The best open source backend framework there.