https://github.com/bitmakerla/estela
estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or...
https://github.com/bitmakerla/estela
Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. It would be great to have feedback to consider it when prioritizing some over others :-)
It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.
estela has been recently published as OSS under the MIT license:
https://github.com/bitmakerla/estela
More details about it can be found in the release blog post and the official documentation:
https://bitmaker.la/blog/2022/06/24/estela-oss-release.html
https://estela.bitmaker.la/docs/
estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.
All kinds of feedback and contributions are welcome!
Disclaimer: I'm part of the development team behind estela :-)
Scrapinghub Ltd. is looking for Python Engineers, Data Scientists, Erlang Engineers and more: https://scrapinghub.com/jobs
Quick summary of the open positions:
- Python Engineer (scraping): you’ll be in charge of designing, developing and testing Scrapy web crawlers.
- Data Scientist: you will work on NLP, record matching, clustering and classification of semi-structured data, leveraging massive datasets.
- Erlang Engineer: you will work developing and maintaining a high load distributed system.
We're a fully distributed company with more than 100 engineers and staff. Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you have the right skills (though everyone inside the company is encouraged to contribute).
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed, curious, and interested in working in open source environments. More on Open Source at Scrapinghub: http://scrapinghub.com/opensource/.
We've been growing a lot during the past two years and we're looking for great additions to our team, wherever you're located! Positions outside Ireland are on a contracting basis, equivalent to full-time (160 hours per month).
Application process consists of two rounds of interviews and a small programming trial (done at your own time/pace, not live).
You can apply here: https://scrapinghub.com/jobs
Got questions? Feel free to contact me at [email protected].
Scrapinghub Ltd. is looking for a Senior Designer, QA Engineer, Support Engineer and more: https://jobs.lever.co/scrapinghub
Quick summary of the open positions:
- Senior Designer: you’ll be in charge of everything related to UX and graphics for our entire suite of platform products.
- QA Engineer: you will estimate, plan, and coordinate testing activities to ensure product quality for all components of the Scrapinghub platform
- Support Engineer: you will provide customer support across all our platforms, working closely with development, sales and other technical teams to ensure our customers are receiving the best service possible.
We're a fully distributed company with 130 engineers and staff. Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you have the right skills (though everyone inside the company is encouraged to contribute).
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed, curious, and interested in working in open source environments. More on Open Source at Scrapinghub: http://scrapinghub.com/opensource/.
We've been growing a lot during the past two years and we're looking for great additions to our team, wherever you're located! Positions are full-time and include 20 days of paid vacations per year, your local country holidays, a hardware allowance, and an online services program for professional growth (Online courses, screencasts, books and more).
Scrapinghub Ltd. is looking for Software Engineers (Scrapy, Django), Data Scientists, Product Managers, Senior Designer, Head of Sales, Head of Finance and more:
We're a fully distributed company with 130 engineers and staff. Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you're good enough (though everyone inside the company is encouraged to contribute :)
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed, curious, and interested in working in open source environments.
More on Open Source at Scrapinghub: http://scrapinghub.com/opensource/
We've been growing a lot during the past two years and we're looking for great additions to our team, wherever you're located! Positions are full-time and include 20 days of paid vacations per year, your local country holidays, a hardware allowance, and an online services program intended for professional growth (Online courses, screencasts, books and more).
We're a fully distributed company with 130 engineers and staff. Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you're good enough (though everyone inside the company is encouraged to contribute :)
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed, curious, and interested in working in open source environments.
More on Open Source at Scrapinghub: http://scrapinghub.com/opensource/
We've been growing a lot during the past two years and we're looking for great additions to our team, wherever you're located! Positions are full-time and include 20 days of paid vacations per year, your local country holidays, a hardware allowance, and an online services program intended for professional growth (Online courses, screencasts, books and more).
Scrapinghub Ltd. is looking for Software Engineers (Scrapy, Django, devops) and Growth Hackers:
We're a fully distributed company (largest founded outside of the US!) with 112 engineers and staff. Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and a lots of other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you're good enough.
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed, curious, and interested in working in open source environments. More on Open Source at Scrapinghub: http://scrapinghub.com/opensource/
We've been growing a lot during the past two years and we're looking for great additions to our team, wherever you're located! Positions are full-time and include 20 days of paid vacations per year, your local country holidays, a hardware allowance, and an online services program intended for professional growth (Online courses, screencasts, books and more).
"I’ve researched your account and have confirmed your account is eligible for the AWS Free Usage Tier. Though the promotion does not actually start until November 1st accounts created from 10.21.2010 or later are eligible."