zlagen
zlagen on Hacker News
the most annoying thing about new laptops is how difficult is to find replacement batteries that can be trusted and work well. The battery situation is a downgrade compared to the previous pluggable ones.
data as a business, now everyone needs good data to plug it into their ai.
Interesting how this site talks about the bad experience using google and while I scroll through it I'm shown more than 10 advertisements.
in the perfect world we would have a very high trust internet in which everyone follows the rules, checks robots.txt, rate limits, etc. But we don't live in such a world. Getting angry at these ai bots is useless. People should start considering what and how they host their data. If you're worried about bandwidth costs you have many alternatives to host your data for free or at a very little cost. i.e: github/gitlab/cloudflare.
If you're worried about your data getting scraped and used then maybe you can consider putting it behind a login or do some proof of work/soft captcha. Yeah, this isn't perfect but it will keep most dumb bots away.
Some people are hosting their sites like we're still in 1995 and times have changed.
that's a good point, in search we have google as a monopoly and since a big percentage of sites only want to be crawled by them it reinforces the monopoly. So a lot of people complain about bots not following robots.txt but if you follow them to the letter it's impossible to make anything useful. Also AFAIK robots.txt doesn't have any legal standing
Wait until legitimate users start getting their fake responses. Who is willing to trust this company to serve random data?
shouldn't QUIC be supported at the operating system level similar to TCP? that way it could get adopted much easily in different languages
in Argentina some people call facebook marketplace the "deep web", not sure if it has that reputation in other countries. Lots of stolen things being sold and also scams
this kind of surveillance is childish and a bit creepy too
They still have 62% pass rate in WPT so my guess is that there's still a lot to do to make it usable as a browser.
Does anyone know about people doing similar reverse engineering but for javascript code in websites?
I'm using chrome on linux and noticed that this year cloudflare is very agressive in showing the "Verify you are a human" box. Now a lot of sites that use cloudflare show it and once you solve the challenge it shows it again after 30 minutes!
What are you protecting cloudflare?
Also they show those captchas when going to robots.txt... unbelievable.
After using the AI chatbots for some time, I think that they are not so useful for non programmers other than for doing small tools, that may be difficult to modify and polish by a non programmer.
But they still fail and have subtle errors too often so they are more useful for programmers which already know what the AI is doing and can spot mistakes.
it may be that google is checking the referer header.
fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.
thanks, it doesn't seem like it's the direction it's going at the moment. If you look at the robots.txt of many websites, they are actually banning AI bots from crawling the site. To me it seems more likely that each site will have its own AI agent to perform operations but controlled by the site.
what do you think would be the use cases for this project? being lightweight is awesome but usually you need a real browser for most use cases. Testing sites and scraping for example. It may work for some scraping use cases but I think that if the site uses any kind of bot blocking this is not going to cut it.
I wanted to try a diffent approach which is to use chromium's network stack directly instead of patching curl to impersonate it. In this case you're using the real thing so it's a bit easier to maintain when there are changes in the fingerprint.
They use different SSL libraries/configuration. Chrome uses BoringSSL and other libraries may use OpenSSL or some other library.
Besides that the SSL library may be configured with different cipher suites and extensions. The solution these impersonators provide is to use the same SSL library and configuration as a real browser.
It replicates the browser at the HTTP/SSL level, not TCP. From what I know this is good enough to bypass cloudflare's bot detection.
Posted by
zlagen on Wednesday April 03, 2024 @10:57AM
from the show-hn dept.
https://github.com/lagenar/python-cronet
Hi HN!After being frustrated to need to run headless chrome/selenium in my small scale scrapers because of Cloudflare and Akamai performing TLS fingerprinting. I decided to create a python library to use chromium's network library called cronet. If you don't need to run javascript and are only affected by TLS fingerprinting, using this library should be more efficient than pupeteer/headeless...
It's mostly a piece of content marketing. The service he provides is mostly about respecting user's privacy. Reminds me a bit of Duckduckgo's strategy.
The example demo looks very slick! it looks like Zapier but a bit more low level.
Also for anyone interested in running functions but not interested in handling the complexity of aws lambda I'd check Cloudflare workers
https://developers.cloudflare.com/workers/
Users know their information is public and they have the option to make it private on Linkedin. If Linkedin is worried about the privacy of their users they should let them know about the risks of having a public profile.
any examples of such cities?
SEEKING WORK - Remote, Buenos Aires, AR
Backend developer
Skills:
+ Freelance developer specialized in web data extraction and business intelligence apps
+ Backend development with Django and Flask
+ 5 years of experience working with Postgres as a backend db
I'm looking for part time work mostly to complement my current contract work.
LinkedIn: https://www.linkedin.com/in/lucas-moauro-89b64039/
Email: [email protected]
SEEKING WORK - Remote, Buenos Aires, AR
Backend developer
Skills:
+ Freelance developer specialized in web data extraction and business intelligence apps
+ Backend development with Django and Flask
+ 5 years of experience working with Postgres as a backend db
I'm looking for part time work mostly to complement my current contract work.
LinkedIn: https://www.linkedin.com/in/lucas-moauro-89b64039/
Email: [email protected]