Kodiack

Why yes, 10-minute-old account with random letters for a name and one single comment, both sides are absolutely the same and people just shouldn’t feel encouraged to vote am I right?

You’re not even remotely subtle smh

I use Octopus in NZ and I’m a big fan of what they’re doing compared to a lot of our other electricity retailers. Savings sessions in particular are great with a beefy home battery because exporting counts in the comparison average. Those are easy money-making opportunities for us.

LLMs are incredibly powerful and useful tools, but offloading your writing to them oftentimes means you’re placing an undue burden on whoever you’re sending the generated output to.

To take it to the extreme, a “3x worker” is by definition relative, and someone malicious could achieve that by overloading a colleague with generated text that they themselves hadn’t bothered to read in full. You can stay at your baseline but be “3x” if you’re sufficiently able to sabotage your coworkers.

I’ve been on the receiving end of needing to review “documentation” that was generated almost entirely by LLM. It wasn’t anything that was done with malicious intent, but it’s still exhausting, and you need to exercise an even greater degree of caution because of the potential subtle mistakes or inaccuracies that a real human would never make.

They have their own ASN, which I’ve explicitly allowed requests from.

https://www.peeringdb.com/asn/7941

I run some small websites, including a tiny forum that’s been a goldmine for scrapers. I had to significantly tweak some firewall rules and configuration after scrapers behind residential proxies suddenly accounted for over 99% off requests.

However, I also relaxed rules for automated traffic that was well-behaved, and I went out of my way to ensure that the Wayback Machine was able to hit everything. I should kick a small donation their way. They provide an incredibly valuable service and I love the benefit that I get from them just for personal side projects.

As someone who hosts a few “tiny websites” that are “hiding behind Cloudflare”, it’s because of some incredibly misbehaved botnet traffic that’s otherwise persistently scraping via residential proxies.

This is a hobby for me. It’s for my enjoyment and it allows me to provide resources that others enjoy using. However, I’m not going to allow literally 99%+ of requests to be aggressive scrapers that won’t give up until you’ve got a heavy-handed solution in place.

I hate it too, but the alternative is even more consolidation, so unfortunately this is just the reality right now and you’ll have to get over it until when/if things improve.

Noise and air pollution are valid concerns with a lot of these datacentres being built as well. Some of these recent datacentres have been built close to residential areas, and the health harms are far more worrying than the property valuations IMO.

I use my home connection for VPN access remotely. I back up snapshots of data every day. I like to be able to download games and Linux ISOs practically on demand. I work from home and often enough faster speeds can avoid several minutes of additional waiting in a day.

This connection is shared as well. My partner relies heavily on cloud syncing. We both like to stream 4K HDR video. I like being able to get devices updated and ready to use with minimal time spent waiting for downloads.

I also live in NZ, where multi-gigabit fibre connections are often cheaper than what Americans have to pay for a fraction of the bandwidth. It’s not a notable financial burden or anything, and it’s not like we have data caps to worry about. It’s very much a situation where the use cases naturally find themselves once the option is there.

Also, 25/10 Mbps is painfully slow for a shared home connection in the modern day. There’s videos on YouTube that can push a higher bitrate than that. The absolute slowest plan that my ISP even offers is 100/20 Mbps for about $35 USD per month, while the most common/baseline plan for most households in NZ is 500/100 Mbps after the fibre carriers continued to increase speeds at the lowest tiers.

I love seeing trees in more places, but for parking lots in particular they do have some downsides compared to solar panels. They often require more space; they attract birds that that poo on vehicles; and there’s a higher risk of collateral damage during windstorms. Not to mention that solar panels directly produce electricity, of course.

We absolutely should see more trees in many cities, but they introduce their own challenges in parking lots, especially if they’re placed retroactively.

The extent of that heavily depends on where you are. Where I live in NZ, the grid export rates are very low while the import rates are very high.

Our peak import rate is 3x higher than our solar export rate. In other words, we’d need to sell 3 kWh hours of energy to offset the cost of using 1 kWh at peak.

We’re currently in the process of accepting a quote for home batteries. The rates here highly incentivise maximising self-use.

If you’re using Linux you’re almost certainly using your package manager to get any relevant updates.

Automatic updates are absolutely not peak stupidity. Most users’ devices would have nasty security vulnerabilities wide open for a much longer period of time without automatic updates.

I was thinking the same, and read it as “Xi”kipedia. Then sure enough, one of the articles that it immediately showed when it loaded was for “General Secretary of the Chinese Communist Party”, or Xi Jinping.

Coincidence?! Yeah, probably.

Hey. I run a small community forum and I've been dealing with this exact same kind of behaviour where well over 99% of requests are bad crawlers. There used to be plenty of "tells" for the faked browsers, HTTP/1.1 being a huge one. As you said, however, they're getting a bit smarter about that and it's becoming increasingly difficult to differentiate it from legitimate traffic.

It's been getting worse over the past year, with the past few weeks in particular seeing a massive change literally overnight. I had to aggressively tune my WAF rules to even remotely get things under control. With Cloudflare I'm aggressively issuing browser challenges to any browser that looks remotely suspicious, and the pass rate is currently below 0.5%. For my users' sake, a successful browser challenge is "valid" for over a month, but this still feels like another thing that'll eventually be bypassed.

I'd be keen to know if you've found any other effective ways of mitigating these most recent aggressive scraping requests. Even a simple "yes" or "no" would be appreciated; I think it's fair to be apprehensive about sharing some specific details publicly since even a lot of folks here on HN seem to think it's their right to scrape content with orders of magnitude higher throughput than all users combined.

I really don't know how this is sustainable long-term. It's eaten up quite a lot of my personal time and effort just for the sake of a hobby that I otherwise greatly enjoy.

If you have a "legitimate scraping pursuit", identify yourself appropriately that way. I'm happy to let most well-behaved scrapers access my content.

Hiding behind a residential proxy and using random user agents? Gross. Learn what consent is.

Thank you for speaking some sense. As a site operator that's been inundated with junk traffic over the past ~month where well in excess of 99% of it has to be blocked, the scrapers have brought this upon themselves.

I actually do let quite a few known, "good" scrapers scrape my stuff. They identify themselves, they make it clear what they do, and they respect conventions like robots.txt.

These residential proxies have been abused by scrapers that use random legit-looking user agents and absolutely hammer websites. What is it with these scrapers just not understanding consent? It's gross.

I run a really small forum and I've been absolutely inundated with a bunch of junk traffic. I had to tighten my Cloudflare WAF rules a whole bunch, and start issuing browser challenges way more aggressively.

Excluding known "good" crawlers, well over 99% of the traffic trying to hit the site has been attempting to maliciously scrape. Most of this traffic looks genuine, but has random genuine-looking user agents and comes from random residential proxies in various countries, usually the US.

For the traffic that does make it all the way to a browser challenge, the success rate is a measly 0.48%. Put another way, over 50% of traffic is already blocked by that point, and of the under 50% that makes it to a browser challenge, more than 99.5% fails that challenge.

It's been virtually no disruption to users either, since I configured successful challenges to be remembered for a long period of time. The legitimate traffic is a gentle trickle, while the WAF is holding back garbage traffic that's orders of magnitude above and beyond normal levels. The scale of it is truly insane.

For many ephemeral workloads, sure, but that comes at the expense of generally worse and less consistent CPU performance.

There are plenty of workloads where I’d love to double the memory and halve the cores compared to what the memory-optimised R instances offer, or where I could further double the cores and halve the RAM from what the compute-optimised C instances can do.

“Serverless” options can provide that to an extent, but it’s no free lunch, especially in situations where performance is a large consideration. I’ve found some use cases where it was better to avoid AWS entirely and opt for dedicated options elsewhere. AWS is remarkably uncompetitive in some use cases.

I like that https://discordstatus.com/ shows the API response times as well. There's times where Discord will seem to have issues, and those correlate very well with increased API response times usually.

Reddit Status used to show API response times way back in the day as well when I used to use the site, but they've really watered it down since then. Everything that goes there needs to be manually put in now AFAIK. Not to mention that one of the few sections is for "ads.reddit.com", classic.

I set up Immich last week and I absolutely love it. Docker is my "happy place" and I found the setup pretty straightforward, though it does have some rough edges that I anticipate will be sorted out as the project continues to mature.

I showed Immich to my partner and they loved it so much that we've ordered significantly more storage for the server to accommodate it. We're currently using both Google Photos and OneDrive, but with this we'll be ditching OneDrive and filling that niche with Immich (as well as expanded network storage in general).

The website and documentation is super clear about not using it as the only source of photos. This is why we'll keep using Google Photos, and why I'll also be backing up Immich and portions of the network storage to B2 via restic. I've used this snapshotting pattern for my general server data for years, and it's even saved me a couple of times. Backups are something you hope to never need to use, but boy are they satisfying when you do need to use them and have them set up properly!

This is similar to my setup and experience as well. I have a multi-purpose server running Samba with 10 Gbps SFP+. I have 2.5 GbE on my desktop and easily saturate that when transferring files.

I even use the network for stuff like old game ROMs and ISOs for emulation. The random access times are still magnitudes better than I’d be able to get with any HDD.

Meanwhile here in New Zealand we can get 10 Gbps FTTH already.

Sorry about your NBN!

I wonder what us kiwis did to deserve this.

“Sorry, this content is not available in your region.”

What a short and sad story.

I set up a self-hosted FreshRSS instance as well, and I feel the same benefits from it. I got back into RSS after Reddit pushed its API changes through, and it’s been so refreshing. I’m using social media a lot less now and those tingly FOMO feelings all but disappeared.

You can engine brake with a lot of automatic vehicles as well, since they’ll still have semi-automatic modes. I drive a car with a CVT and I’m able to “down-shift” for engine braking by using fake gears at specific ratios.