modernpacifist

Either mark-for-delete has the same impact as deleting in terms of shooting all the Cloud resources associated with the subscription, at which point the outage still happens but maybe the recovery is smoother or you've just delayed the inevitable by a week because no one will look at it unless there is actual impact.

Obviously carrier pigeons carrying messages encrypted with post-quantum ciphers where keys have been sent ahead of time using USPS because no one would be so rude as to read someone elses mail.

A very complicated pattern matching engine providing an answer based on it's inputs, heuristics and previous training.

In reality, we are likely about to get yet another data point on where the lines for the average person really lie in the dimensions of functionality, friction, network effect and privacy.

There are those that will stay on Discord because the benefits of the first three outweigh the degradation of privacy. Then there are those that will leave because the first three aren't important enough to outweigh the privacy loss. There will be all sorts of people in between.

HN has a rather amplified showing of folks who won't trust anything unless it's completely decentralized using E2EE clients verifiably compiled from source that they'be personally audited running on hardware made from self-mined rare metals. The reality is that there is a spectrum of folks out there all with different preferences and while some folks will leave (in this case) Discord, others will remain because thats where the folks they want to chat/game/voice with.

Likely the intended meaning here is that the practicality of space data centers goes against the physical realities of operating in space. The single most prevalent issue with operating anything in space is heat dissipation in that the only method of doing so is via radiation of heat, which is very slow. Meanwhile, the latest Nvidia reference architectures convert such ungodly amounts of power into heat (and occasionally higher share prices) that they call for water cooling and extensive heat-exchange plant.

Even if one got the the economics of launching/connecting GPU racks into space into negligable territory and made great use of the abundent solar energy, the heat generated (and in space retained) by this equipment would prevent running it at 100% utilization as it does in terrestrial facilities.

In addition to each rack worth of equipment you'd need to achieve enough heat sink surface area to match the heat dissipation capabilities of water-cooled systems via radiation alone.

A leased line though will only get you A<->B where sure, A and B can be anywhere but have to be concrete locations/hand off points when provisioned. It does ultimatley come down to the service that one orders from a commercial entity.

A hypothetical court order saying something like "kill internet access" would likely cause an IP transit service to stop working (implemented by said provider no longer announcing global IP routing tables to that service) but a leased line between two locations would likely remain untouched since that isn't an "internet" service. So they might not need to come knocking if they're reasonably confident that all such edge cases like leased lines end up at dead-ends because any internet-capable product they might be enabling access to is sufficiently disabled.

I do imagine though that if they get as far as "kill the internet" that obtaining a subsequent court order to go after some suspicious leased line would be trivial.

As a side note, I find that IP transit is typically the cheapest aspect of providing an internet service since a cross-connect at a well connected DC will cost well under $1/Mbps/month unmetered. Plus the cost is very well amortized when residential users are the target. This has tended to hold when one takes into account the co-lo costs as well since network gear doing relatively basic packet forwarding/internet table routing doesn't take up that much space or power.

The [ONT → OLT(+BNG)] → Internet] sections of the paths will continue to be owned by commercial entities that can still be the subject of court orders and/or government pressure.

Even if you were to roll your own cable in the ground to your own ONT/OLT/BNG at some point you will need to acquire IP transit or peering from other commercial entities.

A DoS that will disappear once you close the funnel. Tailscale are proxying the traffic so your public IP isn’t exposed. Your choice of port makes no difference.

The comparison breaks down since, in race car terms, the great acceleration isn’t enough to offset the negatives that make electric cars poor race cars. So in a sense it is pointless cosplay. Even the acceleration might be working against itself since the great acceleration comes at the cost of the battery pack expending more energy, contributing to heat build up.

This isn’t to say the heat problem couldn’t be managed, but one of the biggest issues with race cars generally is heat management so starting from a platform with a unique and significant heat problem isn’t ideal. Then the weight and overall longevity of the battery pack comes into play.

To tout the acceleration without discussing the drawbacks involved in delivering it or the practicalities of leveraging it suggests that it’s such a great feature that the drawbacks either don’t exist or don’t matter.

> Most race cars aren't electric though? That analogy makes no sense.

No, they aren't. I attend a significant amount of track events as a driver and I will see maybe 1 electic car every few events. Besides the lack of charging infrastrucutre at most race tracks, the one positive of instant torque/power is significantly outweighed by their overall mass and significant heat generation.

The latter tends to result in a Tesla S being unable to last more than 20 minutes at Laguna Seca or Sonoma before the battery pack overheats and reduces power output requiring the car to exit the track.

Removing the security mechanism in my comment is akin to opening the safe to enable the search or entering your PIN number on a phone to unlock it. I can't really see how otherwise removing a roadblock to enable law enforcement to perform their court approved mandate would lead to further charges for the act of helping them do so. Of course if you're referring to poison-pill mechanisms that upon removal destroy the data they wanted to search for then sure, more charges are coming.

> The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

If they have a search warrant then a judge has, from a legal perspective, determined that the request/search is reasonable. So while you have the right to secure against unreasonable cases I think it is a reasonable trade off that those security mechanisms/processes/etc should either be removed by yourself or you should expect them to be removed for you.

I don't see how banning payments would inherently create more opportunity for ransomware attacks. Assuming that the operators are already attacking as much as they can (why wouldn't they be - its more profit that way since its business after all) the only way to maintain profitability with lower per-attack yields would be to ask for more ransom per-attack which would likely drive the yields down even further.

Reminds me of https://youtu.be/9pOiOhxujsE?si=GG6X16c8efr0I3Ey&t=213

It seems reasonable to suggest that the number of profit-driven ransomware endeavors and the number of for-fun ransomware endeavors can both be non-zero and contain some overlap and some non-overlap. Therefore it seems that to make it unprofitable would at least eliminate the former reason which under all by the worst case scenario where those numbers are perfectly equal and overlapping would result in fewer ransomware endeavors.

To say we shouldn't do X because it doesn't perfectly eliminate/solve Y is akin to saying we should do nothing because by that standard, we'll never do anything about Y.

I would agree. My standard engagement with CS in Australia was a lot more personable and once it was recognized that a situation was off script they were far more willing to go into problem solving mode.

I'll +1 to this. Coming from Australia to the US I've found that (generally, especially in big corp entities/govt.) American customer service is always extremely courteous and eager but ultimately unhelpful or severely limited in what they can do. As soon as you are off script, good luck.

Not exactly - they apply to whichever dictates your capacity - be it ingress or egress. Overwhelmingly, capacity for hyperscalers is dictated by egress peaks. Since most non-residential network links are delivered symmetrically, the capacity for the other direction is already there as a by-product.

Also don't underestimate the benefit of simplification - why bill for 2 things separately when one of them is the primary driver of the cost, the comparative cost to supply the other is negligible and is probably more effort to bill for than it's worth.

I'm not dismissing the vendor lock-in aspect, but I don't think it is the only reason at play.

Having seen the other side of the fence (the hyperscaler side) I’m kind of bored with egress cost being continuously compared with your standard bare-metal hosting provider for a few reasons:

- Let’s first dispense with the idea that egress has to be provided without profit/margin in a capitalistic society. There will be profit in it sure and I don’t dismiss the idea that egress pricing is used to keep activities on-platform, not that I’ve been part of the decisions to set the price that way.

- Typically the more basic a network is, the easier it is to provision, manage and scale. Having a single DC with a couple of local transit providers and BGP routing brings with it a wildly lower cost base compared with a global network with piles of different POPs.

- Many providers, by charging only by usage, are effectively saying that the network is infinite in capacity and “just works”. You would be surprised how many engineers believe this to be true as well. To that end, as the complexity of the network grows you need to charge in a way that allows you to keep capacity ahead of demand for every path you manage. And then you need geographic redundancy for every such egress path and systems/people to manage said failover.

- In the case of GCP Premium tier, Google is hauling your traffic as far as it can on its private network towards its destination before exiting via a POP. Usage forecasting and pricing as a result needs to effectively assume any VM wants to send any amount of traffic to anywhere in the world. Even then the premium tier pricing separates kit China and Australia as special cases.

- In the hyperscaler case and even many of the larger VM/bare metal hosts you’ll find software defined networks which can often have a non-zero per-byte CPU processing cost involved. AFAIK this is essentially written off when traffic is in the same zone or region but escalates for various reasons (say rate limiting, egress path calculations, NAT, DoS prevention) when going inter-region or to the internet.

- Many of the hyperscalers do allow you spin up private interconnects but often charge for usage across it. This shifts away from being raw cross-connect cost to being more enterprise-y where the value of having dedicated, private capacity becomes the price. There is also the cost of managing said interconnect since it most certainly doesn’t get handled the same way as other egress paths (thus is more of an exception and exceptions cost money/time/effort).

Do all of these things add up to the “high” egress costs plus a decent margin for that evil profit? That is mostly up to the reader and what they value. Many others will say they don’t need all these features/management, but the reality is the hyperscalers don’t build for you, they build to cater to everyone that might become or is a customer. And it turns out to build a network capable of handling what “everyone” could potentially do with it is expensive.

> Here at HN we want a post mortem for a cloud failure in a matter of hours.

I'll go one further - I've yet to finish writing a postmortem on one incident before the next one happens. I also have my doubts that folks wanting a PM in O(hours) actually care about its contents/findings/remediations - its just a tick box in the process of day-to-day ops.

I don't know about others, but I can't help but smile when I read the detailed series of events in aviation postmortems. To be able to zero in on what turned out to be a single faulty part and then trace the entire provenance and environment that led to that defective part entering service speaks to the robustness of the industry. I say that sincerely since mistakes are going to happen and in my view robustness has less to do with the number of mistakes but how one responds to them.

Being an SRE at a FAANG and generally spending a lot of my life dealing with reliability, I am consistently in awe of the aviation industry. I can only hope (and do my small contribution) that the software/tech industry can one day be an equal in this regard.

And finally, the biggest of kudos to the Kyra Dempsey the writer. What an approachable article despite being (necessarily) heavy on the engineering content.

By not reading past the word "Autopilot" in the product description and ticking the Accept Terms box instead of having their personal lawyer review it, like the rest of us do.

It provides a consistent IP address (in the CGNAT range) that the end-device is always reachable at. On top of that you can use MagicDNS or regular DNS records to refer to it.

That IP is usable regardless of how that device and your device actually reach the internet. Further, no one device acts as a “server” and needs a stable public IP thanks to NAT traversal and the DERP fallback path. Keys are handled automatically with an option to not trust Tailscale infra in doing that (Tailscale lock) and I just need to auth devices with my Google Workspace/Gsuite SSO.

Mostly standard VPN use cases. They can access my Plex server, Mealie instance and in turn I can remote access their devices without something like TeamViewer when they need IT Support or their home automation stuff is acting up.

Would their lives fall apart without it? No. But it makes my life as the family SRE much easier.

- Entire site-to-site tunneling/routing. I didn't have to do anything for my parents I just dropped a subnet router at their place.

- Access my services/servers at home from anywhere in the world. Friendly mobile apps as well that allow the same.

- In cloud environments (for work and fun), don't even bother provisioning public IPs and having to deal with those firewall rules, just use Tailscale

- https://tailscale.com/blog/tailscale-auth-nginx/ describes how you can integrate nginx proxying with Tailscale auth to both leverage SSO and the authenticated endpoint

- I have a bootmod3 WiFi adapter plugged into my street/track car with a combo 5G/Linux unit in the car connected to my Tailscale that streams continuous telemetry about the car whenever its turned on. I could in theory re-flash the ECU via this.

- Using https://tailscale.com/kb/ondemand-access/ alongside node/subnet grouping to create a very neat first step towards auditing access to sensitive production services/environments.

- I use server-based dev environments to keep my portable laptop as clean as possible with no source code on it. VS Code remote + Coder server are fantastic over Tailscale.

+ others. Tailscale I think solves the problem of node-to-node-to-subnet connectivity at a convenient and flexible layer.

For a pure client/server VPN between two devices sure, but I think that’s where the equivalence between Tailscale and “some lines of config” end.

> What isn't easy about forwarding packets destined for port 80/443 of your public IP to the local service in question and being a part of the public Internet like things were from the start?

- Not every home internet service gets a publicly routable IPv4 address anymore (e.g. CGNAT)

- Not every home internet service gets a static IPv4 address so folks have to handle DynDNS

- Not everyone is comfortable exposing their home network IP address in DNS (Tailscale only shares the endpoint IP once the endpoint is auth'd onto the network)

- Not everyone is comfortable configuring heavy auth/fail2ban/app layer safeties (Tailscale makes the services uncontactable unless you are auth'd into the Tailscale network)

- Not everyone is comfortable/can be bothered configuring Wireguard in highly dynamic environments

> Using Tailscale is the opposite of self-hosting, you're bringing someone else's third party service in, and adding more complexity and another point of failure.

Self-hosting need not be a zealot position - rather one can pick and choose what makes sense for them. Tailscale allows you to build your own network where all the nodes are auth'd (and tailscale lock means you don't even need to trust their keys by default) and non-public internet routable but still globally reachable from known safe devices. This can actually make folks more comfortable with self-hosting their own stuff since it removes so many other considerations. There is also headscale if folks want to self-host the coordination server.

Some argue that a third party service adds complexity and a point of failure. I'll point out that configuring a self-hosted publicly exposed thing from scratch for the first time has a rabbit hole of unknown complexity to the uninitiated. A tool like Tailscale can remove some of those complexities allowing focus on others.

In seeing all of these complaints about the cloud providers getting too big, owning the internet etc, have folks forgotten how to run their own equipment or what renting a dedicated server is?

The internet did function prior to cloud computing existing and surprise, running a rack full of NUCs can give you a lot of capacity to run a glorified message board.

Seriously? Any half decent administrator knows that for every SSL certificate you install, you set a date some time in the future where you need to change it out for a new one.

To your comment of "one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous", this is working as intended. The certificate is to establish trust and identity along with encrypting the data in transit. The identity described by the certificate is only valid up until the expiry date after which it ceases to be valid for that purpose. You now have an encrypted connection to something that can't prove its identity, which is certainly a lower level of "secure" than what it was before the certificate expired.

Ignoring expiry dates would mean any keys that were compromised ever could be used for MITM attacks and no one would be the wiser.

You can sort of achieve the same outcome just my having all the various services/container specs inside a single .yml file that gets fed to 'kubectl apply'. docker-compose up does create networks and containers unique to the app your developing, but you could have the .yml setup a custom namespace and pile everything into that and get close. No option to concat all the log output though AFAIK.

To answer as best as possible given the vagueness:

Cookies: Highly compatible with most if not all browsers (and even headless tools). Lends themselves to having an expiry date and generally best when used with time sort of short-lived expiring session.

JWT: Everyone tends to use these in a stateless manner which means once issued, somewhat impossible to revoke without invalidating all JWTs or having a blacklist (and we're back at stateful). Plus some implementations had issues with downgrade attacks I think it was...

OAuth 1: Fine assuming you could keep clocks in sync and guarantee the secure storage of the keys/secrets. Good in that it didn't rely on HTTPS but if the secrets ever got compromised then the attacker would have free reign of that account until detected.

OAuth 2: Basically a complicated way to get short lived session tokens, potentially from a 3rd party API/auth source. Relies entirely on HTTPS to keep data secure in-transit and less of an issue if the session token gets compromised since its ideally short lived (because most want it to be stateless - those who keep a valid list of tokens sometimes skip the token expiry).