News for hackers. Stuff that matters.
Re:We raised a bunch of money by geraldwhen on Wednesday June 28, 2023 @10:28AM (#36507013)
“ The result of this is an Internet where all of the world's CRUD apps are hosted in Loudoun County, VA (motto: "where tradition meets innovation"), at Amazon's us-east-1 in Ashburn, a city with so many Rails apps that one of them was elected to the county Board of Supervisors.”

So true it hurts

Re:We raised a bunch of money by sneak on Wednesday June 28, 2023 @10:40AM (#36507147)
How much of the raise is directly attributable to the fact that their early staff has at least two excellent writers on it?

Their brand marketing is impeccable as a result. Every post on their blog comes through with this very clear, slightly snarky voice.

If you want to be a famous hacker, you must also be a slightly (industry) famous author.

Re:We raised a bunch of money by madeofpalk on Wednesday June 28, 2023 @10:59AM (#36507426)
It's great, and it shows in their docs also.

Their tone of voice reminds me of early Slack before that became rote and boring.

Re:We raised a bunch of money by TheFlyingFish on Wednesday June 28, 2023 @11:01PM (#36515833)
Oh man, early Slack was great. Back in ~2015 or so I submitted a ticket about something trivial (why I could no longer hide the menubar or something, I forget), but just for fun I did it all in stilted archaic language like you'd see in the 1600s or something. "Behold, in days of yore when the Users of the Land sought refuge from the tiresome typing that had so bedeviled their lives hitherto..." etc. Anyway, their CS team responded in kind. Hands-down the most fun I've ever had with a support ticket.
Re:We raised a bunch of money by hgsgm on Wednesday June 28, 2023 @11:11AM (#36507624)
Does VC like snarky?

Snarky is passable in open source, a genius whose abrasive manner is the cost of the amazing product.

In a corpo it's sleazy

Re:We raised a bunch of money by jeremy_k on Wednesday June 28, 2023 @11:23AM (#36507831)
I've thought a lot about this too. Their blog posts are consistently interesting and well written. I wonder how much of their success is related to the product marketing as well?
Re:We raised a bunch of money by swyx on Wednesday June 28, 2023 @06:13PM (#36513380)
theyre extremely correlated. writing is thinking.
Re:We raised a bunch of money by ignoramous on Wednesday June 28, 2023 @07:23PM (#36514095)
You'd hope they also write good code to keep up with the marketing ;)
Re:We raised a bunch of money by xNeil on Wednesday June 28, 2023 @11:23AM (#36507838)
Genuine question - what does being a hacker have to do with being an author?
Re:We raised a bunch of money by xeromal on Wednesday June 28, 2023 @11:42AM (#36508139)
A lot of the famous SF folk are known for being a bit opinionated and snarky in their writing.
Re:We raised a bunch of money by sneak on Wednesday June 28, 2023 @04:25PM (#36512086)
It's like that scene at the end of Dr Strangelove - "why didn't you tell the world, eh?!"

Doing stuff isn't enough - you also need to communicate to the Earth that you have done stuff.

Re:We raised a bunch of money by renewiltord on Wednesday June 28, 2023 @11:24AM (#36507850)
Hard to say but good devrel drives grassroots adoption. Ultimately, though, VCs no longer will fund without revenue growth in addition to user growth.
Re:We raised a bunch of money by moneywoes on Thursday June 29, 2023 @10:42AM (#36521025)
Hmm so their content marketing strategy has been very effective it sounds like

Was it just articles or more?

Re:We raised a bunch of money by buf on Wednesday June 28, 2023 @10:29AM (#36507025)
I'm at least 50 of them
Re:We raised a bunch of money by yieldcrv on Wednesday June 28, 2023 @10:36AM (#36507095)
> that one of them was elected to the county Board of Supervisors.

can you explain this part of the joke?

Re:We raised a bunch of money by dswalter on Wednesday June 28, 2023 @10:37AM (#36507111)
The idea is that Rails apps are such a large portion of the population that they elected a representative from within themselves.

It's not intended to be taken literally (I assume).

Re:We raised a bunch of money by LeifCarrotson on Wednesday June 28, 2023 @11:40AM (#36508083)
EC2 instances are almost certainly a supermajority of the population. What's surprising is that there are still humans on the county board!

Must be gerrymandering or something, maybe aided by voter suppression by English speakers against those in the county whose native language is x86-64.

Re:We raised a bunch of money by jeremyjh on Wednesday June 28, 2023 @01:24PM (#36509615)
Most Rails apps just aren't engaged politically. More education and activism is necessary to mobilize them.
Re:We raised a bunch of money by SonOfLilit on Wednesday June 28, 2023 @10:42AM (#36507175)
In Florida you have so many hispanic citizens that they have significant representation in local politics. In this specific county you have more cloud servers serving rails apps. However, they're not citizens and can't get elected to public service.
Re:We raised a bunch of money by titanomachy on Wednesday June 28, 2023 @10:58AM (#36507419)
This answer, while correct, somehow feels very GPT.
Re:We raised a bunch of money by SonOfLilit on Wednesday June 28, 2023 @05:19PM (#36512751)
Honestly the question felt very Llama in the sense that it didn't really pass my Turing test, so I couldn't really answer it in a way that didn't feel robotic.
Re:We raised a bunch of money by afterburner on Wednesday June 28, 2023 @11:14AM (#36507685)
Maybe also related to the recent news of a small town in Delaware allowing corporations to vote in municipal elections (not a new thing for Delaware apparently).
Re:We raised a bunch of money by arandr0x on Wednesday June 28, 2023 @11:40AM (#36508098)
In a reverse Turing moment, I actually submitted this joke to a chatbot, and it got it.
Re:We raised a bunch of money by yieldcrv on Wednesday June 28, 2023 @03:13PM (#36511162)
ChatGPT doesn't know that I was hoping there was some truth to it after September 2021.
Re:We raised a bunch of money by candiddevmike on Wednesday June 28, 2023 @10:50AM (#36507288)
Unfortunately for fly, us-east-1 has better reliability then their product.
Re:We raised a bunch of money by arandr0x on Wednesday June 28, 2023 @11:39AM (#36508067)
I've worked on the reliability team at a place that made what they call an "edgy app" (long before fly, so we made our own stuff on baremetal) and to be fair, reliability is the actual challenge when you're talking multi-region seamless deployments of scalable web platforms. It makes or breaks that kind of product at least for the original market.

Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.

Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @10:53AM (#36507336)
We're not. For a company of our age and scale? To claim competitive reliability to the world's most important data center? Risible. We're making fun of the fact that everybody deploys their apps in a random city in Virginia with like 3 restaurants in it. We're not saying they're dumb to do it, or that Ashburn isn't up to the challenge --- it self-evidently is.

(The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).

Re:We raised a bunch of money by candiddevmike on Wednesday June 28, 2023 @10:57AM (#36507397)
You guys may want to rebrand once you get your issues sorted out. Fly is going to have the same problem as btrfs, where folks will have a hard time deciding if it's ready for production yet.
Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @11:01AM (#36507455)
It's comments like this that moved me to claim, in our infra/ops job post, the benefit "The most lovable user base in the whole technology industry, and a company culture that will keep you in touch with them directly."
Re:We raised a bunch of money by culi on Wednesday June 28, 2023 @02:01PM (#36510195)
Fly.io was the darling of the webhosting world amongst techworkers particularly because of the honest transparent communication and blog posts explicitly admitting fuck ups

Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes:

1. It gives you a chance to show concrete steps to prevent specific concerns we have

2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks

Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch.

I've never seen a company rebound from losing that confidence

Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @02:09PM (#36510289)
If we're "honest" and "transparent", it's not a conscious strategy to sign up customers. It's just the easiest way for us to write. Meanwhile: we're a hardware-based global hosting company. We've always been externally funded. That's how you build a global hardware-backed hosting company, because servers and colocation and infra engineers are expensive.

There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers.

There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.

Re:We raised a bunch of money by PKop on Wednesday June 28, 2023 @10:19PM (#36515561)
>Raising a bunch of money just isn't a "good" announcement to most of us

Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?

Re:We raised a bunch of money by mananaysiempre on Thursday June 29, 2023 @06:06AM (#36518482)
The issue is not raising money, precisely, the issue is raising money by selling large amounts of controlling equity (as is usually the case for VC funding and IPOs, with a few notable exceptions). The purchasers will expect their money’s worth, and they will (eventually) not hesitate to exercise the control it bought them in order to get that.

VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model.

So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier.

IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately.

So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community).

As usual, the question remains—why does none of this seem to be priced in? I don’t know.

Re:We raised a bunch of money by hgsgm on Wednesday June 28, 2023 @11:10AM (#36507599)
Does PR know you are posting this? Number of restaurants is a measurement of hosting quality? Uptime doesn't matter?
Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @11:14AM (#36507679)
This is neither here nor there, but every time I read a comment like this, all I can think about is that scene in The Office where Dwight Schrute is screaming into his cell phone "Are you saying you invented paper?!"
Re:We raised a bunch of money by xNeil on Wednesday June 28, 2023 @11:23AM (#36507832)
Yeah this is pretty funny - if you speak to PR before posting something, it's 'PR-riddled bull** written by some executive', or 'corporate talk'. If you don't speak to PR before posting something, it's 'Did you speak to PR before posting this?' There's no winning.
Re:We raised a bunch of money by robertlagrant on Wednesday June 28, 2023 @11:49AM (#36508238)
The Office? Dwight Schrute?

ARE YOU SAYING YOU INVENTED THE OFFICE?

Re:We raised a bunch of money by ralgozino on Wednesday June 28, 2023 @11:18AM (#36507751)
the whole point is deploying closer to your users, unless your customer base is the owners of those restaurants, it would be better to deploy to other (even all!) regions
Re:We raised a bunch of money by alex_lav on Wednesday June 28, 2023 @11:47AM (#36508210)
To be fair (this comment is mostly a joke), Ashburn/Loudoun County is pretty nice for a suburb/strip mall hell. I've had some really good food in that area.
Re:We raised a bunch of money by rirze on Wednesday June 28, 2023 @12:31PM (#36508853)
Ashburn does not have "like three restaurants". The datacenter zoned part of the city has "like three restaurants". If you ever drove >5 minutes from those large shed warehouses, you'd know what you're talking about.
Re:We raised a bunch of money by mike_d on Wednesday June 28, 2023 @05:15PM (#36512721)
You are probably thinking of Sterling, Chantilly, etc.

Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.

Re:We raised a bunch of money by rirze on Thursday June 29, 2023 @11:45AM (#36522015)
I lived here my entire life, I know where Sterling is and how it's a 15 minute drive from there to Chantilly (which is not in Loudoun County btw).

Take a look at this map here: https://www.loudoun.gov/DocumentCenter/View/171141/2022Elect...

Ashburn is not literally just one shopping plaza. It's a giant area zoned into different parts of datacenter, suburbs, and commercial sections.

Even forgetting all of this, just type "restaurants in Ashburn" into Google Maps see the plethora of results.

Just because someone has some imaginary boundary of what constitutes as Ashburn doesn't make it definitively small.

Re:We raised a bunch of money by tptacek on Thursday June 29, 2023 @01:40PM (#36523855)
As you can see from the post, we did that. :)
Re:We raised a bunch of money by urda on Wednesday June 28, 2023 @08:56PM (#36514964)
So you fuss at your users who have valid complaints about the availability of your product? It's a valid complaint when the uptime of a service, program, or feature running is critical to your business.

I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.

Re:We raised a bunch of money by dcow on Wednesday June 28, 2023 @09:37PM (#36515258)
> I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important.

I'm confused, when did any of Fly's employees say as much?

Re:We raised a bunch of money by nickpeterson on Wednesday June 28, 2023 @10:54AM (#36507358)
Well if you’re right maybe in a couple years we’ll see a ‘We razed a bunch of money’ post, seems on brand.
Re:We raised a bunch of money by mrkurt on Wednesday June 28, 2023 @11:03AM (#36507488)
I was planning an "Our incredible journey" post, but "We razed a bunch of money" is much better.
Re:We raised a bunch of money by nickpeterson on Wednesday June 28, 2023 @11:15AM (#36507703)
It’s from the Joker school of money management. Although I believe he only burns his half.
Re:We raised a bunch of money by jchw on Wednesday June 28, 2023 @10:59AM (#36507445)
Honestly though, Fly.io feels like it's off into the future in terms of hosting platforms. Undoubtedly they'll have to work on the reliability problems, not denying that - however, for anyone that can tolerate a bit of reliability issues, which to be honest, is more people than will readily admit it, it might just be worth the trouble for developer experience that legitimately feels like it's from the future.
Re:We raised a bunch of money by candiddevmike on Wednesday June 28, 2023 @11:07AM (#36507557)
What good is a product like this if it's not stable? Most folks can yolo a global infrastructure if they disregard SLAs...
Re:We raised a bunch of money by jchw on Wednesday June 28, 2023 @11:10AM (#36507613)
It's definitely not unusably unstable, at least from my PoV, and in fact, I haven't really been hit by most of the instability personally; or at least, if I have, it has gone mostly unnoticed by me. I had an issue where I couldn't deploy for a while, but the services that were up seemed fine. (Disclaimer: I do not use the Redis/database features.)

That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.

Re:We raised a bunch of money by kkielhofner on Wednesday June 28, 2023 @11:05AM (#36507518)
People forget AWS is 20 years old. When I routinely remind people of that fact they often remark (incredulously) with "WHAT? Really?" When I think about it the entire paradigm of "regions" itself seems completely antiquated in the grand scheme of things. AWS has made a few moves on this but in the end you're largely still tied to this fundamental regional concept in terms of control planes, etc and you incur the ridiculousness that is VPCs, wild billing and availability issues, all kinds of hacks, stacks, and effort all over the place to have multiple regions/AZs, etc.

I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.

When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.

Re:We raised a bunch of money by kortilla on Wednesday June 28, 2023 @11:16AM (#36507711)
The realities of regions don’t go away. It’s a primitive to save you money. If a new entrant doesn’t have it they are either just operating in one region or you’re paying cross-region prices for everything.
Re:We raised a bunch of money by kkielhofner on Wednesday June 28, 2023 @12:22PM (#36508701)
Cloudflare products and pricing (what I'm most familiar with) are in wild opposition to this view.

I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea.

Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?).

One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".

Re:We raised a bunch of money by grog454 on Wednesday June 28, 2023 @01:24PM (#36509611)
> Where do my Workers run? Don't know, don't care

Glad that works for you but for my game servers edge computing is not ideal and generally currently not even applicable.

Re:We raised a bunch of money by SantaCruz11 on Wednesday June 28, 2023 @01:36PM (#36509815)
Who do you use for game server edge computing? Edgegap?
Re:We raised a bunch of money by aeyes on Wednesday June 28, 2023 @02:04PM (#36510237)
Cloudflare also depends on investor money, they haven't made a profit ever and continue to just throw stuff at the wall to grow.

At some point in time they'll have to start thinking about profitability.

Re:We raised a bunch of money by short_sells_poo on Wednesday June 28, 2023 @02:43PM (#36510712)
It's important to qualify this though. E.g. Amazon was not profitable for a very long time, but only because they furiously reinvested everything. They could've turned profitable at any time.

Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.

Re:We raised a bunch of money by applied_heat on Wednesday June 28, 2023 @03:05PM (#36511040)
Big assumption?
Re:We raised a bunch of money by jamwil on Wednesday June 28, 2023 @09:34PM (#36515235)
Their financials are public.
Re:We raised a bunch of money by applied_heat on Wednesday June 28, 2023 @11:53PM (#36516115)
Ah! No need to make any assumptions then.
Re:We raised a bunch of money by 015a on Wednesday June 28, 2023 @05:01PM (#36512530)
It really doesn't though. Sure, it would if the difference is only "16 mega-regions (regional-model) versus 300 mini-regions (edge)". But that's not the difference.

The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere.

This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry.

In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations.

Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has.

Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete.

The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.

Re:We raised a bunch of money by bfung on Wednesday June 28, 2023 @11:22AM (#36507811)
Stateless compute is relatively simple to be region-less, but even fly’s homepage has a map of regions.

When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.

Re:We raised a bunch of money by bob1029 on Wednesday June 28, 2023 @12:31PM (#36508850)
> When state is introduced

Only the most trivial business systems can operate without any state.

If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same.

For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.

Re:We raised a bunch of money by swyx on Wednesday June 28, 2023 @06:10PM (#36513340)
im relatively well read on databases and couldnt figure out - what is the EC in your comment? EC2?
Re:We raised a bunch of money by bob1029 on Wednesday June 28, 2023 @06:33PM (#36513573)
eventual consistency
Re:We raised a bunch of money by bfung on Thursday June 29, 2023 @02:35PM (#36524764)
Let’s hope someone doesn’t put the backup/replica in different region, the one big instance gonna be tough to restore!
Re:We raised a bunch of money by kkielhofner on Wednesday June 28, 2023 @12:33PM (#36508881)
This is Fly - a much earlier stage company and them receiving early funding is the genesis of this entire discussion.

I point to Cloudflare (what I'm most familiar with). Across their entire stack of compute, storage, db, etc product lines you won't see a hint of region anywhere. It just works, everywhere. Same with Fastly/Oracle (although they like Fly are quite a ways behind Cloudflare).

I suspect (hope) that as Fly matures their apparent true goal of edge and region-less applications will mature more. Their compromises on region awareness for DB has always struck me as a more of a resource restraint compromise on their part than end goal. As you say these challenges are hard (expensive).

Re:We raised a bunch of money by bob1029 on Wednesday June 28, 2023 @05:09PM (#36512648)
> you won't see a hint of region anywhere.

https://developers.cloudflare.com/d1/learning/data-location/

  Hint Hint description
  wnam Western North America
  enam Eastern North America
  weur Western Europe
  eeur Eastern Europe
  apac Asia-Pacific
There is no free lunch here. This is always a game of information theory and physics at the end.
Re:We raised a bunch of money by kkielhofner on Thursday June 29, 2023 @08:44AM (#36519623)
If I were at Cloudflare I'd suggest calling these "geographies" where the magic still applies (any one of those geographies has at least dozens of component POPs that are invisible).

The fact remains this is somewhat down in the docs because it is automatic and invisible to the point it works well enough where most developers don't know, don't care, and aren't exposed to it in any meaningful way. Contrast this with AWS where the first thing you are presented with is picking a region where everything lives, while not clearly and explicitly explaining that many things are ultimately dependent on us-east. Cue the shock and confusion when us-east has yet another issue and those in all other regions don't understand how and why they are impacted.

You are correct about the free lunch, information theory, and physics at the end. I don't think a reasonable person can think when you write data to a pop in Tokyo that it could possibly show up in Chicago instantly. That is fundamentally impossible and always will be. I think CF does a pretty good job making it "magic" for most devs and use cases while explaining these fundamental realities and limitations to those who need to know (potentially) for whatever their application is.

Re:We raised a bunch of money by skybrian on Wednesday June 28, 2023 @05:13PM (#36512691)
"Won't see a hint of region anywhere" is an exaggeration.

Here's the Durable Objects documentation [1]. Don't those "location hints" look like regions?

Also, here's the "Data location" page for Cloudflare D1 [2]. See "available hints" at the bottom.

I think it would be more accurate to say that Cloudflare's architecture is designed around having lots of regions, automatic migrations between them, and no guarantee about which one you'll get. But they're still there.

[1] https://developers.cloudflare.com/workers/runtime-apis/durab...

[2] https://developers.cloudflare.com/d1/learning/data-location/

Re:We raised a bunch of money by kentonv on Wednesday June 28, 2023 @05:41PM (#36513005)
These are offered more as an escape hatch in case the system doesn't do what you want automatically. But the goal is definitely for the system to do what you want automatically, so that no one ever has to provide these hints.

I actually argued against adding these hints at all, since it confuses the messaging and most people don't need them... but they were easy to add and solved an immediate problem for a few customers, so we did.

In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...

In any case, most people don't have to think about regions on Workers, and for the few that do, our goal is to change that.

(I'm the tech lead for Cloudflare Workers.)

Re:We raised a bunch of money by kkielhofner on Thursday June 29, 2023 @08:34AM (#36519536)
Thank you for the context and this is what I was getting at - if you really want to dig in on things like KV, D1 and DO you will see occasional mentions of regions (I'd really call it geography) to add context on things like "KV is eventually consistent, instant in the pop where the write takes place but takes X ms to propagate everywhere". For many (most?) applications it doesn't really matter, I get the sense for these you clearly document these things so people understand the more-or-less inescapable fundamental reasons why things are what they are. As other commenters have noted "there is no free lunch" when it comes to things like the physics of getting data from a pop in Tokyo to Chicago but Cloudflare does a good job and making it happen more-or-less invisible.
Re:We raised a bunch of money by mahmoudimus on Tuesday July 04, 2023 @11:42AM (#36588164)
> In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...

That is an excellent way to design a product. Great job.

Re:We raised a bunch of money by asim on Wednesday June 28, 2023 @11:48AM (#36508230)
THIS. No one cares about regions, you just care that the app is fast and globally available. The way we did things is a relic of the past and everyone mimics it because trying to guide new user behaviour is hard. The reality is we just need the app to be globally deployed with a globally distributed database and anycast DNS to route you to the local most DC. There's a lot more to it, have built this architecture more than a couple times, but it all needs to be in the service of something and if you're UX isn't better than the next guy, it won't matter much.
Re:We raised a bunch of money by SamuelAdams on Wednesday June 28, 2023 @12:28PM (#36508806)
As a counterpoint, anyone working with compliance data needs to know and be certain that data does not cross geo-boundaries. Intelligence data for US gov’t agencies must typically be kept stateside: AWS uses the GovCloud region specifically for satisfying gov compliance requirements.

Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible. I haven’t worked with GDPR requirements but I have heard they are similar. AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.

Re:We raised a bunch of money by kkielhofner on Wednesday June 28, 2023 @12:51PM (#36509137)
Cloudflare does this although to my knowledge it's not available without an Enterprise subscription which is the only thing they offer with this level of compliance anyway.

All lower plans maybe get PCI DSS and that's it.

It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data.

We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day.

I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers.

Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.

Re:We raised a bunch of money by scarface_74 on Wednesday June 28, 2023 @09:01PM (#36514999)
> Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant".

N-2 companies ago I was a tech lead in 2015 where we were migrating to AWS. AWS was constantly explaining which underlying services were HIPAA compliant and what you build on top of it was your responsibility. I had to answer to separate non AWS compliance folks about my architecture.

AWS also gave guidance. But made no promises that your implementation was compliant. That’s where the entire “shared responsibility model” comes in.

Now, I work at AWS in ProServe. I am very familiar with our messaging regarding HIPAA compliance. We are very careful not to rubber stamp things as compliant. I know the best practices in and out. But when I’m asked if something is compliant, I say this is the guidance we are given. I’ll do a presentation before an Architecture Review board. But I would never make anything that hints as a guarantee.

Re:We raised a bunch of money by kkielhofner on Thursday June 29, 2023 @10:22AM (#36520733)
Both anecdotal experiences but in my recent exercises with HIPAA compliance with AWS, etc they were nowhere near this explanatory/helpful.

The messaging/communication was more like what I described - "Oh yeah here's BAA you're good to go".

Re:We raised a bunch of money by scarface_74 on Thursday June 29, 2023 @01:51PM (#36524062)
How large were your employers? If you have Enterprise support, you definitely get one or more TAMs dedicated to helping your company.

If you are a smaller organization, you may be able to request an SA.

My personal experience is from when companies pay for consulting hours from ProServe.

At the n-2 company I referred to above, the only thing we got from AWS is a BAA and we had an outside consulting company and compliance auditors. I was just learning AWS then.

Re:We raised a bunch of money by kentonv on Wednesday June 28, 2023 @05:46PM (#36513078)
Compliance is a completely different conversation.

In the Cloudflare Workers platform, we have a number of features to control "jurisdictions" where compute runs or where data is stored. These are meant for compliance, not for performance. Our goal is that no one has to think about regions for performance reasons, but certainly compliance issues are not going to go away -- in fact, they are getting more onerous over time. So features to control jurisdiction are likely to expand.

(I'm the tech lead for Workers.)

Re:We raised a bunch of money by makestuff on Wednesday June 28, 2023 @12:28PM (#36508801)
Regions do matter for stuff like GDPR though. It is beneficial to know that all of the data is in the EU region and will stay there.
Re:We raised a bunch of money by aleksiy123 on Wednesday June 28, 2023 @04:00PM (#36511792)
This is MAY be true consumer applications and even then not all.

But it is rarely true for enterprise customers. They want the guarantees their data never leaves certain regions even in a request along with other security and compliance guarantees.

Re:We raised a bunch of money by cnity on Wednesday June 28, 2023 @11:49AM (#36508237)
The question of whether "the edge" takes off will come down to whether or not the culture at large will swallow the illusion at its heart. You cannot remove the concept of a region any more than you can remove the concept of "the computer" in a cloud environment.
Re:We raised a bunch of money by kkielhofner on Wednesday June 28, 2023 @01:05PM (#36509322)
"The cloud is just someone else's computer" - that ship sailed at least a decade ago.
Re:We raised a bunch of money by pests on Wednesday June 28, 2023 @02:03PM (#36510216)
> any more than you can remove the concept of "the computer" in a cloud environment

But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.

Re:We raised a bunch of money by lightbendover on Wednesday June 28, 2023 @02:54PM (#36510873)
Except, you always interact with computerS no matter what provider or architecture you use. It's inescapable as long as you're doing business in software. The trade-offs in architecture and regionality do NOT go away just because a company abstracts over it. They may be able to set up read replicas and consensus on the fly and be able to detect when various approaches make the most sense, but you are still at the mercy of the accompanying constraints. For most use cases, none of this may matter. When you do run into a use case where it matters, you should be fully aware of what trade-offs you're making and have control over them.

Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.

Re:We raised a bunch of money by washywashy on Wednesday June 28, 2023 @12:32PM (#36508867)
Regions also important for more regulated geographies
Re:We raised a bunch of money by alpha_squared on Wednesday June 28, 2023 @01:38PM (#36509835)
> It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.

I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough).

Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.

Re:We raised a bunch of money by scarface_74 on Wednesday June 28, 2023 @02:07PM (#36510263)
I work at AWS all opinions are my own.

AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.

Re:We raised a bunch of money by yencabulator on Thursday June 29, 2023 @11:30AM (#36521780)
These days, "edge" just means "has more than N data centers" (where N depends on the speaker). Fly and Cloudflare have regions too, try `flyctl platform regions` or see https://www.cloudflare.com/network/
Re:We raised a bunch of money by jgrahamc on Thursday June 29, 2023 @11:39AM (#36521935)
Cloudflare does not have "regions": your stuff runs everywhere on our network. And we can be really smart about where's the best place: https://blog.cloudflare.com/announcing-workers-smart-placeme...
Re:We raised a bunch of money by yencabulator on Thursday June 29, 2023 @12:15PM (#36522478)
That sounds like "Cloudflare automated region handling", not that regions don't exist.

For example, make your app talk to two backends in different EC2 regions, and now none of the smart placement helps you.

The poster we're replying to seems to be gleefully ignorant of speed of light limits and fundamental distributed systems challenges.

Re:We raised a bunch of money by jgrahamc on Saturday July 01, 2023 @06:33AM (#36548646)
No. That’s wrong. We don’t “manage regions”. 300 data centers. Everything running everything. Only time you’d need to think of a region is if you have some compliance situation (“keep all my data in North America” or “only terminate TLS in the EU”) and ask to subdivide our network using Regional Services. But there’s no decision on here you say “I’ll use Cloudflare-EU-North-42” and get stuck with it as no such things exist.
Re:We raised a bunch of money by jonluca on Wednesday June 28, 2023 @11:11AM (#36507617)
This is a weird coincedence - I just published a blog post on this two days ago https://blog.jonlu.ca/posts/snappy-internet
Re:We raised a bunch of money by mrkurt on Wednesday June 28, 2023 @11:18AM (#36507750)
This effect is bananas. The internet feels remarkably different in Sydney and Tokyo than it does in Chicago.
Re:We raised a bunch of money by awkward on Wednesday June 28, 2023 @11:22AM (#36507826)
Unfortunately, the motto is "I byde my time". Seems like he could have still made a joke without making it up.
Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @11:37AM (#36508048)
We got this off the Loudoun County website! We do our research for these things!
Re:We raised a bunch of money by margalabargala on Wednesday June 28, 2023 @02:07PM (#36510271)
While "where tradition meets innovation" is a set of words that the county puts on their logo, and appears to meet the definition of a motto, the parent is correct that the Official Motto is indeed "I Byde My Time".

https://www.loudoun.gov/177/Coat-of-Arms

"Where Tradition Meets Innovation" appears to be a branding push by the county dating to 2019 rather than an official motto. https://www.loudoun.gov/ArchiveCenter/ViewFile/Item/8563

Re:We raised a bunch of money by tptacek on Wednesday June 28, 2023 @02:11PM (#36510320)
People wonder why I'm so addicted to this site, but I mean, it's obvious right here, isn't it?