News for hackers. Stuff that matters.
Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @11:17AM (#33743795)
Oof, that sucks and I feel for you. That said...

> setting up in a new region would be complicated for us.

Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud.

When using the public cloud, we do tend to take it all for granted and don't even think about the fact that physical hardware is required for our clusters and that, yes, they can run out.

Anyways, however hard getting another region set up may be, it seems you've no choice but to prioritize that work now. May also want to look into other cloud providers as well, depending on how practical or how overkill going multi-cloud may or may not be for your needs.

I wish you luck.

Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @11:24AM (#33743889)
I'll reply to my own comment in response to a since-deleted reply that went something to the effect of "this is terrible advice for a young startup trying to get to product market for":

I'm totally on board with the idea of being scrappy and taking shortcuts in order to get to PMF as soon as possible. However, it seems the proof is in the pudding here. If you can't service customers due to lack of compute resources, you can't get to PMF.

Also, yes there are certain infrastructure and network topologies that would absolutely be overkill for a young startup. I don't think multi-region is one of those things. I don't have experience with Azure directly, but on every other cloud providers, going multi-region is not something that requires huge amounts of time or resources. You just need to be mindful of it from the outset. And if you decide not to be, then at least be intentional and conscious about the risk and have a plan in place for what happens when you get bit by deciding not to go multi-region.

Re:Azure has run out of compute. Anyone else affected? by cheese123 on Friday November 25, 2022 @11:34AM (#33744020)
GDPR might be a problem here. But this brings us to an important point: this is not your infrastructure, but someone elses.
Re:Azure has run out of compute. Anyone else affected? by soco on Friday November 25, 2022 @11:39AM (#33744105)
There are quite a few European regions, but we don't know how the others stand with their computing limits...
Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @11:36AM (#33744052)
Good point, I didn't think about that.
Re:Azure has run out of compute. Anyone else affected? by masom on Friday November 25, 2022 @12:28PM (#33744719)
GDPR isn't really related to the infrastructure, and isn't a problem if you built your product knowing you'll need to conform. Shopify is GDPR compliant, for all merchants, and runs on Google Cloud in multiple regions.
Re:Azure has run out of compute. Anyone else affected? by pm90 on Friday November 25, 2022 @12:38PM (#33744826)
Genuine question: is knowledge on how to do this well known? Without that accessibility, I'm picturing folks operating in EU being unwilling to take the risk of not being compliant and just hosting everything in a single region.
Re:Azure has run out of compute. Anyone else affected? by nr2x on Friday November 25, 2022 @01:13PM (#33745194)
They’re wrong.
Re:Azure has run out of compute. Anyone else affected? by unionpivo on Friday November 25, 2022 @01:30PM (#33745344)
GDPR is not some boogieman, it can be pain to do on existing products that were build pre GDPR, but if you are starting new project, being GDPR compliant is pretty straight forward and not hard/time consuming, unless you are explicitly trying to do something shady*

*privacy invasive that GDPR is explicitly set up to make harder, so duh

Re:Azure has run out of compute. Anyone else affected? by revicon on Friday November 25, 2022 @12:37PM (#33744811)
There are specific requirements in Germany that require user data to not leave the country. I believe that was what OP was referring to.
Re:Azure has run out of compute. Anyone else affected? by nr2x on Friday November 25, 2022 @01:12PM (#33745184)
Privacy shield very much matters where your servers are. EU cracking down hard on extra territorial transfers in the past year with more to come.

Also, lots of companies assert GDPR compliance via magical thinking. They most often are wholly wrong. Shopify can say whatever they want, but there’s no certification body.

Source: I’m the person who evaluates and builds compliance systems for a range of services you almost definitely use.

Re:Azure has run out of compute. Anyone else affected? by logisticpeach on Friday November 25, 2022 @12:41PM (#33744868)
I'd add to this by asking: how much more PMF can you get when you have a two week horizon of new customers before you literally run out of compute resource in a major cloud provider data centre?

Sounds like customers are coming in thick and fast.

If this is the dynamic and the company can't spare a few weeks to solve it, something has gone seriously wrong in a very interesting way.

Re:Azure has run out of compute. Anyone else affected? by arcturus17 on Friday November 25, 2022 @01:04PM (#33745105)
Also, n8n arguably has product-market fit so the advice was impertinent to start with…
Re:Azure has run out of compute. Anyone else affected? by janober on Friday November 25, 2022 @11:24AM (#33743881)
Thanks a lot! You are totally right, it is for sure something we will find a solution for. But honestly, do I not want to. As a startup, you have very few resources and deliberately place some exact bets. Deprioritizing everything to work on something for a long time that was not prioritized, just to then end up again where you were before (a working cloud solution) is the last thing any startup should be forced to do. Anyway, it seems like we do not have much choice here.
Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @11:31AM (#33743978)
I hear you. It's not a fun position to be in. And sometimes you're correct to take calculated risks and maybe the expected value was positive here, despite what ended up happening.

Without knowing the details about your services and infrastructure, it's hard for me to know what's involved in going multi-region now. Are you sure it's such a a gargantuan effort? I would've thought one person working full-time on this for a week or two would be enough, but again I don't know the details of your setup.

One option would be to pay a consultant who is an expert in Azure/cloud stuff to come in and help. May not be cheap, but could be a lot better and quicker for you and better for the business, especially if none of you are really big experts in Azure.

I've been here before (I think)...had to wear many hats and scramble to make sales, build the tech, act as de facto DevOps person even without a lot of experience doing it, etc. That is the way, but stuff happens.

Happy to chat about specifics if you want to bounce ideas off of me or go through your particular situation. Can't promise I'll have concrete advice, but happy to talk it through.

Re:Azure has run out of compute. Anyone else affected? by janober on Friday November 25, 2022 @11:37AM (#33744073)
Thanks a lot, is really super nice of you and appreciated! Luckily we have somebody very knowledgeable on our team. Will tell him to reach out if he wants to have a peer to brain-storm some ideas.
Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @12:43PM (#33744883)
Glad to hear you have the right people. Good luck, my friend.
Re:Azure has run out of compute. Anyone else affected? by martinald on Friday November 25, 2022 @12:39PM (#33744841)
I disagree with the other poster you should have been multiregion from the start. It adds a load of complexity and failure cases for early stage Startups.

Very poor position to be in, apparently this happened in azure UK recently too.

Re:Azure has run out of compute. Anyone else affected? by dh2022 on Saturday November 26, 2022 @12:45AM (#33750322)
I do not think is a bad idea to be multi-region from start. For the most part Azure has at least two regions in each country (Germany North/West Central, UK South/West, Sweden South/Central, Norway East / West, UAE North / Central, France Central / South etc....)So if stuff happens being able to bring up your service in a different region in the same country could be helpful. I do not know specifics but it seems to me that having an abstraction layer on top of the region is not that hard to do (most of Azure services are supported in all regions). OF course, is a lot easier if done at the outset. Being forced to do it quickly and with little notice is no fun at all....
Re:Azure has run out of compute. Anyone else affected? by jnsaff2 on Friday November 25, 2022 @02:50PM (#33746091)
I feel for you. Also it sucks to be in this position.

Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company.

In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig.

That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.

Re:Azure has run out of compute. Anyone else affected? by websap on Friday November 25, 2022 @11:30AM (#33743960)
Is it that much cheaper for you to build a new region on Azure versus getting setup on AWS?

If you rely on Kubernetes for orchestration and have minimal cloud API dependency, it may be worth that evaluating this option.

Also, do you have a TAM associated with your account? Are you just going through regular support channels? Can they deliver different instance types (not sure what the Azure parallel is), can they deliver short term capacity, etc?

I would try to push Microsoft more here. It's not like they've stopped on-boarding new customers into that region right? What happens if you create a new account in that region?

Re:Azure has run out of compute. Anyone else affected? by janober on Friday November 25, 2022 @11:42AM (#33744134)
We already tried to push Microsoft, sadly have they been not very helpful. Still trying to get in contact with somebody that can actually make a difference. After all, are we also not asking for a hundred machines. Can really not imagine that they can not somehow make the resources we require available.
Re:Azure has run out of compute. Anyone else affected? by dekhn on Friday November 25, 2022 @11:44AM (#33744161)
I'll ask again what the person above asked: do you have a TAM.

If you don't you're at a big disadvantage.

Re:Azure has run out of compute. Anyone else affected? by cyptus on Friday November 25, 2022 @12:16PM (#33744573)
we tried escalating this through CASM and it did not work. The region is blocked for every quota, even a single instance.
Re:Azure has run out of compute. Anyone else affected? by janober on Friday November 25, 2022 @11:56AM (#33744303)
Sorry, missed that. No, we do not.
Re:Azure has run out of compute. Anyone else affected? by websap on Friday November 25, 2022 @12:05PM (#33744451)
Get one asap. Your TAM is the insider and should push for you.
Re:Azure has run out of compute. Anyone else affected? by jiggawatts on Friday November 25, 2022 @04:07PM (#33746756)
Some tricks you can try is to switch to a different SKU. Most Azure databases have different generations of underlying compute. They may be out of just on model. Try a different one.

Similarly, just keep trying to change the size. Often it’ll go through when someone else decommissions something.

Re:Azure has run out of compute. Anyone else affected? by psychphysic on Friday November 25, 2022 @11:55AM (#33744293)
> Deprioritize all other work, get everyone working on this little DevOps/Infra project.

This is doubly worthwhile as if this stumble kills the startup (it can happen) this will be excellent experience to take to the next employer :)

Re:Azure has run out of compute. Anyone else affected? by lanstin on Friday November 25, 2022 @01:43PM (#33745474)
Also a lesson here is don’t create any prod asserts manually ever. Terraform or some other software to define your cloud assets. Then this issue is just a matter of adding a top level loop or maybe adding a region parameter to a layer of software. Cloud is only efficient if you take the software defined every thing seriously. Otherwise it is premium hosting where you are likely a small fish.
Re:Azure has run out of compute. Anyone else affected? by snotrockets on Friday November 25, 2022 @11:59AM (#33744341)
“Being multicloud from the outset” is a very silly idea for most use cases.

The way to get more from most cloud is by becoming a partner, not just a customer. And the way to do that is increase dependency and usage.

Re:Azure has run out of compute. Anyone else affected? by salawat on Friday November 25, 2022 @01:58PM (#33745630)
t. Sales department of Cloud provider
Re:Azure has run out of compute. Anyone else affected? by snotrockets on Monday November 28, 2022 @02:00PM (#33777952)
Good sales (which I'm not), is all about aligning with the customer: solve their problems for them, and they'd be happy to pay for it. Getting them to buy stuff they won't need is a sure way to loose future business.
Re:Azure has run out of compute. Anyone else affected? by astrostl on Friday November 25, 2022 @12:40PM (#33744849)
"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.
Re:Azure has run out of compute. Anyone else affected? by gumby on Friday November 25, 2022 @12:48PM (#33744926)
That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS.

(As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

Re:Azure has run out of compute. Anyone else affected? by marcosdumay on Friday November 25, 2022 @01:02PM (#33745078)
That generalizes to every kind of lock-in: have a viable escape plan, but only execute it if you need or it becomes cheap enough that it won't harm you.

Just having the plan is already expensive enough.

Re:Azure has run out of compute. Anyone else affected? by jiggawatts on Friday November 25, 2022 @02:57PM (#33746152)
My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.
Re:Azure has run out of compute. Anyone else affected? by 29athrowaway on Friday November 25, 2022 @02:58PM (#33746172)
And Active Directory integrates horribly with everything outside Microsoft.
Re:Azure has run out of compute. Anyone else affected? by SgtBastard on Friday November 25, 2022 @04:11PM (#33746809)
Azure Active Directory has both SAML and OpenID Connect endpoints… what’s missing?
Re:Azure has run out of compute. Anyone else affected? by nxm on Friday November 25, 2022 @09:45PM (#33749368)
We use it across many non-MS services without issues. Care to expand?
Re:Azure has run out of compute. Anyone else affected? by gumby on Friday November 25, 2022 @07:23PM (#33748503)
We’ve had reliability and availability problems esp with azure and also Google. less with AWS.

None are ideal.

Re:Azure has run out of compute. Anyone else affected? by dangus on Friday November 25, 2022 @08:23PM (#33748885)
Why do I want AD or Microsoft 365?
Re:Azure has run out of compute. Anyone else affected? by philwelch on Friday November 25, 2022 @09:00PM (#33749079)
You personally? No idea. You probably don't. But many (most?) businesses use AD and Office and aren't particularly interested in migrating to alternative solutions.
Re:Azure has run out of compute. Anyone else affected? by gumby on Saturday November 26, 2022 @04:51PM (#33757153)
Yes, Microsoft focuses on the customer (corporate IT), not the user.

This is how the iphone was able to nuke windows phones which were designed to meet the needs of IT

Re:Azure has run out of compute. Anyone else affected? by dangus on Saturday November 26, 2022 @08:35PM (#33759131)
I mainly work in startups. None of them in recent memory have bothered with AD or Office. Okta and Google Workspaces take their place.

Those MS problems have an “IBM mainframe” problem. New businesses won’t choose it.

That’s why I say “why do I want them?” If I was starting a new business I’d have no reason to use them.

Re:Azure has run out of compute. Anyone else affected? by toomuchtodo on Friday November 25, 2022 @09:18PM (#33749192)
O365 = teams, docs, outlook, etc. workspace tools

AD = identity, access, privileging, SSO

Re:Azure has run out of compute. Anyone else affected? by dangus on Saturday November 26, 2022 @08:37PM (#33759148)
I know what they are. No company I’ve worked for recently used them.

In my mind they’re legacy business products.

Sure, most businesses use them, but I don’t necessarily believe that is a forever thing. At one point most businesses had mainframes.

Re:Azure has run out of compute. Anyone else affected? by toomuchtodo on Sunday November 27, 2022 @10:04AM (#33763368)
Refuse reality at your own peril. These tools are actively used at scale across most enterprises, and likely will be for the majority of your career.
Re:Azure has run out of compute. Anyone else affected? by dangus on Sunday November 27, 2022 @12:15PM (#33764587)
How many Fortune 500 companies were founded in the SaaS era?

I'm going to guess that the 5% that don't use Azure/Office 365 represent newer entrants to the Fortune 500.

71% of Fortune 500 companies use mainframes [3], and yet they are considered a dead technology with essentially no future. Do you know anyone or anyone who knows anyone who learned how to develop on mainframes in college in the current millennium? It sure wasn't part of my CS curriculum!

Small businesses represent almost half of US economic activity [1] and represent 99.7% of firms with paid employees [2]

[1] https://advocacy.sba.gov/2019/01/30/small-businesses-generat...

[2] https://cdn.advocacy.sba.gov/wp-content/uploads/2021/12/0609...

[3] https://www.precisely.com/blog/mainframe/9-mainframe-statist...

Re:Azure has run out of compute. Anyone else affected? by kennxfl on Sunday November 27, 2022 @10:10AM (#33763401)
Just to steel man your statement: You should strive for excellence in deployment with all providers (dabble) but have your initial core setup on one cloud (YAGNI principle) until when multicloud capacity is needed (at scale)
Re:Azure has run out of compute. Anyone else affected? by ghayes on Friday November 25, 2022 @12:50PM (#33744952)
Completely agree, though certain aspects, such as running on k8s or Docker might make it easier to switch if you ever decide to, versus say, being tightly coupled with many bespoke cloud products.
Re:Azure has run out of compute. Anyone else affected? by rikthevik on Friday November 25, 2022 @02:02PM (#33745668)
My philosophy is to make switching to a new cloud possible. It doesn't have to be easy. We just shouldn't nail our feet to the floor.
Re:Azure has run out of compute. Anyone else affected? by d3ckard on Friday November 25, 2022 @04:12PM (#33746823)
Or you could just deploy on metal, which will be cheaper and sufficient for vast majority of cases. Plus you can always migrate to VMs with relatively low hassle.
Re:Azure has run out of compute. Anyone else affected? by aftbit on Friday November 25, 2022 @01:27PM (#33745330)
If you use generic enough services (container hosts, load balancers, VMs, object stores, even hosted SQL DBs, etc), then the multi-cloud journey is not that hard. The challenge comes when you have build a whole architecture on top of some AWS magic that simply does not have an easy alternative in the non-cloud world.
Re:Azure has run out of compute. Anyone else affected? by elwell on Friday November 25, 2022 @02:22PM (#33745874)
Surely it would depend on the reliability demands of your product.
Re:Azure has run out of compute. Anyone else affected? by askvictor on Friday November 25, 2022 @03:58PM (#33746674)
On top of which, startups often don't have that luxury; you have often need to ruthlessly prioritise your effort.
Re:Azure has run out of compute. Anyone else affected? by brian_cunnie on Friday November 25, 2022 @10:56PM (#33749754)
True. When Nextel was bootstrapping in the 90's, a VP said, "We have to buy gas for the car now. Later we'll buy the seat belts"
Re:Azure has run out of compute. Anyone else affected? by xwowsersx on Friday November 25, 2022 @05:08PM (#33747348)
Who are you quoting? I said "multi-region from the outset" and later acknowledged that multi-cloud would probably be overkill.
Re:Azure has run out of compute. Anyone else affected? by worthless-trash on Saturday November 26, 2022 @09:51AM (#33753101)
This was exactly what I was thinking, its amazing what people read into, probably myself included.
Re:Azure has run out of compute. Anyone else affected? by arghwhat on Friday November 25, 2022 @07:02PM (#33748348)
You always only need another one when everything has gone to shit, either from failure or cost from vendor-lock-in, so drinking your chosen providers Kool aid equals taking the reactive route and scrambling to rearchitect when the issues hit.

Multi-cloud is really not a big deal. Main nuisance is billing differences, followed by slight variations in e.g. Terraform config.

Re:Azure has run out of compute. Anyone else affected? by nevf1 on Monday November 28, 2022 @08:40AM (#33773503)
Totally agree. If the service you're providing is so important, build your system so it can fly on one engine or at least land safely. Multi-cloud is the equivalence of trying to transfer all of your passengers to a different aircraft mid-air.

Multi-cloud should only be for mission critical infrastructure. Very little infrastructure is mission critical. Most other use cases can be temporarily wallpapered over with an "Under maintenance" page unless there's a good reason otherwise.

Multi-cloud introduces more risk than it prevents. Which is why things like simulated failovers and BCP testing is constantly required.

Re:Azure has run out of compute. Anyone else affected? by mhitza on Friday November 25, 2022 @12:44PM (#33744893)
I take offense with you comment. It's not the first time I'm hearing about multi-region/multi-cloud in online tech forums, however reality doesn't match.

I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practice.

If you can somehow have a separate database per region/cloud, sure, I can understand that, but if you have to shard your database across many clouds, I'd dread having to tame such a beast, especially within a startup.

Re:Azure has run out of compute. Anyone else affected? by panarky on Friday November 25, 2022 @01:00PM (#33745057)
> dismiss the virtue signaling of high resilient ...

So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.

Re:Azure has run out of compute. Anyone else affected? by quesera on Friday November 25, 2022 @01:24PM (#33745303)
If your single-AZ, single-region cloud is not giving you 3 or 4 9's of reliability out of the box, you are using the wrong cloud.

Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability.

Sometimes it is worth it. Sometimes it is not.

Re:Azure has run out of compute. Anyone else affected? by champtar on Friday November 25, 2022 @02:11PM (#33745768)
Depends on your needs, but having your data & database multi-az to ensure durability can avoid you having to restart from backups. I'm thinking about an old AWS incident were they actually lost EBS data: https://www.bleepingcomputer.com/news/technology/amazon-aws-... Also make sure your backups are in a different AZ (thinking about OVH ...) or region or even at a different provider.
Re:Azure has run out of compute. Anyone else affected? by rch on Friday November 25, 2022 @01:05PM (#33745110)
Thanks largely to k8s, running on multiple cloud providers and your own hardware is a lot more convenient than it was a few years ago. Component interfaces and protocols are a lot more consistent across platforms as well.
Re:Azure has run out of compute. Anyone else affected? by theteapot on Friday November 25, 2022 @01:07PM (#33745122)
> everyone working on this little DevOps/Infra project.

Everyone? That's not going to help.