News for hackers. Stuff that matters.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by djfobbz on Saturday March 11, 2023 @12:19PM (#35110409)
Way to go, I approve! Been running bare metal for my SaaS since 2007.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by bhahn on Saturday March 11, 2023 @12:31PM (#35110537)
I've only worked at companies running in the cloud. Is there a console (open-source or otherwise) and other tooling similar to AWS' for managing your resources in your own data center?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by ocdtrekkie on Saturday March 11, 2023 @01:37PM (#35111189)
Plenty of them, yes. I probably would decline to move to a job with a cloud-based setup. It's short-term thinking at it's finest.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by mdaniel on Saturday March 11, 2023 @03:22PM (#35112276)
The OpenStack console is pretty good, imho

If you have an 8GB vm you can see it in action via https://docs.openstack.org/devstack/zed/guides/single-vm.htm... (takes about 45 minutes) or if you want the "full" experience fire up an education cluster from https://openmetal.io/use-cases/on-demand-openstack-cloud/ (it takes about 4 hours or so but it's also a "real" setup unlike the vm which for sure allows running instances via qemu but isn't going to allow exercising the complete suite as easily)

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by Nextgrid on Saturday March 11, 2023 @12:31PM (#35110545)
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by karmicthreat on Saturday March 11, 2023 @12:32PM (#35110553)
I can definitely see advantages for very specific use cases in favor of bare metal colo. But AWS does provide a good amount of value in flexibility and reliability. If you have a whole lot of bandwidth or storage needs you should probably look at a hybrid system. But you will probably pay for it with increased engineering and maintenance costs.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by Xeoncross on Saturday March 11, 2023 @12:32PM (#35110554)
Another interesting thing is that the farther you go from the CPU cache, the more copies of data and the more extra processing you have to do.

Think of how many copies of the data you need (and how much processing it takes) to use:

- a L3 CPU cache.

- a 16GB RAM cache.

- an embedded C/Rust/Go SQLite database.

- a distributed, fault tolerant Cassandra/elastic cache cluster.

You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes.

I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers.

Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your unoptimized vanilla Ruby or Java install you didn't even have time to write tests for.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by hot_gril on Saturday March 11, 2023 @01:18PM (#35110982)
For use cases with simple business logic and high throughput, yeah the optimized Rust, maybe on your own hardware, makes sense.

Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by Xeoncross on Saturday March 11, 2023 @03:33PM (#35112404)
Yeah its kind of sad that we still have to use node and java with better options out there. Don't get me wrong, Typescript, Deno, JVM, etc.. are all doing their best to right those ships, but it's would still be better to just scrap that stuff for a better foundation.

That is what everyone knows,

so that is what's taught,

so that is what everyone knows,

so that is what's taught.

I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by moomoo11 on Saturday March 11, 2023 @05:06PM (#35113277)
The problem is that our industry democratized to the point that people want to hire the cheapest resources who can ship something. A mediocre node or ruby developer costs peanuts to hire compared to one who is mediocre at Go or Erlang. Btw.. If you hire a mediocre go or Erlang developer you’re fucked.

For IO bound work there’s not much difference between a service built with Go or Node. It’s really when you need services that have soft/hard real-time requirements or continuous processing requirements that imo it’s necessary to hire the right people for the right stuff.

For anyone else who is making a web platform that doesn’t have much complex backend processing, they can just use ruby or node.

Personally I use Go for all backend tasks and node for any one off scripting work. Go spits out a binary I can run on any Linux based system. Node scripts are easy to write and I use them for fixing issues or for simple single responsibility tasks.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by hot_gril on Monday March 13, 2023 @07:14PM (#35144571)
> The problem is that our industry democratized to the point that people want to hire the cheapest resources who can ship something

This doesn't sound like a problem to me.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by hot_gril on Saturday March 11, 2023 @09:19PM (#35115688)
Doing high-level stuff in those languages takes longer even if you know what you're doing, though.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by invalidname on Saturday March 11, 2023 @02:00PM (#35111408)
JVM can do 1M concurrent on a single machine. Don't lump it with Ruby please. This will be even easier with Loom as we won't need asynchronous APIs.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by mdaniel on Saturday March 11, 2023 @12:33PM (#35110558)
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by Atlas22 on Saturday March 11, 2023 @12:35PM (#35110583)
Just gotta wait for the comment claiming admin/maintenance costs makes non-cloud more expensive than cloud then we can have yet another full repeat discussion.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by elijaht on Saturday March 11, 2023 @12:54PM (#35110770)
Yea, I thought that was weird. Was wondering why there weren’t many comments when it had been on the front page for so long
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by ramesh31 on Saturday March 11, 2023 @12:39PM (#35110617)
This fits Ahrefs because computation and data is their secret sauce, not software. For the average SaaS this isn't the case.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by twelve40 on Saturday March 11, 2023 @12:45PM (#35110705)
great point, i didn't even realize how much compute power they need (and how much money they make)
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by PaywallBuster on Saturday March 11, 2023 @12:54PM (#35110768)
right!

Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem

AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances

but, the "AWS premium" becomes a problem: - 10x when dealing with bandwidth which you cannot do much except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives

The way I see it, AWS is great but you need to consider your 1) key infra requirements 2) consider cost effectiveness of different solutions/services 3) consider alternatives i.e.

- Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best! - Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective)

Consider 1) what you need to get started and ship 2) what you need to scale up 3) what's the most cost effective solution long term

The requirements will change over time, be ready to adapt!

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by marcopicentini on Saturday March 11, 2023 @12:40PM (#35110630)
Me too! Running a B2B SaaS using Digital Ocean and Cloud66.

I am free to switch vendor every 12 months without pain.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by intev on Saturday March 11, 2023 @12:50PM (#35110739)
How has the experience been with Cloud66? Why them over using something like caprover?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by marcopicentini on Sunday March 12, 2023 @01:20PM (#35122989)
I don’t know about caprover.

I like Cloud66 because:

- Pricing is fair. I spend 100$/month for my stack - They back up db and host it to a different cloud provider (AWS)

- It’s easy to import Db

- UI is clean and easy to use

- It’s easy to switch vendor. Just clone the app and it will install all in a new server.

I have switched from Digital Ocean to Azure in a couple of hours with 30 min downtime.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by codingbot3000 on Saturday March 11, 2023 @12:50PM (#35110740)
It looks clear to me that given the hardware they need, and the scale they operate at, buying their own metal makes sense. Also them sticking to similar servers is a good idea imho.

What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do.

Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by pphysch on Saturday March 11, 2023 @01:09PM (#35110908)
A small-medium DC can be a lot of work to install, specifically in terms of backup power/cooling, but not much to maintain on a daily basis.

Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required.

Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by indymike on Saturday March 11, 2023 @02:24PM (#35111702)
I've had great luck just leasing what I need from a company who already has the DC and is willing to manage the hardware as part of the lease.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by kkielhofner on Sunday March 12, 2023 @01:49AM (#35117477)
Building a DC is yet another level. Co-location facilities drop redundant power in your cabinet and manage cooling. All you need to do is make sure you plug your redundant power supplies into the right PDU. Also, any decent co-lo provider will rack and provide remote access to HW.

For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute.

For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate).

I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them.

That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by tatersolid on Sunday March 12, 2023 @08:24AM (#35119268)
Remote hands, maintenance contracts, and employees who will do network/firmware/firewall config, patching and maintenance aren’t included.

Also there’s no software costs… if you’re going to compare with the could you need to include VMware/Microsoft suite licenses to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by kkielhofner on Sunday March 12, 2023 @10:40AM (#35120562)
Remote hands from co-lo facilities are typically free for simple activities like hitting a switch, plugging something in, etc. Hourly rates are extremely reasonable and initial racking is often included. Reality is if you’re using remote hands after the initial install you’re doing it wrong - out of band management, PXE boot to full machine bootstrap via ansible, etc isn’t any more complicated than managing a bunch of VPCs, cloud firewalls, routing, cloud templates/terraform, etc.

Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light.

AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective.

VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely.

I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash.

The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left for at least one additional FTE to manage the architecture (if it even comes to that). Cloud egress bandwidth pricing markups alone can pay for a FTE when you’re pushing the kind of traffic that would warrant this approach.

I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”).

Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud spend.

Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cottage industry of consultants and startups that exist solely to help orgs try to understand their cloud spend and attempt to optimize cost. Hardware leasing, fixed price co-lo, power, and bandwidth results in cost controls that just aren’t possible with cloud. Almost every startup has at least one story of scrambling to figure out what happened when they blow through their monthly cloud budget in many cases literally overnight - or begging their cloud provider to write off or steeply discount a monthly invoice that was wildly beyond predicted cost.

In multiple rounds of investment, etc in startups I’ve often had the experience during technical due-diligence where they look at hosting costs and literally can’t believe how cost effective and performant it is vs the “all cloud always” companies in their portfolio.

At a level of scale to necessitate it hardware and co-lo is practically cheating.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by hdjjhhvvhga on Saturday March 11, 2023 @01:24PM (#35111040)
> What I am missing though in the monthly cost is the personnel cost.

It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by kevincox on Sunday March 12, 2023 @07:08AM (#35118744)
I'm pretty confident that at small scale AWS requires less operating costs that colo. Things as simple as s3 take lots of work to setup and run in a colo (well probably at least 3 locations if you want really good durability) and things like backups quickly build on that. Even a single computer will have occasional hardware failures, system installs and more. You usually can't restore an entire disk from backup like you can on AWS if a DB upgrade goes wrong.

But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service.

It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are.

As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by drunkenmagician on Saturday March 11, 2023 @07:12PM (#35114454)
Nope, senior devops / sys admin folk here in Singapore are more in the 7K - 8.5K (usd) range. Unless of course the employer is paying FFANG level salaries
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by aldarisbm on Saturday March 11, 2023 @12:54PM (#35110772)
> So we decided to consolidate all the costs spent for this data center, divide them by the number of servers, and compare them to the cost of a similar installation in the Amazon Web Services (AWS) cloud (we use this as the leading IaaS example).

so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity.

Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by margorczynski on Saturday March 11, 2023 @01:10PM (#35110913)
Once your operation is stable and you know (almost) exactly what you need then sticking to the Cloud makes less and less sense. The premium they charge is enormous.

Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by marcinzm on Saturday March 11, 2023 @01:29PM (#35111096)
What the cloud gives you is the removal of the inter-department conflict between IT and Product Engineering. This is especially bad since IT is a cost-center rather than a revenue generator. Having a single provider who in turn has a single client is a horrible dynamic. It can work for the short term but long term you get really odd dynamics. Sure the cloud is expensive but so is the aggregate opportunity cost of waiting 3 months and spending 20 hours in meetings for a single dev server.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by trollied on Saturday March 11, 2023 @01:33PM (#35111146)
This quite often comes down to CapEx vs OpEx.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by kkielhofner on Sunday March 12, 2023 @01:35AM (#35117408)
Hardware leasing/equipment financing is OpEx. The difference vs cloud is you’ll get much more capability at a fraction of the cost and own an asset at the end.

I talk about it a lot on HN but if you really want to get the financial types excited about leasing there are all kinds of interesting options like Section 179 Qualified Financing that can often sweeten the pot further.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by yegle on Saturday March 11, 2023 @01:49PM (#35111305)
A 24/7 oncall rotation team needs 5 people to keep it <1 rotation per month. This alone costs 1M/y.

I think many companies simply get away by having unhealthy oncall rotations. E.g. ask people to oncall outside business hours constantly.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by cbrozefsky on Saturday March 11, 2023 @01:55PM (#35111364)
Or their stuff doesn't break such that being oncall is a formality.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by pgwhalen on Saturday March 11, 2023 @05:19PM (#35113378)
This is a great way to put it, and I don’t see this reality acknowledged on HN very often.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by MrFoof on Saturday March 11, 2023 @06:17PM (#35113812)
>” Or maybe they have some special agreement with much lower prices than market.”

These exist and falls under the “Enterprise Discount Program”, but there are other PPAs (Private Pricing Addendum).

The idea with EDP is if you have qualifying on-demand use that not covered by a Savings Plan or Reserved Instance, you instead get the EDP discount applied to the line item.

Where I work is actively renegotiating its EDP discount. Right now we get 14% off for applicable on-demand usage.

There are also other programs. Such as the “Amazon Partner Network” “Workload Migration Program” (APN WMP) for SAAS companies, which are credits (lump sum, not per line item) that are awarded for moving customers to your SAAS products that are hosted on AWS infrastructure. These are big 6-digit chunks for us. Many other credits and programs as well.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by andrewstuart on Saturday March 11, 2023 @06:42PM (#35114006)
There a fantasy that says you don’t need experts to run your cloud infrastructure.

The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host.

Cloud is crazy expensive and delivers much slower computing power per dollar.

Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by AtlasBarfed on Saturday March 11, 2023 @08:56PM (#35115510)
It's bottom lines like the article's where I wonder if all the companies who had HORRID penny-wise pound-foolish (6 months of paperwork to get a server, and only then would they order the servers) and went cloud simply did what these guys appear to be doing:

buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by lamontcg on Saturday March 11, 2023 @10:52PM (#35116437)
I sold all my AMZN stock by 2006 when I quit thinking that everyone would figure out the secret to doing your own hardware cheap.

I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money.

It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella.

If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco.

It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs.

At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all.

And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe...

But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem.

And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by not_me_ever on Monday March 13, 2023 @04:16AM (#35132627)
How can you save >$150M/year on a service that should cost <$2M/year to run?