News for hackers. Stuff that matters.
Re:Localstack – Local AWS Emulator by shiftpgdn on Sunday October 10, 2021 @01:46PM (#28820409)
I’ve been saying for a couple of years whoever builds a tool to one click deploy an AWS alike to run on local bare metal has built the next billion dollar a year company.

I know of a lot of big shops that are desperate to get out of the cloud due to 7-8 figure AWS bills but their software and engineering is too tightly wound into AWS tooling.

Re:Localstack – Local AWS Emulator by IceWreck on Sunday October 10, 2021 @02:12PM (#28820644)
But AWS is huge. They employ thousands of engineers. Granted most of the complexity is due to its huge huge scale, but still it has too many features to do what youre asking. And then there are AWS exclusives like DynamoDB.

Yes there are open source public/private cloud alternatives like OpenStack but they won't solve the problem of lock in and you'd still have to change all your tooling.

Re:Localstack – Local AWS Emulator by kortilla on Sunday October 10, 2021 @04:43PM (#28821804)
> they won't solve the problem of lock in

Conflating lock in to an open source tool with lock in to a vendor who you literally have to keep paying money to survive is a shitty tactic used by companies like Amazon.

Using open stack absolutely solves the problem of lock in from a business perspective. You can’t be cancelled, the price can’t be changed, etc.

Re:Localstack – Local AWS Emulator by NathanKP on Sunday October 10, 2021 @05:11PM (#28822017)
I legitimately wonder how many people feel free from lock in after they finish paying $75k to Canonical for a 12 node OpenStack "private cloud build" (https://assets.ubuntu.com/v1/d4766546-Private-Cloud-Build_Da...), not to mention ongoing support bills and billing for addons "starting at $25k".

Just because something is open source does not mean your org will actually have the expertise to be successful at running it yourself. That's how most "open source" projects like Kubernetes and OpenStack become so successful: the supporting orgs behind the project make lots of money charging companies for support and maintenance and issues that pop up with the setup.

There are many forms of lock in. Buying a big upfront investment that has to pay itself off for the next X years is a form of lock in. Paying for expensive professional consulting support to build out the system and keep the system up and running is a form of lock in. Or even worse, hiring or training expensive dedicated employees to learn the open source system and become the internal professional consultants for the rest of the org, that is also lock in.

Re:Localstack – Local AWS Emulator by kortilla on Sunday October 10, 2021 @05:36PM (#28822210)
> I legitimately wonder how many people feel free from lock in after they finish paying $75k to Canonical for a 12 node OpenStack "private cloud build"

Here’s the part where it’s not lock-in. We stopped paying for support from Redhat when we got big enough that our own SRE expertise to run our own open stack cloud.

You know what the impact was on our applications running on it? Absolutely nothing.

How much do you pay Canonical to use grep every month? Inexperienced developers have just been conditioned by cloud providers to think that an IaaS platform must cost something in payments per month to some tech company. It does not.

This is no different than Windows vs Linux on servers. I can’t wait until 20 years from now when all of the proprietary shit looks ridiculous in hindsight.

Re:Localstack – Local AWS Emulator by BoorishBears on Sunday October 10, 2021 @07:23PM (#28822870)
> to think that an IaaS platform must cost something in payments per month to some tech company. It does not.

I agree fully! As you explained, it can cost several full time employees and their payrolls and their HR and their management!

Edit: And before someone misses the point, I'm not saying the math never makes sense for in-house, but to me the inexperienced take is thinking either approach should obsolete the other...

Re:Localstack – Local AWS Emulator by grepfru_it on Sunday October 10, 2021 @08:13PM (#28823167)
Sounds like you’re pushing a hybrid cloud approach in your edit. I have been consulting on this topic for 7 years and companies are still tied to their data center or the cloud and can’t readily pull their components apart because they originally built them so intertwined with the cloud environment of the time
Re:Localstack – Local AWS Emulator by NathanKP on Tuesday October 12, 2021 @12:40PM (#28841825)
Even if it doesn't cost you in monthly payments to a tech company it still costs you in "our own SRE expertise to run our own open stack cloud" so salaries, benefits, 401k contributions, etc.

> I can’t wait until 20 years from now when all of the proprietary shit looks ridiculous in hindsight.

I'd posit that in 20 years there will still be tech which is more efficiently managed by a central provider, rather than having each company hiring their own independent expertise.

Re:Localstack – Local AWS Emulator by zimbatm on Sunday October 10, 2021 @05:39PM (#28822230)
I'm sure you can find other OpenStack professionals that are willing to manage your Ubuntu OpenStack deployment.

Of course every solution costs something. Lock-in means that there is no other choice, or that the cost to switch is so steep that it becomes prohibitive to do so.

Re:Localstack – Local AWS Emulator by grepfru_it on Sunday October 10, 2021 @08:15PM (#28823180)
Zerostack sounded like a turnkey openstack solution akin to VMware but I think the owners got stuck in the consulting grind and never pitched up for growth
Re:Localstack – Local AWS Emulator by temp_praneshp on Sunday October 10, 2021 @05:41PM (#28822238)
> 12 node OpenStack

I share your disdain(?) for openstack, but the 12 node count there is minimum I think (to deploy the control plane). You can likely add a reasonable number of hypervisors at no cost.

(I think I'm correct based on the table in the last page of your doc, but happy to be corrected).

Re:Localstack – Local AWS Emulator by jcims on Sunday October 10, 2021 @07:24PM (#28822876)
Everybody is focusing on what happens on top of the CPU, but very few people talk about what happens beneath it.

Datacenters are fucking expensive to build and run, buying network and compute hardware is a pain in the ass, maintaining sufficient capacity to sustain growth while operating on a 3-6 month deployment lead time is atrociously expensive, and that's without all of the financial calculation around depreciation/taxes/etc.

The fact that cloud providers are able to ball all of that shit up into a flat opex rate for a server is unbelievably attractive.

Re:Localstack – Local AWS Emulator by selfhoster11 on Monday October 11, 2021 @04:32AM (#28826093)
This. My company is transitioning to the cloud for this reason.
Re:Localstack – Local AWS Emulator by starfallg on Monday October 11, 2021 @05:35AM (#28826466)
Exactly, open source solves the problem with vendor lock-in in your tooling. It's more needed than ever now that the world is moving towards proprietary SaaS black boxes.
Re:Localstack – Local AWS Emulator by nostrebored on Sunday October 10, 2021 @06:40PM (#28822631)
It's the same thing. You're paying for engineers to work with a service or you're paying for a service.

See how your comment ages in five years with kubernetes.

I've absolutely seen disasters with open source that bleed companies dry. Openstack is a great example.

Re:Localstack – Local AWS Emulator by belter on Sunday October 10, 2021 @03:55PM (#28821442)
You can install DynamoDB locally although you will still miss the rest of the stack.

"Setting Up DynamoDB Local (Downloadable Version)" https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Re:Localstack – Local AWS Emulator by jdreaver on Sunday October 10, 2021 @04:03PM (#28821526)
https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

> The downloadable version of DynamoDB is intended for development and testing purposes only. By comparison, the DynamoDB web service is a managed service with scalability, availability, and durability features that make it ideal for production use.

Re:Localstack – Local AWS Emulator by belter on Sunday October 10, 2021 @04:05PM (#28821540)
Correct. I meant to use it to save some money while developing your apps, not as a solution for local deployment.
Re:Localstack – Local AWS Emulator by jdreaver on Sunday October 10, 2021 @04:11PM (#28821581)
Ah understood. I thought this comment thread was talking about replacing _production_ AWS use cases with a bare metal system that has a compatible API.
Re:Localstack – Local AWS Emulator by belter on Sunday October 10, 2021 @04:48PM (#28821844)
LocalStack says its about testing:

"...Whether you are testing complex CDK applications or Terraform configurations, or just beginning to learn about AWS services, LocalStack helps speed up and simplify your testing and development workflow..."

Re:Localstack – Local AWS Emulator by unethical_ban on Sunday October 10, 2021 @03:26PM (#28821187)
I think it would not be impossible for a talented, well-funded group to pick the top 20% of AWS's services and then implement them with the top 80% of features, with an API-compatible layer.

Even "just" to emulate a single region with 3 AZs would be useful for testing and for prod for many people. I know of several large organizations that to this day haven't scaled their CI beyond one region.

Re:Localstack – Local AWS Emulator by GordonS on Monday October 11, 2021 @03:52AM (#28825876)
This makes sense; I would guess that 90% of AWS customers are only using the top 10% of available AWS features.

Same deal with Azure and GCP, although note that Azure has Azure Stack, which is a self-hosted version of Azure (I guess it's still expensive though, and I don't know how many services it includes).

Re:Localstack – Local AWS Emulator by sekh60 on Sunday October 10, 2021 @02:28PM (#28820793)
OpenStack isn't a redhat project, it is contributed to by a large number of players. Redhat, SUSE and Canonical all provide their own OpenStack service subscriptions, but one can easily deploy OpenStack with a project like OpenStack Kolla. I run it at home and it is great.
Re:Localstack – Local AWS Emulator by Sanguinaire on Sunday October 10, 2021 @04:41PM (#28821783)
OpenStack is an over-complicated mess. Most enterprise IT orgs already struggle to hire the people with skills to operate a public cloud account, nevermind having to build the underlying infrastructure as well.
Re:Localstack – Local AWS Emulator by sekh60 on Sunday October 10, 2021 @07:42PM (#28822987)
I suppose it is complicated, just a homelabber here, but I don't think it's that bad. I deployed my cluster at home using kolla, and I've had relatively few hiccups that non-sysadmin/non-coder me couldn't get around. I've had the same cluster going for a few openstack releases now. Kolla really makes it easy.

That said before I found Kolla I did deploy openstack manually a few times before messing things up and having to restart.

Re:Localstack – Local AWS Emulator by Spivak on Sunday October 10, 2021 @03:23PM (#28821162)
I mean the value is really implementing AWS’s API surface area in a single control-plane overtop of the usual on-prem tooling. K8s is kinda getting there but is too narrow in scope I think for what people actually want.
Re:Localstack – Local AWS Emulator by thayne on Sunday October 10, 2021 @05:01PM (#28821942)
For DynamoDB specifically, ScyllaDB had a Dynamodb compatible mode. There are several options (open source and otherwise) with s3 compatible APIs. I think it is more likely we'll end up with replacements for individual services and maybe at some point it will be possible to glue them together as a replacement for a significant subset of AWS
Re:Localstack – Local AWS Emulator by dimitrios1 on Sunday October 10, 2021 @09:33PM (#28823671)
What's so exclusive about a half-baked Dynamo implementation? Dynamo is just a worse Cassandra. DynamoDB itself doesn't even implement the Dynamo Paper.
Re:Localstack – Local AWS Emulator by Aeolun on Sunday October 10, 2021 @07:09PM (#28822788)
Not necessarily if your end goal is to be a unicorn. You can get a lot of investment for that.
Re:Localstack – Local AWS Emulator by simonh on Sunday October 10, 2021 @02:50PM (#28820946)
I worked on a system called Quartz at Bank of America. It wasn't an AWS clone, but was targeting the same flexibility for the development and deployment of applications on 'programmable' infrastructure. The team that built it had done similar projects before at J. P. Morgan and Goldmans. It was a set of infrastructure and services all accessed through Python APIs, with YAML config, all stored in a single distributed version control system. It was a lot of fun to work on.
Re:Localstack – Local AWS Emulator by magmasystems on Monday October 11, 2021 @08:16AM (#28827462)
Wow, my brother works on Quartz. But I think that half of BofA does. Higgins and Kirat formed their own company to bring the Quartz/Athena paradigm to the masses.
Re:Localstack – Local AWS Emulator by no_wizard on Sunday October 10, 2021 @07:47PM (#28823021)
I would not be shocked if they just had some abstractions built around Ansible[0], it definitely fits the description

[0]: https://docs.ansible.com/ansible/latest/user_guide/playbooks... Intro to playbooks - Ansible Documentation

Re:Localstack – Local AWS Emulator by simonh on Monday October 11, 2021 @03:33AM (#28825769)
It’s very different, much more like AWS or Azure. The underlying infra is there, you just allocate resources using config and APIs. For example there’s a server farm for running server applications such as batch processing or web applications. To stand up a new application you develop the app on a local instance of the Quartz Python runtime, check it into version control, then to deploy it by configuring a YAML file in version control that defines which server farm it runs on, the path to the code in VC, the schedule and various resource limits. That’s about it. Job done.
Re:Localstack – Local AWS Emulator by whummer on Sunday October 10, 2021 @03:25PM (#28821177)
Wow that sounds super interesting! If you can shoot us a message to [email protected] we'd love to chat and learn more about your journey!
Re:Localstack – Local AWS Emulator by 1vuio0pswjnm7 on Sunday October 10, 2021 @05:38PM (#28822225)
Need more details. Matching AWS service for service would be overkill I imagine, so what services does this need to have. Is hardware virtualisation is must-have.

The way I made systems/applications before AWS existed was to create bootable images. I could already use a hypervisor, Xen, because the OS fully supported both guest and host, all virtualisation modes, before Linux did, and before AWS existed. But because I am not a hosting provider I saw little need for virtualisation. The OS I used also had "unikernel" capability, before Docker, etc. existed. As it happened, this inexpensive, self-determination was not to be the future, instead we got "the cloud". Sharing servers with other "tenants". Less expensive for the hosting provider, but more expensive and more limited for the subscriber. No argument, the "limited" part has improved since then, but it is still expensive (for continuous use that is, spot pricing was a neat benefit of "the cloud").

Anyway, I "deploy" images to "local bare metal" which is a laptop, RPi, or some other smaller form factor. I can use unikernel for some kernel drivers in userspace. Basic filesystem is embedded in the kernel, and larger filesystem is on the USB stick filesystem in compressed format. Updates are easy enough. I put two kernels on the USB stick, one is the running kernel and the other is the update kernel. Same for the larger filesystem which may contain servers and configuration files. Can update or go back to last working kernel/configuration by selecting one or the other in the boot menu/renaming the larger compressed filesystem.

Here is someone running a search engine out of his living room. AKAIK, his setup survived a sustained HN thundering herd, hug of death without a hiccup.

https://news.ycombinator.com/item?id=28552805

The expense of AWS is an obvious point of discussion but another one not mentioned here is control. When I create images for bare metal I do not need to jump through any hoops as I would in order to create an image that will run on AWS. Nor do I need to fiddle with all the AWS knobs. There are no silly marketing names for every program I run. I know the system I am creating as well as I know the OS and the software I choose. That is much better than how well I know every aspect of AWS which just gets more and more complex every year. AWS documentation is as cringeworthy as it it is voluminous. The ever-increasing complexity of AWS, including the "tooling", is, IMO, How To Create Lock-In 101

Re:Localstack – Local AWS Emulator by galaxyLogic on Monday October 11, 2021 @01:14AM (#28824923)
> The ever-increasing complexity of AWS, including the "tooling", is, IMO, How To Create Lock-In 101

That is an interesting point. Complexity creates lock-in. Why? Because when you are interacting with a complex system you depend on it working in the complex ways it does. It is unlikely that anybody else could duplicate those features of the AWS your application is depending on.

This all runs counter to the idea of "encapsulation". You should be able to use a system via a well-defined interface. Once the interface is well-defined, other providers can provide their own implementation of the same interface.

So, AWS is basically bad software engineering, lacking encapsulation?

Re:Localstack – Local AWS Emulator by pdimitar on Sunday October 10, 2021 @06:38PM (#28822625)
I would absolutely love reading you blogging in details about this. It's a lost art, and one I've ever been good at. I want to lately, but people like you are always undercover and don't document their knowledge. :(
Re:Localstack – Local AWS Emulator by ragona on Sunday October 10, 2021 @05:27PM (#28822149)
I think this misunderstands the hard part of running AWS. You're paying for the software, sure. What you're really paying for is the hundreds of oncall engineers supporting you at all times, and the culture of ensuring that those services keep working.
Re:Localstack – Local AWS Emulator by kevinsundar on Sunday October 10, 2021 @05:38PM (#28822224)
As an Amazon engineer who frequently goes oncall, this is Amazon’s true competitive advantage.

The company works because every night there’s 1000s of engineers sleeping with a pager by their head.

Re:Localstack – Local AWS Emulator by devit on Sunday October 10, 2021 @06:15PM (#28822483)
Why do that for general customer support instead of hiring in different time zones?
Re:Localstack – Local AWS Emulator by kevinsundar on Tuesday October 12, 2021 @12:29AM (#28836064)
Because minutes after a metric drops that indicates the possibility of an issue an engineer who probably wrote the code or has the code checked out on their laptop and is intimately familiar with it working on the problem. No waiting for a support team to triage, find the owner, and get ahold of them.
Re:Localstack – Local AWS Emulator by hansel_der on Monday October 11, 2021 @03:39AM (#28825808)
cultural cohesion?

we have often delayed calls because it's often hard to explain it to some "foreigner"

Re:Localstack – Local AWS Emulator by asim on Sunday October 10, 2021 @04:08PM (#28821552)
Agreed. Although it's a big ask. I have something that's slowly going in that direction but it'll be a while before it gets the interest I'm looking for.

https://github.com/micro

Re:Localstack – Local AWS Emulator by TonyCoffman on Sunday October 10, 2021 @08:22PM (#28823233)
Eucalyptus built a clone. It wasn’t one click but it also wasn’t hard to deploy. It really never made much of a dent.

I’m pretty sure it ended up where all software goes to die: HP.

Re:Localstack – Local AWS Emulator by rzzzt on Monday October 11, 2021 @01:21AM (#28824962)
The website is still up and points to a fork on GitHub:

- https://docs.eucalyptus.cloud/eucalyptus/5/install_guide/int...

- https://github.com/corymbia/eucalyptus/

There are quite a few moving parts. I think I got stuck around just comprehending the networking bits.

Re:Localstack – Local AWS Emulator by SkyRocknRoll on Monday October 11, 2021 @01:19AM (#28824950)
Nutanix is strong contender for this use case. We see lot of workloads moving from Cloud to onprem on nutanix stack. P.S I work on engineering side in nutanix.

https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...

Re:Localstack – Local AWS Emulator by rkeene2 on Sunday October 10, 2021 @03:41PM (#28821322)
I built this a few times. The most current implementation is called CloudSeed, though I am trying to fundraise for an open source version (with a RedHat-like business model of support).

CloudSeed has been a key part of several >$1bn contracts for KnightPoint/Perspecta/Peraton, most recently DHS DCCO ($2.7bn).

Re:Localstack – Local AWS Emulator by salex89 on Sunday October 10, 2021 @02:22PM (#28820733)
Ever looked into Nutanix (https://www.nutanix.com/)?
Re:Localstack – Local AWS Emulator by SteveNuts on Sunday October 10, 2021 @07:27PM (#28822896)
Nutanix is getting pretty good at this. My enterprise style company switched from AWS to nutanix this year, we bought three all nvme/ssd nodes with 1TB memory in each.

We were heavy EC2/RDS users, now we run their NX whitebox nodes with their KVM based hypervisor, and their database management platform "Era".

Everything is one click for deployment and upgrading of things, down to stuff like upgrading the SSD firmware, NIC firmware, etc. It auto migrates the VMs around hot, and does one node at a time. They also have a kubernetes platform called Karbon that seems to be pretty good.

Re:Localstack – Local AWS Emulator by phendrenad2 on Monday October 11, 2021 @02:29PM (#28831325)
But why? Because they can run an on-prem production "AWS"? Or because it's easier for testing? If the former, well, sure, but why stop there? Why not an easy on-prem Heroku clone? If the latter, I don't think that goes far enough. Simply emulating AWS (as LocalStack does) is barely usable for testing. You really would want a GUI that lets you see everything in the LocalStack config and data stores. Running a fake DynamoDB is okay, but it's painful if you need to use DynamoDB queries to see the data.
Re:Localstack – Local AWS Emulator by randmeerkat on Monday October 11, 2021 @08:24PM (#28834613)
So are you saying that the tooling needs to deploy to their datacenters or AWS, or just deploying to their own datacenters needs to be as simple as using AWS? If it’s both, use Kubernetes. If it’s deploying to their own datacenters, then maybe a new product could disrupt that space. You would still need to have a team competent in datacenter operations and hardware procurement though.
Re:Localstack – Local AWS Emulator by kongin on Sunday October 10, 2021 @05:23PM (#28822118)
It's kind of stunning that starting with fully open technologies we build another closed mainframe on top of it.

All that's old is new again.

Re:Localstack – Local AWS Emulator by whummer on Sunday October 10, 2021 @05:48PM (#28822286)
Agreed, that's really stunning - seen the analogy of cloud as the "modern mainframe" come up more and more recently.

I guess the (business) reason is that the centralized computing model enables enormous economies of scale. (Makes perfect sense from an operations point of view.)

What it seems to miss in the current incarnation/iteration, though, is the "power of distribution" - leveraging the fact that local compute capabilities (especially during development/integration) can help reduce the cognitive load, increase the efficiency, and thereby reduce overall costs.

There seems to be a tendency to focus mostly on the operational aspects, rather than the overall end-to-end developer journey. What remains to be seen is whether future iterations of "the cloud" will do a better job of embracing the power of distributed and/or hybrid.

Re:Localstack – Local AWS Emulator by kongin on Sunday October 10, 2021 @10:03PM (#28823863)
The reason why AWS got popular everywhere I worked was that you didn't need to get buying new hardware past ops. You just spun up your own, then when it was supporting half the business you pointed at it and said "Gee wouldn't it be nice if we had a box in our own data center to run it?" then there'd be a fire lit under the ass of ops and you'd get your computers in a week instead of next financial year.

Now it's exactly the same thing in AWS. My next guess is that you're going to be running production code on fleets of devs computers because you don't have to get extra budget for AWS next financial year to afford spinning up another instance.