Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard.
So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.
We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.
I've always struggled estimating my DB size, curious what you mean by your estimate.
A snapshot is effectively identical to a killed database server (due to OOM, or power outage), and database servers should be crash-resistant.
With PostgreSQL, checkpointing the database before taking the snapshot may make the crash recovery quicker when loading the snapshotted database.
Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.
I noticed you mention Azure BS in your RFCs as a potential backend. Have you done much work towards that yet?
However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/
This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy.
Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only.
So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.
Neon is 100% compatible from Postgres b/c we didn't (or almost didn't) change the Postgres engine.
CockroachDB already does true scale-to-zero if that's your requirement: https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-...
Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet.
Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.
If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers.
The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.
1 - Is not an actual relational DB
2- Doesn't really scale to zero
Planetscale does scale to zero, but has a ridiculous billing model.
Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.
Neon seems like a vast vast improvement & great & desperately needed potential leap for mankind.
> "great & desperately needed potential leap for mankind"
Are you being serious? That's very hyperbolic if so.
Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much.
Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them.
Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data.
We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]:
> This is the missing piece on cloud for masses
> "But ones that are not ours, that we cant hack on, that we cant run ourselves."
Who is "we"? Is your entire issue that not everything is open-source?
> "these are not humanity's heritdge & humanity is cut off from active participation with them."
This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking
The most notable feature of Neon, that makes it so exceptionally smart, is that it takes the humanity's existing first best choice default-go-to database, by a country mile, and layers in really smart decoupling of storage to help it scale. Reinventing things from scratch can be a win, but that this is already a well known quantity, much loved & cherished & used by all, hacked by all, grown by all is a planet-sized huge plus mark.
I think you vastly overrate the broad availability of alternatives to this, of potential starting places others might explore. Most of your list is dominated by proprietary whatever. Even skipping that, yes, there is a ton of novelty & promise, the potential for breakthroughs. But this is starting with the best, and pushing it into the cloud-layer, into the troposphere.
There are dozens of "scalable postgres" alternatives from Redshift to Citus to CockroachDB to Yugabyte. I find it strange that you say its overrating the availability when you're also comparing it to something that isn't even a full product yet.
Thank you for the comment rektide. We of course would love to see constructive comments and criticism.
- Neon CEO
I appreciate your concern for the integrity of HN discussions, but the thing to do if you suspect abuse or manipulation is to email [email protected] so we can look into it. Posting about it in the threads themselves is explicitly against the rules.
but please take a look at the flagging attack on all of my comments outside this thread. the timing is too peculiar here
somebody really didnt like what i wrote above ;)
Cockroach DB license - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/LICENS...
Mongo license - https://github.com/mongodb/mongo/blob/39e4b70/LICENSE-Commun...
IIRC, backup was enterprise but is now part of the core. However, restore is still enterprise.
Given how much performance you can squeeze out of a $5/month VPS (I've been spinning them up and indeed down regularly over the last couple of years), is this really a paradigm shift?
I'm an (former) SRE and run my own Kubernetes cluster for fun, but still use serverless (containers as a service, static website hosting) depending on the project.
Don't you need that as well as cloud-specific knowledge if you go serverless?
> and ideally more than that to know not to do stupid things like chmod 777 and database exposed on the public internet.
You still need some arcane knowledge to make sure your serverless doesn't experience cost overruns, right?
IME (your's obviously differs), the amount of cloud-specific + vendor-specific knowledge needed to avoid using a $5/m VM is a lot more in volume and a lot less in stability[1] than learning basic Linux once and using VMs everywhere[2].
[1] How the different cloud providers bill, when they bill, how to control your limits, etc changes much more often than knowing how to keep your server patched. Knowing how to get your serverless DB going on AWS doesn't help when you want to use Azure. And each cloud vendor regularly requires you to update your knowledge. Knowing how to keep a PostgreSQL-on-Linux up-to-date can be learned once and used for years. Even if running a managed DB, you'll still need to gain some of that knowledge anyway.
[2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
I know people who have literally written their own analysis tooling just to figure out what's impacting their AWS spend. It's gotten better, but I could have retired many times over on what I've seen clients overpay to cloud providers because they didn't understand what would drive cost.
> [2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
The "problem" with the cloud story is that at that point you also have a team that could save you a fortune with a hybrid setup. Cloud providers get their margins off those who don't understand how much they're overpaying or who are too small to a) care or b) have leverage. Those big enough to have leverage who understand either negotiate hefty discounts or build out cheaper setups (basically at the point you're spending 7 figures a year, if you're paying anywhere near list prices for cloud services you're a chump; below that it's hit and miss)
I'm not at all against using cloud services, but I wish more people actually understood their costs and picked based on merits rather than cargo-culting. Some teams benefit greatly from cloud services, but usually if they're not cost sensitive. In my current job we have everything on AWS because we're never going to scale to somewhere where it'll get expensive and it's convenient. We'd save money if I moved it to, say, Hetzner, but the hosting bill is too small to matter. For that use it's fine.
The moment the bill starts to bite people ought to at least price out alternatives, and consider hybrid setups. E.g. I've had setups where even just putting a caching proxy in front of AWS to cache images to cut the egress bill would have paid for a team to keep it running. Their egress cost is still bad, but at the time it was just pure highway robbery.
I've done SRE/devops work in various capacities including consulting longer than cloud services have existed, and my experience is that I've consistently earned more from clients who insisted on cloud services because they consistently need more help. Nothing is driving more demand for devops consulting services than cloud providers.
This is not hard to get right. Yes, you need to learn how to do it, but the amount of money I've made from clients who thought cloud was simple and proceeded to create massive security holes for themselves is fairly substantial. People who think they're reducing their attack surface by using these services need to reevaluate - they're large, complex architectures that very few users understand properly. You need to learn the skills either way.
I appreciate there are indeed billions of people for whom $5 is a lot of money, but just how many of them are "students and enthusiasts" itching to get started with Postgres?
I realise that - perhaps particularly here - a $5/month VPS is a deeply unsexy thing. You can, however, achieve (and learn) an awful lot with one.
Some links and discussions:
- https://cloudirregular.substack.com/p/please-fix-the-aws-fre...
Lambda has many limitations.
In particular, for some reason AWS is allergic to providing a container deployment service that actually scales to zero.
Isn't this what Fargate is?
Is there any cold start delay for neon?
I like this perspective a lot & think it's absolutely key here.
We- the world- still pick single-node writer postgres & read replicas when we have to store & query data. There's great Kubernetes postgres operators, but it's still a distinctly pre-cloud pre-scale type of technology, & this decoupling & shared-storage sounds ultra promising, allows independent & radicaly scale up & scale down, sounds principally much more managable.
Or for try-out apps, as you mention, you could just run Postgres next to your app in the same container.
This might be possible with fly.io, or will soon, I think.
I’m not sure how comfortable I am using a custom flavor of Postgres (even if it’s just the storage layer).
The problem with most other "serverless" databases is that they don't offer HTTP API to query them from restricted environments like serverless functions.
https://ldoughty.com/2022/05/exploring-aws-aurora-serverless...
Short answer if you don't want to read my post: it constantly uses CPU, it's always on. After creation, waiting 2 days, never logged into it, never ran a script against it, never gave it access to any networks, minimum cost is $43/month because it can't actually scale down to 0.5 units unless you CAP it at 0.5, which makes it unusable, because it consumes all of that capacity just to exist.
It sounds like this Neon offering is exactly what I hoped AWS was offering... Or they are using Language to suggest it and mislead the customer just the same ... If it's the former, if probably sign up and try it out. If it's the latter, I'll probably never touch it for the false hope.
Edit: lots of typos from phone keyboard
It's sitting at 23% of 0.5 ACU.
So its either the replication setting (haven't tested with it on yet), or.. AWS is a shared service... wonder if it's similar to EC2, in that sometimes you get an instance on a machine that's more overloaded and the instance doesn't perform as well.. and you have to destroy it and try again. Might want to try it again.
Edit: I don't think its the replication setting... tried that with a new db and its at 25% on each replica after an hour.
> It automatically starts up, shuts down, and scales capacity up or down based on your application's needs.
The only tradeoff is the additional latency someone will have when connecting to the db after it has shutdown and waiting for it to spin back up and become ready.
> You pay only for the capacity your application consumes.
> Scales down to 0.5
But it actually can't scale down to 0.5 or the DB falls over just existing.. auto scaling won't let you go down that low unless you set 0.5 as the max, which literally makes it not scale up, and it's dead, because the DB can't run with that little CPU.
So it's fair to ask if neon can scale to 0, both in marketing, and in practice.
It still has a significant latency on the first connection attempt after suspend (1-2 seconds), but we are working on that and it seems to be realistic to put the startup time under 1 sec.
Pricing model is still work-in-progress, so cannot say much about it. Yet, my personal intention is to make it cost-effective for both end-user and us. I'd prefer to don't build a service with claims like 'here is your free-tier serverless Postgres with zero-latency on connect', which actually means that under the hood there is an always-running compute burning the investors money. Hope it's realistic to achieve :)
-- Cloud engineer @ Neon
Looking forward to seeing how this works out. I have no issues paying for services, I just hate that the minimum entry level cost is $20... I can't imagine why, at scale, it can't be more affordable for hobby/fun level projects.
I tried fast booting of PostgreSQL instances and it always took multiple seconds. So i am really curious!
Basically, to start we need to know this LSN and to bootstrap the Postgres processes. This is really that quick. After that compute is ready to accept connections and serve requests, as it's able to get any missing pages from pageserver with GetPage@LSN request.
We do have the whole bunch of problems to solve: queries latency after cold start; startup after the unexpected exit of the heavily loaded Postgres instance could be slower; etc.
- Initializing shared memory -> We, for now, have only small instances, so that doesn't hit us as hard
- Reading data directories -> We don't have to do that at all
- Replaying WAL from a previous unclean shutdown -> We don't need to do that, PageServer is responsible for that
- When initializing a whole new database: Initializing the data directory -> We have a copy that each instance gets initialized from, which makes the process "copy those ~16MB in the background", which saves us from having to do the costly initialization process.
And there's several more infrastructural optimizations, such as pre-loading the docker images onto the hosts.
All serverless, scale-to-zero or pay for demand...
Scale doesn't matter for mini applications and scaling vertically (=throw money for a bigger server) will work for 99% of the companies. The 1% who need horizontal scaling will have custom everything regardless and will need to hire experts, not a good niche to release a product.
These are reasons why one might go for a hosted database at AWS/GCE/Azure. There are tons of good servers out there for small to medium projects and I don't think this service is right for those, unless they can make do with the free tier. The real benefit is in the larger cloud application space.
A system that does this type of scaling automatically while also reducing the dependency on a single cloud provider's service can be a gamechanger for some companies with huge database servers that risk getting locked in.
I think using this service for most existing applications will introduce a performance drop, a rise in expenses, and a complicated migration path, but on the other hand I think that developing against this system for new projects that are very likely to grow in scale will end up with some major data control benefits.
The open source nature also allows for competitors in markets like the EU to start serving databases that don't break privacy laws (although most companies don't care until they receive a fine).
I'm not sure how this company will become profitable while giving away its special sauce that can be modified to run at competing companies relatively easily, but that's a whole different story.
This opens up a world of try-out mini applications that cost cents to host. serverless db (postgres) + serverless compute (cloud-run) + use as you go storage+network. This is a paradigm-shift stack. Exciting days ahead.