Many teams would prefer a postgresql compatible database with full sql support without compromises, missing features, etc. So, this could break that market open a little. Both AWS and Google are unreasonably expensive for this stuff. Most teams don't need a huge database and would be able to run postgresql on a tiny instance and get away with it. Have 2 of those and failover and backups and it's good enough for a lot of small shops.

Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard.

So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.

We have a single big bare-metal machine. We run Postgres with a ~1TB DB, moderate load, on a Hetzner AX101 (16C/128GB ram). It has 2* 3.84TB nvme drives (zfs mirror with hourly snapshots) used for postgres storage only, and a seperate pair of mirrored sata drives for system/boot (had to request the extra drive, ask support to change boot option in BIOS, and reinstall OS using rescue system). It's about 100 EUR/mo with unlimited data. We bounce all incoming requests from clients (the machine also runs a node backend) through a digital ocean machine (NGINX proxy), as their peering agreements are better, without this some users in Brazil, Turkey etc. have very slow access. OVH I think would be even better for this use (better? peering and IIRC cheaper data). ZFS snapshots are backed up with sanoid to a machine under my desk with spinning disks. AX101 can be fitted with up six 3.84TB drives, that's almost 12TB of mirrored storage, we should be good for a while. You can (should) use at least lz4 compression on zfs.. can consider zstd-1, bit slower, that could double the effective space. The compression also applies to the in-ram ZFS cache, that can be 100GB+.

We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.

How do you measure the 1 TB db estimate? Is that in index, as a CSV, or back of the napkin math based on estimates?

I've always struggled estimating my DB size, curious what you mean by your estimate.

Sounds neat. What’s the weak link here though?

No read replicas that can be promoted to master in an outage?

Ah, nop. If it goes down, we'll figure it out.. 24hrs outage wouldn't be the end of the world. Would have been nice to have a second (read replica) machine in East Asia, to reduce latency for users there, but didn't find a provider like Hetzner.. maybe could take a server in a suitcase and install it in a collocation center there. Bit of a hassle though.

I have a similar setup as OP, and I use ZFS snapshots to be able to quickly rebuild the whole machine on another host. Of course, it still requires manual intervention in an outage (it could be automated, theoretically), and I may lose up to 5 minutes of data due to my snapshot schedule.

Maybe that zfs snaphot under a running database? Is that ok?

Yes, ZFS snapshots are atomic, so as long as the database use sane semantics to do disk I/O, you're fine (AFAIK, PostgreSQL and MySQL/InnoDB are ok, MySQL/MyISAM is not though).

A snapshot is effectively identical to a killed database server (due to OOM, or power outage), and database servers should be crash-resistant.

With PostgreSQL, checkpointing the database before taking the snapshot may make the crash recovery quicker when loading the snapshotted database.

Hi Mani! For sure there are many serverless options - fewer that separated storage and compute and fewer that are open source end-to-end. Neon is also 100% compatible with Postgres (unlike CockroachDB) because compute is Postgres.

Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.

You should update your HN profile :). Neon looks really great. Far more interesting to me and my team than S2.

I noticed you mention Azure BS in your RFCs as a potential backend. Have you done much work towards that yet?

For now we've focused on making the product production-ready on AWS, so that we can get users to try it out. Once the model has been proven we'll likely branch out to other clouds.

However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/

Exact reason why I asked :) Thanks for the response & all the work you're putting in, it looks really promising.

Haven't done anything with Azure yet. Shouldn't be hard to add though, we don't rely on any special cloud storage features, just simple upload/download of files.

(Some of these are options I've not looked deeper into - Fauna, Planetscale)

This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy.

Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only.

So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.

There is a big difference between in architecture between Neon and PlanetScale, CockroachDb, and Yugabyte. Neon is shared storage (storage is distributed but shared) and the others are shared nothing. Shared nothing systems are hard to build with supporting all the features of the base system. E.g. https://vitess.io/docs/13.0/reference/compatibility/mysql-co....

Neon is 100% compatible from Postgres b/c we didn't (or almost didn't) change the Postgres engine.

What is the effective difference to you? Technically all compute can enter hibernation by dumping RAM (or just using virtualized memory backed only by SSDs).

CockroachDB already does true scale-to-zero if that's your requirement: https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-...

The effective difference is commoditization of the paradigm - there's confidence in a commodity technology that a proprietary cannot give (standardized use and wide support and community, multiple providers, self-hosting).

Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet.

Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.

I think it is more "market leadership" than commoditization that's working in favour of the MySQL-Postgres duopoly.

> "minimum self-deployable unit of cockroachdb"

If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers.

The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.

Most stuff you mentioned:

1 - Is not an actual relational DB

2- Doesn't really scale to zero

Planetscale does scale to zero, but has a ridiculous billing model.

Both CRDB and Planetscale "scale" to zero - but neither expose any concept of individual instances so I'm still not sure what difference it makes.

What's the billing model / how is it ridiculous?

Per row read / write billing -> how do you estimate that?

Either you have an existing db running and check the metrics, or you don't have a workload to move and you can just try it out from scratch. You can always migrate away if the billing model doesn't work for you.

Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.

MongoDB & CockroachDB are the only open source ones, the only ones we can hack on & improve & grow.

Neon seems like a vast vast improvement & great & desperately needed potential leap for mankind.

Planetscale is Vitess which is also opensource: https://vitess.io/

> "great & desperately needed potential leap for mankind"

Are you being serious? That's very hyperbolic if so.

> Are you being serious? That's very hyperbolic if so.

Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much.

Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them.

Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data.

We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]:

> This is the missing piece on cloud for masses

[1] https://news.ycombinator.com/item?id=31537313

There are tons of databases, I just named several and there are hundreds more for every possible niche.

> "But ones that are not ours, that we cant hack on, that we cant run ourselves."

Who is "we"? Is your entire issue that not everything is open-source?

> "these are not humanity's heritdge & humanity is cut off from active participation with them."

This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking

Your free to your opinion, but I respectfully do not share your values & sense of collective ownership over remote, far off software I can't see, shape, control or rework.

The most notable feature of Neon, that makes it so exceptionally smart, is that it takes the humanity's existing first best choice default-go-to database, by a country mile, and layers in really smart decoupling of storage to help it scale. Reinventing things from scratch can be a win, but that this is already a well known quantity, much loved & cherished & used by all, hacked by all, grown by all is a planet-sized huge plus mark.

I think you vastly overrate the broad availability of alternatives to this, of potential starting places others might explore. Most of your list is dominated by proprietary whatever. Even skipping that, yes, there is a ton of novelty & promise, the potential for breakthroughs. But this is starting with the best, and pushing it into the cloud-layer, into the troposphere.

> "I think you vastly overrate the broad availability of alternatives to this"

There are dozens of "scalable postgres" alternatives from Redshift to Citus to CockroachDB to Yugabyte. I find it strange that you say its overrating the availability when you're also comparing it to something that isn't even a full product yet.

I've met my share of developers without perspective in time, and hey I was probably one of them for my first 5-6 years as a developer, that I don't think one needs to assume corporate toadying for hyperbolic importance for mankind to be ascribed to a corporate offering.

rektide is not Neon employee, nor is he or she was asked to post this.

Thank you for the comment rektide. We of course would love to see constructive comments and criticism.

- Neon CEO

Can you please stop posting these off-topic comments (which are against the site guidelines, as you'll see if you review https://news.ycombinator.com/newsguidelines.html)? Also, can you please stop posting unsubstantive comments generally? You've been doing that in other threads as well.

I appreciate your concern for the integrity of HN discussions, but the thing to do if you suspect abuse or manipulation is to email [email protected] so we can look into it. Posting about it in the threads themselves is explicitly against the rules.

understood, dang sent you an email

but please take a look at the flagging attack on all of my comments outside this thread. the timing is too peculiar here

somebody really didnt like what i wrote above ;)

Moderators did that—your comments have been breaking the site guidelines quite badly!

that explains why seemingly YC money related threads are my criticisms getting flagged

Users can flag comments too and yours get flagged because they break the guidelines in overt and obvious ways which is pretty easy to avoid.

Doesn't seem like an employee due to another comment they posted. https://news.ycombinator.com/item?id=31537613

Minor correction, both of them are source available, but not open source

Cockroach DB license - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/LICENS...

Mongo license - https://github.com/mongodb/mongo/blob/39e4b70/LICENSE-Commun...

Cockroach is eventually open source which I rather appreciate. Any code that is 3 years old goes Apache if I recall it correctly.

That only applies to the core. The "enterprise" features are proprietary until the copyright expires.

IIRC, backup was enterprise but is now part of the core. However, restore is still enterprise.

There is also gigabyte as open source. In our tests (which means nothing in general as it’s specific to our business case), it outperforms cockroachdb.

Sorry, cannot edit; Yugabyte (ios spellcheck).

Yes, but they are not "scale-to-zero"

Both CRDB and Planetscale are.