mrjn

Author here. Egres is a Postgres-compatible database designed to tackle heavy workloads and run thousands of tenants on the same server. It works with psql and drivers unchanged. The post goes into details about the what, why and how; which I hope you'd find interesting.

Before Egres, I was the founder of Dgraph -- author of Dgraph and Badger. ex-Google, ex-Uber. So, been working on databases for a while now. The idea for Egres all started with a question: "“What would Postgres look like if it were to be designed today?” After years of pondering and a year of work, the result is Egres.

For Egres, I built a new storage engine called zdb, which is pioneering LSM forest concept -- a tree of LSM trees. zdb can run on local disk or object storage, has native roaring bitmaps integration, can do instant zero-copy branches for database or table.

All of this runs on a custom co-routine system I built called zfiber, so all queries execute on that. Egres meters CPU ticks, disk IOPS and memory used per query, per tenant, so it can run thousands of tenants on the same server without noisy neighbor problem.

This release is single node and the query ability is limited to equality (no joins) queries only. Raft replication is already baked in, future releases would allow multi-node clusters.

Egres is open source under Apache 2.0, the GitHub repo is here: https://github.com/manishrjain/egres.

Would love to know your thoughts, and which features to prioritize getting into next release.

Author here. Egres is a Postgres-compatible database designed to tackle heavy workloads and run thousands of tenants on the same server. It works with psql and drivers unchanged. Right now the query ability is limited, and would grow with future releases. The post goes into details about the what, why and how; which I hope you'd find interesting.

Before Egres, I was the founder of Dgraph -- author of Dgraph and Badger. ex-Google, ex-Uber. So, been working on databases for a while now. The idea for Egres all started with a question: "“What would Postgres look like if it were to be designed today?” After years of pondering and a year of work, the result is Egres.

For Egres, I wrote a new storage engine called zdb, which is pioneering LSM forest concept -- a tree of LSM trees. zdb can run on local disk or object storage, has native roaring bitmaps integration, can do instant zero-copy branches for database or table.

All of this runs on a custom co-routine system I wrote called zfiber, so all queries execute on that. Egres meters CPU ticks, disk IOPS and memory used per query, per tenant, so it can run thousands of tenants on the same server without noisy neighbor problem.

This release is single node, but Raft replication is already baked in. Future releases would allow multi-node clusters.

Install: curl -fsSL https://egresql.io/install.sh | sh. Repo: https://github.com/manishrjain/egres.

Would love to know your thoughts, and which features to prioritize getting into next release.

skiplists form the basis of in-memory tables used by LSM trees, which are themselves the basis of most modern DBs (written post 2005).

Talking about dark modes, nytimes still doesn't have an official dark mode (or not I can easily see). This should help.

Wouldn't a Vitess for Postgres look like a Yugabyte?

This would be a bigger problem in community use cases. We're going to introduce a trust level system to tackle this, inspired by Discourse:

https://blog.discourse.org/2018/06/understanding-discourse-t...

Thank you! We went through many iterations to get to this pricing model. So, I appreciate this comment.

OpenAI has strict privacy rules in place. Being on a business account, any of the data that we send to OpenAI wouldn't be used for training.

It's a SaaS. Over the next few months, we'll be going through the SOC2 compliance process.

Will update the footer, thanks for catching.

https://struct.ai/struct-for-slack would allow individuals to use Struct as an interface to Slack, without the whole team having to switch in one go.

Perhaps a good start would be to create a Struct org, for communicating within the team?

We'd surely look into Teams integration at some point. Though, I'm not sure how many takers would be there for Teams/Struct integration. My perception is that Teams users tend to be medium-to-large businesses who are tied to Microsoft Office, and would be very hard to sell to. I might be wrong.

Fair point. We can all do with less meetings. I think tools like chat platforms / loom are all aimed at reducing the number of meetings we need to do.

Though, you do have to abide by Metcalfe's law. Most modern work happens in teams and therefore, you have to pay the price of O(N^2) communication, where N = size of the team.

I don't see communication decreasing overall. We just have to use the right tools to be more effective at it.

This should help: https://struct.ai/struct-for-slack

Could be a feature request to integrate with Matrix APIs. For example, we're doing bidirectional sync with Slack already. And unidirectional sync with Discord.

We'll be sure to go through all those details, as we work towards compliances.

Really wanted the .com domain, but that wasn't available. We got the .co domain, but .ai domain felt more right, given we're using AI, and now it's even part of the pricing model.

Hope to get that .com domain someday :-).

It's built on Tauri, so it picks the system default browser I think. We had applied for getting an EV certificate for the Win desktop app, but it's taking time to get that process done for some weird reason.

I'll get the FE dev to look at the too many redirects issue. Seems strange. Does it open fine on your Edge browser: https://app.struct.ai?

The reason Slack and Teams feels so "urgent", is because there's little going back to reply to an older conversation. Everything is so ephemeral on these platforms, that either you reply now, or it's a goner (slight exaggeration, but mostly true).

Struct's feed system is naturally designed to surface latest, unread conversations. So even if when I've been away for days, I just come back and go thread by thread, like cleaning an inbox. It's a lot smoother system, and you miss nothing.

You can install the Struct bot in Slack: https://struct.ai/install-slack -- This would by default pick up the last 3 months of conversations (we do this to decrease the unpaid load on GPT). You can ping us and we can help sync it from the beginning of whatever history Slack shows.

We don't have an exporter yet, but can surely put one together which provides a SQL / CSV formatted output (or whatever works best for users). We would never charge for exports.

Real-time platforms cause interruptions, yes. But, that's where we are today with Slack.

So, we're starting off from that point. Let's take real-time chat platforms, and reimagine how we can make them more efficient. More useful. Less noisy. That's the idea for Struct.

The list of people we email is different from the list of people we chat with. In a team, that means, team comms happen on chat platform. And external comms happen on emails. Within a team using a chat platform, it's unlikely that people would chat, and then say, hold on, let me send you an email. If that happens, that's very rare.

What's a lot more likely is you jump from a chat conversation into a video call. Hence, the need for a nice integration with video service (like Slack has huddle).

So, for a certain set of people (your team or community), you're going to use the chat platform. And therefore, the chat platform should be built to enable conversations with good retrieval and focus -- so it can be used for complex conversations. Struct hits that pretty well.

Worth a conversation for sure. Maybe find a time to chat? https://struct.ai/schedule-demo

Or, you can jump in here and we can have a discussion: https://chat.struct.ai/join/DXVTmseDdBkeA6mG

It's hard to do summaries for chats in channels. Just not enough focus in them to be useful.

Could also be the model that they were using. I've tried a few and found GPT4 to be most accurate. GPT4-Turbo is still a question mark.

Struct is quite inspired by Discourse. It's a unique blend of forum and chat application.

So, you know, every forum, including email, has threads and feed. But, forums are asynchronous, not designed around real-time chat. The challenge when applying concept of threads and feeds to a chat platform is that you need to make it real-time. And that's the hard thing that Struct is tackling -- with the real-time feed design that we have.

I disagree a bit about, "if it's chat, it doesn't need summary". I've been part of enough conversations which go on and on, to know that's not true. Even with the small team we run, we have threads which have like 100s of chat messages, going back and forth. Having summaries really help.

Or, at the very least, you'd appreciate the titles, so your feed would make more sense.

You can structure your threads using tags, and create feed around those tags. That'd be the equivalent of "folders". (Reminds me of when Gmail came out and people had to learn to map their labels on SMTP folders)

The difference in a Struct thread, v/s say a Discourse thread would be this. Struct emphasizes short form, real-time, back and forth communication. Discourse emphasizes long form, well-thought through, one-off posts. Former is chat, latter is forum.

This is long topic, but something I think about a lot. Designing Struct is hard exactly because of this balance between structure, knowledge and being real-time.

Arr.. we should clean up the error messaging a bit. Not to hide OpenAI, but just make it appear nicer.

I like OpenAI. GPT4 works really well. Some others we tried were cheaper, but the quality wasn't as good.

Absolutely! We already have one chat message fork working. Backend supports multiple message forks as well. So, very close.

Yup. That's the point I'm making. Slack is used as ephemeral messaging system, a knowledge void. Struct aims to proves that more is possible.

Yes. We'd be documenting Struct APIs. And also we want to identify the top 5-7 most popular Slack integrations, and build them ourselves for Struct.