venkat971

we did something very similar at deepsql.ai pl check it out, happy to exchange the notes and lessons. DM me.

First, they launched this as their in-house data agent. Looks like they made it available in ChatGPT Work.

https://openai.com/index/inside-our-in-house-data-agent/

btw, this project influenced me to build deepsql.ai (the brain for RDBMS)

this can't come from FKs, in reality, FKs is bit outdated concept. The relationship among the objects always comes from the joins (from query logs). So by just looking at the schema, in a static sense, would not give us data flow picture. Try to re-draw the schema diagram by scanning the code base of GitLab, I bet you will have a different view here...

we tried to do something similar at deepsql.ai

In the long run, I’m seeing it part of the database itself.

Is this something one can self host? data does not leave the premises narrative is holding very strong in enterprises. We recently rolled out deepsql.ai - AI DBA as SaaS and quickly realized that anything that connects or touches the enterprise data, can't be a SaaS. We had to open source it. Just sharing some learnings! good luck...

Data confidentiality and sovereignty matter a lot to enterprises. You may want to redact access to the most sensitive data from both humans and agents, and it gets tricky to enforce such dynamic policies at the database level. DeepSQL solves this by managing all DB users (humans and agents) at the DeepSQL layer.

DB availability is out of scope for DeepSQL at the moment.

Oracle has something like this, and I worked on this project when I was there. Although I was fascinated by the use cases, the adoption was pretty low in the relational world.

https://www.oracle.com/in/database/integrated-graph-database...

Yes, we never hired one. When we got the time to hire one, we developed this tool.

We are making it open source by month end.

Thank you!

What is the stack that is required to setup your opensource repo? and what's the machine config you recommend.

btw, I liked the pricing page. I am planning to setup a similar pricing page my project deepsql.ai (dba agent for postgres and mysql).

optimizing the queries and schema monitoring just one aspect of Deepsql. It has built in BI server - security and policies enforcing module (on who can read what data) - All these can be wired in to the regular business and data lookup workflows.

Good luck with your project. happy to exchange notes, feel free to ping me at [email protected]

Thanks :) Pl give it a try.

Interesting topic, We built and use DeepSQL (https://deepsql.ai/) at Stayflexi(YC) to address some of these issues. It's an DBA agent to prevent schema bloat, over indexing and does continuous monitoring of query workloads and proposes fixes.

Schema blot issues are real when you are vibe coding. Our engineers vibe coded and bloated our schema from 230 tables to 600+ tables. Many of them have repetitions of columns across the tables and often too much indexing. If your Postgres is on Aurora, the bloat easily multiplies your bills.

Claude would not do the frequent schema drift changes, data relationships, block the query execution due to PII leak, cardinality calculations of key columns… 20 other parameters that effectively works like a well trained DBA. It’s all about harnessing the agent. And our harness can work with Claude or other coding agents to perform efficient DBA tasks.

Eg: our security harness blocks sensitive data from being queried, and this can be enforced at the user level. It takes effort to build and maintain such harnesses around claude at the org level.

Hi, yes. Right now we are bundling our key to maintain the consistency of DeepSQL agent performance. We will lift this key bundling very soon, for enterprise deployment we might allow BYOK (bring your own key)

Yes, this requires docker. Here is more technical documentation on prerequisites and suggested setup (including AWS CFT)

https://docs.deepsql.ai/quickstart/

We have cross checked with industry standard tools like pganalyze. Most them analyze the workloads in isolation. There are no continuous evaluations and context carry forward across the runs.

DeepSQL does continuous evaluation of queries, we maintain performance life cycle of a query by customer.

A dashboard query for customer A might run for 2s, but for customer B it would take 50s. Here data clustering is the problem, skewness of distinct join keys … various factors. If query fix has to happen by looking at that query alone, one would make wrong decision (probably creating index). But a holistic decision would be partitioning here.

removed that metric.

Our goal was to tell a convincing story with case studies. However, you are right on many dimensions that this number means nothing for many optimized scenarios.

Sorry for the confusion. It's not opensource. For first few installs, we are offering free 3 months and free consulting service to setup the tool end to end at your org. Post that pricing will be based on AI tokens consumption with some standard monthly pricing (we are also exploring the pricing sweet spot)

I am happy to get connected over email [email protected] and share more details on pricing and deployment help.

We did cut our overall DB spend by more than 3x. Our $9K spend drops to $3k - indexes not kicking in, rapid growth of log tables and schema bloat were the culprits. Also, we removed the retool and appsmith licenses, another $2k savings.

We worked with 6 YC companies who are running on Aurora postgres and worked through the inefficiencies of workloads - too many indexes (in some cases almost every column is indexed), over indexing significantly increases I/O ops (writes), this hidden costs are enormous for fully managed services like Aurora. In all these cases, we could safely come up with a plan to cut down their DB spend by more than 40%.

Deepsql has built in BI dashboards server - You can ask agent to build any BI dashboard from the connected data sources (this can be used by your internal teams or you can even share the dashboard link with customers with password protection, just like how we would share the retool links).

Great point and thanks for the feedback. We made it self-hostable so that data doesn't leave the premises. We haven't open sourced the code yet, but most likely we will head in that direction very soon.

Deepsql doesn't require any predefined rules + stats, nor you need a DBA to operate it. It's an autonomous agent, that learns your business context, schema, join relationships and creates a mental map of everything (just like a DBA who joined your team). This we call it deepsql brain init stage. Once the init is done, it will then look at all slow queries workload, comes up with comprehensive suggestions of indexes, MVs etc. Other DBA tools might look at one query at a time, if we try to optimize one query at a time, we might over index.

Also, deepsql is intelligent in understanding the deployment type. Postgres standalone deployment is very different than Aurora postgres. Cost factors are quite different. Other DBA tools might ignore these factors - Hence they need well trained DBA to use them.

Deepsql doesn't replace your DBA, it gives your DBA super powers. In my org, when we had to decide on hiring a DBA, we chose building deepsql and empowered our senior engineer to handle the database tasks like a senior DBA.