r1290

I just did the exact same thing for my company. I didn’t do the sql lite approach for gdrive though just a direct search.

The one part that is still difficult is the data modeling and table level descriptions etc. Maybe you make an update to a table - remove a column, etc. The 3rd party systems all have their schemas defined but the data warehouse is a bit more loose. So solving that really helps. Did you just use dbt schema to describe tables and columns then sync that to your bot? How did you keep it updated? And end of the day - worth building or buying? Also how did you track costs? I let users choose their model - but have learned it can get expensive fast. As I can see there are a lot of providers trying to solve this one thing. That said the data warehouse aspect is the loosely defined area and I can see dbt or one of those players try to build something.

Still can’t believe there is no true open source cms that sits on top of a Postgres db. Like. Yea you have payload but you are locked into nextjs. Just a cms that hooks into openapi or Postgres natively would be great. I still prefer Django over anything else now days.

Right. Give the agent permissions to search across certain tables. Wonder why reinventing file based detect for the db side is just a current fad?

Agree with this. Is this current world because llms are just trained on the file system? But in a year from now we move to the db?

Wouldn’t touch it with a 10 foot pole. What would you prefer over django of the listed above? Curious.

What’s your opinion on react email?

How does it handle large tables like. 2b rows? And how does it stay updated?

Looks good. Question - is it always better to use a monorepo in this new AI world? Vs breaking your app into separate repos? At my company we have like 6 repos all separate nextjs apps for the same user base. Trying to consolidate to one as it should make life easier overall.

agreed. dealing with a complex polygot projeft. mumtiple mongodb, node hell, etc. hard to onboard.. or move fast

In regards to docs, data and repos. What are you looking for specifically? What entails a good vs bad architecture for a company?

This x1000.

I’ve tried every orm in node and nothing compares to Django orm. The way you can scaffold an apps data models is amazing.

What do you recommend as an alternative?

Would love to get thoughts from others on convex who have used it.

Django, rails or laravel. Big 3? Seem to make it easy

Cool. Can you share more about the tech? Ingestion engine - is this a background task that scrapes the web? You download the mp4 and convert to transcripts then generate embeddings? For each embedding of an episode - how do you break it down? Summary the episode and embed it with metadata? Are you using pydantic ai for structured output? Celery for tasks? Just a curious dev

Same. I’m using Django with fast api. Then I have a nextjs front end. I do wish I could bake it all into Django but I’ve learned that it’s best to keep Django api first unless it’s a simple site/app. But with a lot of interaction, using react helps.

Fivetran should’ve done this a long time ago. I think that both etl and reverse etl is going open source route. With this ai world we live in now. You just need dagster or temporal - and a few lines of python.

Fivetran has been great. But in this new ai world. Something like dragster + dlt and sling. You can have your own fivetran developed in house. I haven’t dove too much into reverse etl- but it would be awesome to see a dtl like open source tool for reverse etl.