aynyc

single writer AND single reader.

https://duckdb.org/docs/2.0/connect/concurrency

It doesn't. That's the issue. DuckDB for us is a pure read-only service pipe. We need other tools to ingest and write to parquet, which has its own limitation.

SQLite does a great job to support single writer and multiple readers. That's what we use for aggregation right now. DuckDB is purely post analytics.

We tested this out extensively. DuckDB + Nvme kicks ass, but if you ingesting and analytzing near real-time quotes/trades, DuckDB doesn't handle well because DuckDB can only support one reader and one writer in read-write mode.

It means a language for the very elite special force unit. In MSFT, that means the Ads teams will be able to serve you Ads in excel very quickly. /s

Please execute my ignorance. And this isn't an attack on anyone, just my own stupid limited understanding.

Aren't JellyFin & Plex essentially just a catalog for your local data (most likely torrented movies, etc.) in a web app? If so, what's the big deal?

This is about their bond, not that share price. If you are in the US, it's like having low credit score, everything you want to do financially such as leasing or financing a car, buying a house, etc.. will be more costly (higher interest rate) from the lender.

NYC owns those lands for water. ADK has forever wild in NYS constitution. They are not gonna get relax for data center because they discharge water and noise and add significant infrastructure change.

Solar farm on the other hand might go up tho.

That's the insider joke. Look up NY1-NY4 data centers, they are all in NJ across the river. NYC just dump their shit into NJ is the usually move. But those areas are full now, and they don't really have anywhere to go but south jersey.

Don't worry, they'll just build the data centers in NJ and still considered NY 1-20.

Sarcasm aside, I don't really know where they would build data centers in NYS. Electricity rate in northern and western NY is going thru the roof. ADK/Catskill have very sensitive environmental laws. Can't really build in lower hudson as real estate cost would be killer.

I've been reading Alastair Reynolds's books. They are pretty good.

OK. I've read it a few times and still don't understand. Where is the distributed part? You store data in a single transaction into postgres. What/who is notifying the message queue?

I'm wondering. Just wondering? Will they ever support multiple storage engines like MariaDB? Having a storage engine that support OLTP or OLAP or append-only would be cool. I totally understand if they don't want to do that.

MySQL has Galera cluster for that.

I just came out of an initial interview for a rather senior enterprise position. It was bad. It was bad in a sense that they really didn't know what to do. The senior manager (EVP level person) asked me about LLM for code generation. I told them about my aws kiro-cli experience. They literally asked me to sit down and show them how they can do it. I'm pretty sure they want me to back for another round.

This whole thing reminds me of when I was in school, showing old timers who to use MS Office and VBA.

We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.

When you say compatibility issues, you mean they are more problematic or less?

It’s pretty common to read Parquet into Arrow for transport.

I'm confused by this. Are you referring to Arrow Flight RPC? Or are you saying distributed analytic engine use arrow to transport parquet between queries?

Yes, but parquet hates small files.

So feather for journaling and parquet for long term processing?

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?

Going back how’s many years? I checked recently and VTI easily out perform in the last 5 years.

You'll be amazed on what the new breed of engineers are using Redis for. I personally saw an entire backend database using Redis with RDB+AOF on. If you redis-cli into the server, you can't understand anything because you need to know the schema to make sense of it all.

This belongs to the "shit that never happened" list.

I lost all my college projects due to source forge bullshit decades ago. And old pictures from digital/film cameras. Right now, I run both google/apple photos so I have 2 backups of my pics and videos.

Honestly, after 20 some years in technology, I don't think it's possible to back up everything unless you are willing to pay and constantly work at it.

I've seen this type of advice a few times now. Now I'm not a database expert by any stretch of imagination, but I have yet to see UUID as primary key in any of the systems I've touched.

Are there valid reasons to use UUID (assuming correctly) for primary key? I know systems have incorrectly expose primary key to the public, but assuming that's not the concern. Why use UUID over big-int?

Yes, we have to in order to use a lot of the features. The core issue for us is really the way Django assumes code represents database state. In normal webapp where the application has full control of the database, that's a good idea. But our databases are overloaded for simple transactions, analytics, users managements, jobs and AI. Business uses the databases in various ways such as Power BI, Aquastudio, etc.. Django app is actually a tiny part of the database. As you can imagine, we duck tape the heck out of the databases, and Django goes bonkers when things aren't matching.

Any docs? Django migration is a HUUGE pain point for us.

That’s what we do now. But it gets repetitive and not leveraging Django core features.

I've been using Django on and off at work for the past few years. I really like it. That being said, I still find its ORM difficult. I understand it now that since it's an opinionated framework, I need to follow Django way of thinking. The main issue is that at work, I have multiple databases from different business units. So I constantly have to figure out a way to deal with multiple databases and their idiosyncrasies. I ended up doing a lot of hand holding by turning off managed, inspectdb and then manually delete tables I don't want to show via website or other reasons. For green webapps we have, django is as good as it gets.