kwillets

I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.

Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.

Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.

Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.

I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.

I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.

But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.

This trend has been obvious since at least the Poland deal. Korea gets much more return on its defense dollar manufacturing exportable weapons systems than relying on imports or domestic-only programs.

I'm curious to see what patterns it's grepping.

This has a lot in common with a columnstore, but there are a few interesting differences.

The main one is subrow versioning. Column stores (in OLAP at least) have always used row-level versioning, which gets in the way of small updates. A single change to a row amplifies into deleting and re-inserting the whole thing, and operations that seem sensible like adding or dropping a column break previous versions. This scheme is the first I've seen that tries to fix that problem.

One other difference is a lack of compression, as it's zero-copy, so the performance gains of operating on compressed data are lost.

One of the founders of Antithesis gave a talk about this problem last week; diversity in test cases is definitely an issue they're trying to tackle. The example he gave was Spanner tests not filling its cache due to jittering near zero under random inputs. Not doing that appears to be a company goal.

https://github.com/papers-we-love/san-francisco/blob/master/...

This site is underweighted on OLAP. Columnstores were invented for precisely this use case; nobody in the field wants to normalize everything.

Which brings me to the question, why a rowstore? Are Z-sets hard to manage otherwise?

Another aspect of wide tables is that they tend to have a lot of dependencies, ie different columns come from different aggregations, and the whole table gets held up if one of them is late. IVM seems like a good solution for that problem.

Can you be more specific about what type of data mining you're doing?

He doesn't want to manage the database the way he manages the rest of his infrastructure. All of his bullet points apply to other components as well, but he's absorbed the cost of managing them and assigning responsibilities.

- Crud accumulates in the [infrastructure thingie], and it’s unclear if it can be deleted.

- When there are performance issues, infrastructure (without deep product knowledge) has to debug the [infrastructure thingie] and figure out who to redirect to

- [infrastructure thingie] users can push bad code that does bad things to the [infrastructure thingie]. These bad things may PagerDuty alert the infrastructure team (since they own the [infrastructure thingie]). It feels bad to wake up one team for another team’s issue. With application owned [infrastructure thingies], the application team is the first responder.

The object storage stuff is new, but it's mostly confirmed that the older architecture works. MPP with shared (S3) storage and everything above that on local SSD and compute delivers the best performance. Even Snowflake finally came out with "interactive" warehouses with this architecture.

Parquet, Iceberg, and other open formats seem good, but they may hit a complexity wall. There's already some inconsistency between platforms, eg with delete vectors.

Incremental view maintenance interests me as well, and I would like to see it more available on different platforms. It's ironic that people use dbt etc. to test every little edit of their manually coded delta pipelines, but don't look at IVM.

Over time I've realized that the best abstraction for managing a computer is a computer.

This is a fairly simple idea of indexing characters for each column/offset and compressing the bitmaps. Simple is good, as the overhead of more sophisticated ideas (eg suffix sorting) is often prohibitive.

One suggestion is to index the end-of-string as a character as well; then you don't need negative offsets. But that turns the suffix search into a wildcard type of thing where you have to try all offsets, which is what the '%pat%' searches do already, so maybe it's OK.

It's been around for quite while, but DB people hate to explain where they got an idea. For all I know Vertica got it from somewhere else; I think postgres got jsonb around the same time.

My last dive into matrix computations was years ago, but the need was the same back then. We could sparsify matrices pretty easily, but the infrastructure was lacking. Some things never change.

I wrote part of that -- it needed a tight summary that covers all the main components, so I packed as much as I could into a sentence or two. I believe the second paragraph is mostly from me, but it was some time ago, back when Hanok revival was fairly new, and a lot of books were coming out on the subject.

A few years after that one of the Korean dailies wrote an English article and copied my wording. :(

This is fun, but I'm confused by the architecture. Duckdb is based on one-off queries that can scale momentarily and then disappear, but this seems to run on k8s and maintain a persistent distributed worker pool.

This pool lacks many of the features of a distributed cluster such as recovery, quorum, and storage state management, and queries run through a single server. What happens when a node goes down? Does it give up, replan, or just hang? How does it divide up resources between multiple requests? Can it distribute joins and other intermediate operators?

I have a soft spot in my heart for duckdb, but its uniqueness is in avoiding the large-scale clustering that other engines already do reasonably well.

Don't forget the 2 hour tableau cloud runtime limit.

I'm just learning about agentic search so I'm a bit adrift.

One of my side projects is a full text index for pattern search, and I'm trying to understand how it might fit with that. You mention tool call overhead, but is that a significant part of the latency in the multi-turn scenario, or is it the coding agent being forced into a serial processing pattern?

Are you actually using grep here? How much data are you searching?

This article seems somewhat fanciful. Medieval Koreans ate two meals a day, and famines were common. Only a small area of the country is suitable for rice cultivation. The photos seem to show only upper-class clothing and furniture.

I've been to that makgeolli place in Jeonju, and it sells drinks and food as a set; there is no free food.

Nam (Man Tea) and other brands have been in convenience stores for years; most have a full shelf of it. As I heard it the male part of the name is due to its use in manly herbal mixtures, but it's not considered a male-specific substance on its own.

I built the ads audience system. Most of the effects of that are already known here; Ads became a big trophy within the org; everybody had to have ads and post-sale revenue, even the fridge people.

Sometime around when the CEO got out of prison a bunch of weirdness occurred. Good managers left, bad managers got hired, and everything became top-down. The group head "retired" but last year un-retired in a different position; I didn't know you could do that.

Engineering-wise it went from technical free rein to "only use this suspiciously chummy cloud vendor" in a few months. I never got to the bottom of that deal, but costs exploded, and revenue flattened.

I worked at Samsung and saw plenty of enshittification (including losing my own job in some kind of sketchy power struggle), but oddly their audio stuff didn't seem like a bad deal.

In TV's they support the higher quality home theater scenario while still making most of their audio money from soundbars that can't compete on audio quality. They're well aware of that fact, and their strategy seems to be to keep all options on the table.

One of the TV execs is an audiophile FWIW.

The SIMD string matching optimization unfortunately missed a trick -- it's more selective to match the first and last characters of a pattern than the first two, and it's the same cost.

Credit to Muła for this one: http://0x80.pl/notesen/2016-11-28-simd-strfind.html#generic-... .

Sorting wins for unbounded strings because it only needs a prefix of each string.

The blog links on the infra listings are dead :(.

My company is doing the dbx thing, and the best I can tell my manager is that I'm neutral on it.

My working theory is that the UI, a low-grade web-based SQL editor and catalog browser, is more integrated that the hodgepodge of tools that we were using before, and people may gain something from that. I've seen similar with in-house tools that collect ad-hoc/reporting/ETL into one app, and one should never underestimate the value that people give to the UI.

But we give up price-performance; the only way it can work is if we shrink the workload. So it's a cleanup of stale pipelines combined with a migration. Chaos in other words.

I think you can just xor the whitespace mask with the shifted one.

Also, when counting 0xFF bytes from a boolean etc., sub the mask; 0xFF == -1.

Vertica runs on commodity hardware, and there's no license cost for cpu, so it's very economical for large workloads. The quickest way to 10x your costs is to move a Vertica workload to Snowflake (last I heard my old job is now up to 40x). I'll have numbers on Databricks in a few months.

It's true that Vertica sales are optimized for large enterprises -- they just don't have the VC cash to hire 3000 sales people to sell it to the low end, so it doesn't appear on many people's radar.

I interviewed for an adtech position there last year, and it went pretty badly. I only had about 5 YOE in this area, but it seemed like they had even less, and it was like we were speaking different languages.