RobinL

One thing we've seen with DuckDB, which they also mention in their docs, is that it's possible to 'over parallelize'. Beyond some number of cores, the overall workflow measured in wall time actually slows down when you add more CPUs, because of the overhead of parallelizing, despite them running at 100%.

We observe this even if we pull the data from S3 onto the local SSD before processing,

There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark.

For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This is particularly the case now that: - Splink 5 (released in September) allows you to run your workload in n chunks, meaning you can distribute it across multiple machines even on DuckDB - DuckDB 2.0 parallelises better on large machines, so throwing a single massive machine at problems is now more viable

In general, after using Spark for a long time, I think for most workloads, it's just not really needed any more. Too complex, too difficult to tune. Perhaps if you have genuinely massive data. I've written a bit more about why I advise against Spark in most instances here: https://www.robinlinacre.com/recommend_duckdb/

Still don't usually see a speedup past about 192vCPU, but prior to 2.0, speed would top out at more like 64vCPU, so this is a big win.

That has enabled us to scale Splink (FOSS record linakge software) to process 1bn records with 10bn record comparisons in 10 mins, which as far as I'm aware makes it the fastest linkage engine of this kind. Most of this performance comes from simply delegating the big calculations to the DuckDB engine! https://moj-analytical-services.github.io/splink/topic_guide...

Thanks to the team for all their work!

So for example, for wind power the consumer effectively pays the CfD strike price, largely irrespective of the spot price.

Fwiw, I do agree we have a problem with power being too expensive, but it's more complex than just the consumer pays the most expensive source.

Another aspect of context that could be relevant to your work may be how people are nested within households. In some record linkage scenarios you can run a linkage on household membership and use this to make person linkages more precise. I imagine an LLM agent could also use this information effectively if it were explicitly provided, but sometimes they miss logical leaps like this

It does not know enough about the records in the context of the overall dataset:

- what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but rare in others.

I've written in more detail about this here: https://www.robinlinacre.com/fellegi_sunter_accuracy/

That said for single images the old model was already okayish for prototypes

This is a quick prototype, made for my 7yo who loves buses.

A few tips:

- Click the freeze icon to be able to zoom and pan without it changing the location of the lens

- Double click to position the lens

Thanks for feedback about the data. The bus stop and route data comes from the Department for Transport’s Bus Open Data Service (BODS), using the national GTFS feed. It seems some things like Manchester metro creep in.

All these caveats aside, as soon as I got this working it felt like a promising direction. I will pay attention to everyone's comments and suggestions and iterate! Thank you!

Incidentally there's a bunch of other free stuff I built for my 7yo on the root domain

A mental arithmetic tower defence game for kids. Now with head to head/two player. Code: https://github.com/RupertLinacre/arithmetic_annihilation

This doesn't mean we should all attempt to do the highest possible paying job available to us, but it does mean there's a genuine loss to society if (say) a high paid software engineer decides to quit and start a bike repair shop i.e. the economic pie will be smaller an so will the tax base. So when making the decision we should at least be conscious of this.

As an economist, I do kind of agree with the premise of the article, but in a different way. As consumers in particular, we can do much better than following incentives.

For instance, we can use the car less, even if it's a bit inconvenient. It's much nicer to see more people out walking, on their bikes or on the bus, leads to less noise and congestion. So there's a positive externality. Or I could buy a smaller car, even though bigger one is more convenient.

Similarly, if you're lucky enough to have a high paying job, you can buy all sorts of status symbols. Or, you could choose to consume less than you earn, and give the difference to charity, or invest in something of social value.

Whether you agree with this or not, economic theory provides a clear framework to think about all of this, both the price signal and the difference between private and social incentives. What bothers me (as the article touches on) is that people often have a simplified view of economics which is essentially that 'market incentives and price signals broadly lead to good outcomes (or at least, better outcomes than communism!), and therefore economics always morally endorses buying things at market prices'. There's nothing in economic theory that says that spending £50k more on a car because it's got a fancier brand is a reasonable moral choice.

Until recently LLMs have been really bad at this task. I always knew it was coming, but with GPT 5.6 they've suddenly become good. It's pretty clear to me that it won't be long before most of my work on this is rendered pointless because the LLM can either do the classification itself (when given agentic access to the canonical list of addresses), or write a classifier itself if given enough labelled data. Of course these two are complementary

However, there may be more to it than I thought. No admittedly the following is ChatGPT's opinion, but I think it's valid and made me think again:

> The second is the better translation as English poetry; the first (Claude's) is the better translation if you want to see the Greek more directly.

> If I were studying Homer, I'd strongly prefer the first. It continually reminds you that you're reading an ancient poem whose concepts don't map neatly onto modern English.