The 'Elbow method' [1], yeah. Sweep a bunch of different k values and pick the one where your target metric starts having diminishing returns. There's still not a particularly great way to pick k for this type of clustering.

There kind of can't be as clustering is sort of inherently subjective. Other methods like HDBSCAN can find the number of clusters themselves. There's still a 'one key parameter' you have to pick yourself, but it can be more intuitive and less sensitive for some datasets, when you dont' know how many clusters there are / how many you want.

Sometimes, picking k exactly is useful for practical reasons too. Like bounding the size of your codebook. Or partitioning a dataset into k equal-ish pieces, say to spread across a pool of servers of a known/fixed size.

[1] https://en.wikipedia.org/wiki/Elbow_method_(clustering)

Yeah, I guess you can't really avoid picking k without hierarchical clustering.

Yeah, it's not a well-defined problem, so there's no one true solution. Then only question left is "well what do you want?". Always have to pick something.

https://en.wikipedia.org/wiki/Cluster_analysis#Definition

There isn’t a well defined correct or ideal answer. It depends on what you’re doing with the clusters. Sometimes the raw data are too numerous to properly deal with and you just want a smaller set of points to reason about. In that case pick the number of points that are convenient to reason about. Sometimes you have a fixed budget for the cluster representation and that defines k. Sometimes just pick sqrt(N) so the average cluster size is k.

You could use X-means instead (https://www.cs.cmu.edu/~dpelleg/download/xmeans.pdf). The idea is to use an information criterion that trades the number of parameters against the reduction in within-cluster variance. The approach essentially splits a cluster when the BIC of the split is larger than without. I’d use this to select k and still run normal k-means though.

Why would you use run normal k-means if e-means gives lower loss and reliably goes much faster ? If you don't believe it you can try e-means at https://e-means.github.io/playground.html

My comment was not about whether or not I believe the claims made by the authors of e-means but how you would solve the problem of choosing the right k.