pleshkov
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Sunday May 10, 2026 @07:56AM (#48083231)
It's close but not the same. Kernel PCA lifts all D coordinates which gives M around 525k at D = 1024. In the post I do PCA first to reduce D to d = 256, then lift only those d coordinates, M = 33k. Much smaller, much faster Ridge solve.
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Saturday May 09, 2026 @07:06PM (#48079159)
I agree that we don't want to reconstruct the whole vector while retrieval and it makes poly-AE toy-like at the current state non production ready. My main interest here in the just taking more recall pp in closed form. And then think about how to make it fast. In all threads I got a good intermediate thoughts about the topic which may help me to bring to closer to production form
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Saturday May 09, 2026 @06:29PM (#48078897)
Just checked the normalization point. You were partially right, sqrt-normalization makes the difference x2 less. I'm updating the numbers in the post.
Interesting moment. I did a smoke test of poly-AE without whitening, and the result didn't change. I won't mention it in the post cause right now I'm not sure if it's a random effect or really a polynomial lift compensates normalization
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Saturday May 09, 2026 @06:18AM (#48073715)
The polynomial lift in this post originally came out of an unsuccessful experiment with hyperbolic embeddings. The idea was to embed corpora into a hyperbolic ball (anisotropic embeddings have a tree-like structure that hyperbolic space could exploit). The lift was a tool to go from hyperbolic latent back to Euclidean for retrieval. Hyperbolic part didn't work; the lift evaluated standalone kept showing real signal, and that became this post.
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Friday May 08, 2026 @11:33AM (#48064610)
Fair point — lam is technically a hyperparameter. In practice I used lam=1e-3 (the default in the code) across all four models without tuning, and the gap to PCA is robust enough that small variations don't change the conclusion. So more accurately: "one hyperparameter with a benign default" rather than "no hyperparameters" — you're right I overstated.
Re:A polynomial autoencoder beats PCA on transformer embeddings
on Tuesday May 05, 2026 @07:32AM (#48021033)
Author here — questions and pushback both welcome.