Is there a meaningful / well defined difference? One could say that lossy compression is extraction of useful information. You need to identify unnecessary information to know what to discard safely.

One is a subset of the other. Compression being the larger category.

A mean is a compression of a dataset and useful information.

My issue in my comment is that 'compression' corresponds to a massive class of technquies and there isnt a lot of content in the observation that useful information is compressive.

However there is some hypey people out there who think this observation has legs. Precisely the people who think intelligence is a mathematical problem, and not an engineering one -- which is my view.

ie., that a body isnt incidental to intelligence, but the heart of it.

Or: devices matter.

I tend to think about it the same way as you. Having an algorithm for multiplying two numbers is qualitatively different than having a lossy compression of a huge dataset of multiplication tables. The latter is what GPT3 has and it just doesn't scale.

Consider a generative adversarial network for faces. Photos have let's say some unique scars

A successful model could create faces with scars in them. But not the exact scar and face and background that they trained on without additional information. What you are looking for is mutual information between the images not compression

Yes you could use a very well trained GAN for face compression. But a GAN model itself would not be able to reconstruct its training input without being shown the images again

Isn't lossy compression pretty much the same thing as extracting some amount of useful information?

According to Wikipedia [data] compression "is the process of encoding information using fewer bits than the original representation" [1] while lossy compression is "the class of data encoding methods that uses inexact approximations and partial data discarding to represent the content" [2]

There is a difference between feature extraction and compression. The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set.

For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression.

[1] https://en.wikipedia.org/wiki/Data_compression [2] https://en.wikipedia.org/wiki/Lossy_compression

"The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set."

The compressor finds a smaller approximation of the data by finding the most redundant data, and as a byproduct it also finds the most unique. At least from the description you gave there is no difference between the two.

"For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression."

You can and we do use compressors to do exactly that. Take a painting (or a set) as reference and use it to compress other paintings and the greater the compression the more similar the styles are.

For an example see: https://www.sciencedirect.com/science/article/abs/pii/S00313...

EDIT: In fact one step in a compressor is feature extraction (usually called modelling) and the other is coding.

No, because lossy compression has the goal of preserving elements for replay. At the extreme you could turn a movie into subtitles, but not a costume from the movie alone. The costume might contain more information from the movie, but you can meaningfully recreate the movie from a single costume.

If I derive Kepler's laws from positions of planets, I have preserved elements of their motion for replay as well.

The laws alone don’t have enough information to replay their motion without each planets location, a foci, the orbital period, and an associated timestamp or equivalent information.

So, include that data and it’s lossy compression, exclude it and it’s not.