It could be a remote X client on another machine.

The valuable data on your machine might not be something permanently stored in the filesystem, but ephemerally recorded in a process. Say, a memorized password.

Couldn't the malware still get it without having to deal with X? E.g., by attaching to your process as a debugger?

Try doing that on a regular Linux install, you might be surprised.

I assume you're talking about Yama, but that's not universally enabled by default. And even if it is, malware could do something like wait for you to open Firefox, then immediately kill it and launch its own version with its own code already baked in.

It's enabled in all mainstream distros that I know of. I work on debuggers and this is the single most common issue that people report with attach.

And yes, if the malware is running under the same account that you use to login, it can do a lot, X or no X. That's where various forms of sandboxing come in. And the problem with X is that it is basically impossible to properly sandbox an X app.