Who in 2022 doesn’t “want graphics”?
Nevertheless, now that is extremely ancient history.
During the last 20 years, I have been using Linux on all my desktops and laptops, without needing any Xorg config.
The only exception is that now I have a couple of lines of Xorg config, which describe how my 3 monitors must be positioned.
Without an Xorg config all would be OK, except that I would have to rearrange the monitors after each power-on.
Of course, there was no need to write those lines of Xorg config, I have just given the command to save the current monitor configuration.
So I cannot imagine what problems still exist nowadays that may require "to redo their Xorg config".
All the problems that I have seen in recent years with Xorg had been caused by either a kernel compiled without some modules needed by the hardware, or by an Xorg server that was installed without some X device drivers needed by the hardware.
With a complete kernel and X server, things should just work.
In my opinion, the worst hardware problems in Linux are usually with Bluetooth or with audio.
I have not encountered yet hardware problems that could not be solved in Linux (unlike for Windows Enterprise, where I have met unsolvable problems).
Nevertheless, in the case of Bluetooth and of audio, I have encountered frequent cases when the default configurations would not work and when finding how to make the right configuration was highly convoluted and unintuitive, so many hours could be lost with that.
Even so, the difficulties of installing Linux versus Windows, Android or Mac OS are just a myth.
Most users see that Windows and the other commercial operating systems just work, because they have been preinstalled by professionals on their computers.
When you are the professional who must install Windows on a new computer and when that computer happens to be slightly non-standard, which is the case for most embedded computers, installing Windows is usually much more difficult than installing Linux.
I have encountered cases when I had to work a week to succeed in making Windows have an acceptable performance on a computer on which Linux just worked from a live USB stick, without any problems.
If you pull out everything complex and thoroughly understand what's left you won't run into the "idk just reset the whole OS" situation.
Something Windows/OSX users coming to Linux might not realize is that almost nothing in the userspace is actually necessary because the kernel does almost everything. Especially if you don't want graphics but even if you consider those components necisary then all you need are: The kernel (one file plus some modules), a device manager (the thing that creates nodes in /dev when you plug stuff in, if you use kdrive you don't even need this), a display manager (Xorg, kdrive or Wayland) and some kind of shell and VTE (bash and Xterm for example.)
Literally everything else can be thrown out if you don't want to deal with it. Don't like configuring systemd? I've used just script in the past at /linuxrc. Don't like what Gnome is doing or maybe it's crashing on you? Get rid of it, for the most part it's just aesthetic and the functionality can be replaced by eg fluxbox and PCFMan (and maybe wpa_gui if you like using a GUI to configure your wireless card.) Did the package manager get confused? You can actually get away with removing the database and starting over (although this will replace all the binaries it previously installed and potentially overwrite configs.) Don't be afraid to nuke things.