I always try to avoid learning by pattern, because it would make me less flexible when composing.
It's important to know the notes on the keyboard, but it's far more important to know how they work together in harmonious patterns. Composing entirely by ear is an option, but you still need to transliterate what your mind is hearing into notes on a page (preferably more complex than just a single note melody).
I 100% get where you're coming from, but I think a grounding in theory helps a ton with composition (even if you don't know how to play a specific instrument).
Additionally let me call out that sight reading and composition are completely different objectives.
I also sincerely doubt playing random notes on the piano as part of a sight reading process would make you a better composer unless you're aiming to be the next Karlheinz Stockhausen.
Even though I could read sheet music, I mostly played piano by ear until around high school when I decided to properly learn to sight-read. At the time, my access to musical resources was limited, so I borrowed the Episcopal Church hymnal from our church.
The great thing about hymnals is that they are choral in nature, usually with four voices, but at the same time, they are rhythmically simple in nature since they are intended to be sung, allowing you to purely concentrate on the notes themselves.
I ran through it sequentially for months and found that my sight-reading capabilities measurably improved.
https://hymnary.org/hymnal/HPEC1940