Latency hardly matters when the majority of the lag is the 100 light year leg: you first spread out your whole message to the surface of the filter with a high bandwidth distribution medium. Then each section of the filter switches in sync, encoding the message without relying on a real-time input from a central location light seconds away
Why not use an array of smaller polarizers that can be modulated when they block the path of a bright but less adjacent star’s light to the recipient?
You can phase the switches and "aim" the star's signal at a point in the sky. Lower-frequency modulation can get signals to observers not on the direct line, at a lower bandwidth, and you can probably sweep these. Worst case you can blink the whole affair slowly; a star with a five second light curve is going to be pretty interesting.
Like the way the earth is large enough to block the sun on the entire moon during a lunar eclipse.
Building a static support for your polariser might also prove challenging.
Your noise floor is now vastly greater.
I've also been thinking of whether or not some sort of stellar laser or maser might be possible. Modulating a narrow, tightly focused beam.
Radio is much better bet.