Direct matter synthesis doesn't make sense in this context, but I've seen a lot of people on the Internet assume this is what happens. The way they talk about it on the show could lead you to either theory, for both replicators and transporters.
But to the extent we're discussing it as technology we could potentially have, and/or may want to make happen, I feel it's worth pointing out that transforming feedstock is a better design, doesn't require mastery of bulk antimatter processing, and is perhaps closer to possibility than one would otherwise assume.
Hell, I'd go as far as saying that there are problems with reasonable "attacks"[0] on the road to a replicator - engineering, chemical and biotech problems we could be working on right now.
-- [0] - In the Hamming sense, see https://www.cs.virginia.edu/~robins/YouAndYourResearch.html.