I've lost contact with many of these after finishing my university, but they are still pretty common terms around me.
Function composition is a pretty trivial concept, but I referred specifically to using ∘ to represent it; this symbol isn't IME as widely-understood to the layman as simply using f(g(x)) (by contrast with college-level and higher math, where it's very widely in use).
The gist of my point was that even the smartest quantitatively-inclined people who went to my private school in a tony part of California aren't likely to have retained familiarity with these concepts unless their college education and/or career involves theoretical math. This is leaving aside people not as smart, or less quantitatively-inclined; my friends who fall into this bucket probably didn't remember what commutativity and associativity were by the time they _entered_ college, let alone later in life. Arguably this latter group wouldn't be part of the target audience interested in reading an article about theoretical math at all, but my friends in the former group certainly would read publications like Nautilus on general mathematical topics.
However that was a long time ago and standards have changed. http://www.corestandards.org/Math/Content/HSN/VM/C/9/ indicates that matrix math is now considered high school material across the USA.
Because they're writing for laymen. I have a degree in math and work in a subfield of CS that relies on math a lot, so I have a lot of practice explaining concepts to some of the smartest people I know without a math background. The tradeoff for your concise expression is that it uses jargon that adds cognitive overhead for people unfamiliar with college-level or higher math, even if they've got a high quantitative intelligence. A non-exhaustive list that would trip up, say, my friend with a PhD in pathology:
1) associative and commutative are not commonly-know terms
2) "A group is a collection of elements 'with' multiplication" is underdetermined. What does it mean to be "with"? The notion that multiplication isn't commutative is unintuitive without understanding that the definition of multiplication is flexible here, defined based on the type of the element in the group.
3) Using o as a composition operator is something that a layman is guaranteed never to have come across
4) What does invertible mean? It's not trivially obvious to a layman that some matrices A have no matrix B such that AB = I.
Etc etc etc. Note that each of these gaps and confusing points (for laymen) compound; I'll occasionally read a paper in a field I'm not an expert in (like economics), and the being bogged down every other sentence by confusion is a severe impediment to understanding how these pieces are composed into something novel. Hell, even when I understand each piece, sometimes the most foreign concepts haven't marinated long enough for me to use them as building blocks yet.
Whats your complaint here? "I have a math degree, how dare nautilus target anyone but me with their articles?"