It checks that you get a signed assertion back, which is valid, and lets you pull out attributes. That's great, but there are a gajillion tags that can be set within an assertion that have genuine, important security implications and zero of them are handled by this library.
If your IdP sends those tags along with the assertion, this library will happily ignore them unless you implement support yourself. And I would bet heavily that approximating zero of this library's users write that support.
One trivial example, there's a `<saml:Conditions>` tag. This tag can have all sorts of attributes like `NotBefore` or `NotOnOrAfter` that specify a time period for which the assertion is valid. This library does not do anything to implement support for `<saml:Conditions>` tags at all, so every consumer of this library will happily process assertions as valid even if they're outside of the duration for which the IdP is claiming it's valid. If I get my hands on an assertion at some point in time, I can hold onto it and replay it forever to unsuspecting users of this library. There are other attributes that can set conditions, and there are other tags with implications on whether or not an assertion should be considered valid. None of them are handled.
Further, only minimal document validation is performed. The signature is checked, but the Issuer is not. Multiple Issuers can share a signing key, though I suspect actual cases of this are rare. At any rate, there's a billion ways users of this library can blithely chug along with an assertion that isn't valid and shouldn't be trusted, but nobody notices or cares because the happy path works.
With authentication, yes it's important that legitimate users are able to sign in. But it's as important that illegitimate authentication attempts are disallowed. And this half is completely forgotten about in literally 100% of the common SAML service provider libraries I've seen in the wild (which is admittedly not 100% of the ones available, just the ones I've looked at).
Edit: As @tptacek points out, every user of this library is trivially susceptible to AudienceRestriction authz vulnerabilities. An IdP can include a statement in an assertion that limits it to one intended service provider (e.g., "this document authenticates user FOO for Google service BAR"). Users of this library will simply see "this document authenticates user FOO" and authenticate that user, even though this document was never intended for them. Again it's been awhile so I might be mischaracterizing it a bit but that should be the gist.
FWIW, yep. It has no knowledge of AudienceRestriction:
$ grep -i AudienceRestriction Saml.cs
$(note that I don't know anything about SAML except how to configure certain applications for it).
https://github.com/jitbit/AspNetSaml/