With all the technical debt associated with it, which is the problem basing your project on a dependency that would allow you to easily scale and add features is a huge benefit.

This is like saying you should roll your own crypto because you only need to do a very limited sub set of crypto operations so why use something like NaCl or Tink.

Encryption is a terrible edge case. If you are forced to half-ass encryption, you should seriously question the project requirements. Bad encryption can be worse than none at all. Things won't end well if data security is treated as a detail.