In my case, if i write a public-facing API, i never trust the caller will do the correct thing, thus the caller must provide all the critical parameters, and i must validate all of them.
At the boundary where untrusted input becomes trusted. This is commonly known as the sanitisation/validation layer.
If you mean ceil(inlen * 4/3) + 1 (even if you use exact arithmetic), then you are passing too small a buffer and you will overflow. Which, I think, proves my point.
You should not need to read the manpage of a function you are calling in order to determine whether you have passed a large enough buffer to avoid overflow.
Reading the manpage to get the function to not return an error would be a different story.
If you let untrusted input reach inlen and overflow that computation, you have already failed:
...on a system where ints are 32 bits. The bottom line is that this doesn't appear to be easily exploitable and IMHO it should be the callers of EVP_EncodeBlock which need to be inspected to determine the actual risk.
using a nontrivial formula
It's base64. 4/3 expansion; round it up and add padding. What's "nontrivial" about that?
and the caller does not pass in the output buffer length, so the implementation cannot validate it
Not necessary, because the caller is responsible for allocation.