Ideally, everything should be testable or replaceable. The non testable parts of your code have network effects on the testability of other parts of the code that are dependent on those parts. This is where dependency injection (e.g via a constructor or function parameters) really shines.

I totally agree that if you have to choose (and usually you do, unless you have unlimited time and funding), choose to implement end to end integration tests first.