The objection here is more structural than technical. The famous dropbox objection is 'anyone could do this' - even though they might not have the wherewithal to do so. The objection here is that the open source project is relying on a closed source paid service to do all the heavy lifting. Someone is going to need to foot the bill, which means this project will eventually have to answer some tough questions about funding, and what the project actually delivers.

Whisper is open source.

Where can I download the source from?

This is not open source. The wrapper may be, but it's using a non open source cloud service.

This thread is about the wrapper, which is open source.

You can run Whisper locally, and it is open source.

Feel free to fork this open source project and adapt it to a locally run Whisper instance.