Postgres built-in search features are really for large document searching -- we've tried using it for indexing e.g. an array of tags for auto-complete and the search times were in the ~500ms+ range, as a matter of fact the non-indexed regex search was only slightly slower. I thought the later was rather surprising - but asking on the mailing-lists, I was told that this was basically working as intended.
edit: I don't fully understand the value-proposition of Algolia either, most of the trouble with going outside of your production database for search - is the problem of keeping your data in sync and current with the production database; compared to that setting up an Elasticsearch cluster really isn't such a big deal, there are also half-way solutions, for example Amazon has an Elasticsearch-service.
Textual relevance is a very complex domain and the Postgres's built in text search features is a simple keyword matching engine compared to Algolia engine that contains a lot of alternative matching. The mesure of textual relevance is also very different of what you have in the Postgres text search.
At the end, this is not only about speed but mainly about relevance
I already have Postgres search queries running on the order of 1-3ms for large queries, basically faster than Algolia, with seemingly the same feature sets.
I don't understand the value proposition over something like Postgres, for sites that already have Postgres databases. (And I'm sure other databases might offer similar built-in search capabilities, I just don't know those other RDBMSs)