One way to do it will be to get OpenStreetMap at least the road and path database. The quality of it vary at lot by cities/countries. The easy way, is to set a reasonable buffer for you GPX points (dunno, let's say 10m at least) and check if they intersect with the roads/path/street they connect to. It is a bit like your way of filtering (and probably a bit like Google must do it : Check around infrastructure and filter based on the probability to be on that road/path/poi between the two points before and after.)

If you want to go the extra miles, I remember this talk of Ilya Zverev "Hundred thousand rides a day"[1] seen at FOSDEM 2019.

[1] https://archive.fosdem.org/2019/schedule/event/geo_gpxtraces...