I mapped almost every USA traffic death in the 21st century

Re:I mapped almost every USA traffic death in the 21st century by et-al on Friday July 19, 2024 @11:42PM (#41013849)

Not to sure how your backend is set up, but it looks like you're generating GeoJSON on the fly and the JSON serialization for this amount of data is going to be slow.

In typical HN fashion, I would suggest looking into Tippecanoe to generate some vector tiles of the data and host that on S3. You can then have the DB perform simple lookups via the accident ID.

https://github.com/mapbox/tippecanoe

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by Bencarneiro on Friday July 19, 2024 @11:51PM (#41013886)

This is very helpful

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by wcedmisten on Saturday July 20, 2024 @12:06AM (#41013952)

I've built quite a few open source mapping projects myself using Tippecanoe and other tools. Feel free to email me if you want help setting this up!

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by woudsma on Saturday July 20, 2024 @02:37AM (#41014571)

You could also take a look at PostGIS to do geospatial queries (if needed). And in the past I’ve used node-mapnik to render vector/image map tiles. You can indeed host and cache them effectively on S3.

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by et-al on Monday July 22, 2024 @11:37AM (#41035653)

Hi Ben, briefly took another look this morning and I'm sorry for the hasty rec because I forgot Leaflet doesn't support vector tiles as easily as MapLibre/Mapbox.

If the bottle-neck is the DB and you haven't enabled the PostGIS extension, I'd look into that as the other commenter mentioned. If it's your server, then yes look into pre-generating all the tiles and hosting them elsewhere, but this might then mean switching over to MapLibre/Mapbox (which is what most companies end up using because of Leaflet's limitations).

For filtering on the frontend, I'd look into Expressions so you're not regenerating the GeoJSON collection every time (this causes a significant delay and flicker).

- https://maplibre.org/maplibre-style-spec/expressions/

- https://maplibre.org/maplibre-gl-js/docs/examples/hover-styl...

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by ppbjj on Friday July 19, 2024 @11:13PM (#41013745)

godspeed, soldier

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by NwtnsMthd on Friday July 19, 2024 @11:46PM (#41013867)

> I mapped almost every USA traffic death in the 21st century

Is your server on that list?

j/k, I’m sorry, I’ll see myself out XD

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by mastersummoner on Saturday July 20, 2024 @12:21AM (#41014009)

I'm not a fan of cheap puns, but that was excellent.

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by kazinator on Saturday July 20, 2024 @12:04AM (#41013941)

Postgres didn't look both ways before stepping off the curb.

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by winrid on Saturday July 20, 2024 @05:03AM (#41015138)

Remember to increase the buffer pool size to use that ram!

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by munch117 on Saturday July 20, 2024 @06:27AM (#41015437)

I was trying out different time periods (comparing 2003-2013 to 2013-2023, to see a rough trend), and I noticed that every click to change the year by one seemed to generate a refresh. Perhaps it would help to lower the load if you change the date selector widget to only refresh after the date selector is closed.

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by carderne on Saturday July 20, 2024 @10:39AM (#41016804)

You’ve had some other recommendations already but I’d suggest also looking into FlatGeobuf [0] for this use case. Have a look at the MapLibre example with a 12GB example [1]. You don’t need a server at all (unlike MBTiles) and will be able to load far more points at once than your current solution. Can easily generate it with QGIS/GDAL/PostGIS. Not sure what your plans are for the project but would be happy to donate some time to get something like that working.

[0] https://github.com/flatgeobuf/flatgeobuf

[1] https://flatgeobuf.org/examples/maplibre/large.html

[ Parent | on HN ]

Re:I mapped almost every USA traffic death in the 21st century by sabman on Sunday July 21, 2024 @12:28AM (#41022374)

Apologies for the shameless plug but do checkout https://geobase.app We built it with such use cases in mind and solves the scaling issues with geo data driven apps. We are in private beta but happy to give you early access if you hit us up.

We were also recently featured on the motherduck blog https://motherduck.com/blog/pushing-geo-boundaries-with-moth...

[ Parent | on HN ]