News for hackers. Stuff that matters.
Re:A new way to make maps with OpenStreetMap by Tomte on Friday April 23, 2021 @03:30PM (#26918288)
> Existing formats like MBTiles are based on SQLite. This limits the ability to self-host maps to those confident running a server in production.

Is that comical, or is there a deeper meaning I don't understand?

Re:A new way to make maps with OpenStreetMap by incanus77 on Friday April 23, 2021 @04:40PM (#26918974)
Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting.

Source: I created MBTiles ;-)

Re:A new way to make maps with OpenStreetMap by bdon on Friday April 23, 2021 @07:56PM (#26920685)
Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles.

Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.

Re:A new way to make maps with OpenStreetMap by TheTaytay on Friday April 23, 2021 @11:32PM (#26922026)
When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
Re:A new way to make maps with OpenStreetMap by bdon on Friday April 23, 2021 @11:42PM (#26922079)
That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
Re:A new way to make maps with OpenStreetMap by TheTaytay on Friday April 23, 2021 @11:54PM (#26922126)
Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
Re:A new way to make maps with OpenStreetMap by incanus77 on Friday April 23, 2021 @11:45PM (#26922088)
That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
Re:A new way to make maps with OpenStreetMap by TheTaytay on Friday April 23, 2021 @11:59PM (#26922145)
Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.
Re:A new way to make maps with OpenStreetMap by incanus77 on Friday April 23, 2021 @09:18PM (#26921293)
Neat! I'm hoping to dig into your design more... looks great.
Re:A new way to make maps with OpenStreetMap by codetrotter on Friday April 23, 2021 @03:35PM (#26918329)
Probably what they mean is that they want to be able to host the maps as static files that are served to the web.

So that you can FTP some html, js, css, images and map data files to a web host and call it a day.

Re:A new way to make maps with OpenStreetMap by rakoo on Friday April 23, 2021 @04:12PM (#26918695)
AFAIU MBTiles still need some application server to transform x/y/z into proper tiles to be served, which means provisioning a server, scaling it, maintaning it, and all the fun that comes with it; PMTiles is a simple blob that the frontend will download (only desired parts) and render directly in the browser.
Re:A new way to make maps with OpenStreetMap by alohec on Friday April 23, 2021 @05:32PM (#26919455)
You can use tippecanoe (https://github.com/mapbox/tippecanoe) with the --output-to-directory flag to output the individual .pbf tiles instead of a single MBTiles file. These tiles can then be hosted on something like s3 and your map-rendering client can query the necessary tiles, no server required.
Re:A new way to make maps with OpenStreetMap by MrMartian on Friday April 23, 2021 @07:00PM (#26920250)
So I fee like it actually missed the mark as to why it's so impactful. When you get to global scale tiling with building accuracy, you literally end up with hundreds of millions of tiles. The upload time / cost per upload STACKS. However, the tilesets combined usually sits between 100GB and 200GB. ONE upload of ~150GB seems is faster and more cost efficient via S3.
Re:A new way to make maps with OpenStreetMap by jleedev on Friday April 23, 2021 @06:57PM (#26920228)
For comparison, Cloud Optimized GeoTIFF (COG) is specifically designed to be readable with HTTP Range requests, so the single file can be placed on S3 and be immediately usable.

As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.

Re:A new way to make maps with OpenStreetMap by leokennis on Friday April 23, 2021 @03:34PM (#26918317)
I read it as: existing solutions are too hard for simple people with less technical skill.
Re:A new way to make maps with OpenStreetMap by bdon on Friday April 23, 2021 @10:11PM (#26921627)
Author here; a narrower way to express what I meant:

A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles:

* Need a virtual machine and disk space

* Need to choose a particular program to read the MBTiles and serve it via HTTP

* Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7

* Need to deal with SSL termination

The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.