Re:Terracotta: Serverless GeoTiff Tile Server
by Waterluvian on Monday July 26, 2021 @07:47AM
(#27958274)
For tile raster services, the story for caching already rendered tiles to a CDN needs to be first class. The Lambda portion is really just the minor detail: the thing that generates tiles if thy haven’t been generated yet.
But depending on your zoom levels and map size, ideally you don’t even have a lambda service and just precache tiles. Then you’ve reduced your service to a dead simple file host problem.
Re:Terracotta: Serverless GeoTiff Tile Server
by dionhaefner on Monday July 26, 2021 @07:51AM
(#27958306)
That is true to some degree, but with Terracotta we allow the frontend to do manipulations on the fly. Like changing contrast, applying colormaps, or band math. So the number of possible tiles is infinite.
If you don’t need that, you can just pre-generate all PNGs - then you don’t even need a server anymore, just a hash function to identify the right file and a big disk.
Re:Terracotta: Serverless GeoTiff Tile Server
by dionhaefner on Tuesday July 27, 2021 @06:34AM
(#27970812)
Only if the client has access to the full bit depth. The rendered PNGs are uint8, but the underlying raw data is often stored in 16 or even 32 bit precision.
Re:Terracotta: Serverless GeoTiff Tile Server
by jhgb on Tuesday July 27, 2021 @07:59AM
(#27971392)
I imagine that with HTTP range requests, it shouldn't be hard to access original TIFF data. Javascript or WebAssembly can then work with arbitrary bytevectors.
Re:Terracotta: Serverless GeoTiff Tile Server
by dionhaefner on Tuesday July 27, 2021 @09:36AM
(#27972250)
Yes, it’s possible. There are some proofs of concept for that IIRC. If you prefer to write everything in Javascript you can do that and do all heavy lifting in the client (plus some extra work to provide indexing and metadata like Terracotta).
Re:Terracotta: Serverless GeoTiff Tile Server
by houzi on Monday July 26, 2021 @05:54AM
(#27957543)
This is sorely missed in the GIS world: A well written and simple tool that isn't over engineered trying to cover all sorts of fringe use cases. Very clean code and good design at first glance.
I get why rasterio is preferred, its API is less annoying, but I'm not sure putting a leaking abstraction over gdal helps much. It just creates an aluring veneer over gdal, without really solving the main problems of gdal, being its monolithic monstrosity of hardly portable yet highly necessary GIS tool functionality.
Doesn't matter too much with gdal's warts, as this server should be deployed with ansible or in a container anyways, so portability is handled in a somewhat sane manner.
Re:Terracotta: Serverless GeoTiff Tile Server
by dionhaefner on Monday July 26, 2021 @07:01AM
(#27957985)
I think the point was that Terracotta might be better off using GDAL directly instead of via rasterio.
Maybe, but I think mapbox is doing a great job with rasterio, and so far we didn’t run into any unsolvable problems (although we got close sometimes). I think for the ease of installation alone rasterio is a blessing.
Re:Terracotta: Serverless GeoTiff Tile Server
by houzi on Monday July 26, 2021 @08:17AM
(#27958485)
Yes. Sorry for the ambiguous comment.
Whenever I see a GIS application, I'm always curious to see how they solve the gdal problem.
The best is always no GDAL, but that gets pretty hard to do quite quickly as GDAL contains some really great algorithms for dealing with non-trivial raster processing.
Re:Terracotta: Serverless GeoTiff Tile Server
by bottled_poe on Monday July 26, 2021 @09:40AM
(#27959215)
Does this project comply with any WebGIS standards for tile-based APIs? I couldn’t see this in the README or intro docs. Would be a shame for a project like this to miss out on uptake for only this reason.
Re:Terracotta: Serverless GeoTiff Tile Server
by anakaine on Monday July 26, 2021 @06:55AM
(#27957942)
Looks good!
Any chance you might consider adding NetCDF support? Used in so many places, contains time, but currently requires a behemoth like geoserver to share as a service
Re:Terracotta: Serverless GeoTiff Tile Server
by dionhaefner on Monday July 26, 2021 @06:58AM
(#27957962)
Author here. Not really, since we rely on cloud-optimized GeoTiffs for performance, in particular their overviews. This makes it so we can serve tiles from huge datasets (10s of GB) in less than 100ms. I don’t think that would be possible with NetCDF.
But depending on your zoom levels and map size, ideally you don’t even have a lambda service and just precache tiles. Then you’ve reduced your service to a dead simple file host problem.
If you don’t need that, you can just pre-generate all PNGs - then you don’t even need a server anymore, just a hash function to identify the right file and a big disk.
Surely these are examples of things that a client could do with raster data independent of the capabilities of a server?
I get why rasterio is preferred, its API is less annoying, but I'm not sure putting a leaking abstraction over gdal helps much. It just creates an aluring veneer over gdal, without really solving the main problems of gdal, being its monolithic monstrosity of hardly portable yet highly necessary GIS tool functionality.
Doesn't matter too much with gdal's warts, as this server should be deployed with ansible or in a container anyways, so portability is handled in a somewhat sane manner.
Thanks for open sourcing this!
Maybe, but I think mapbox is doing a great job with rasterio, and so far we didn’t run into any unsolvable problems (although we got close sometimes). I think for the ease of installation alone rasterio is a blessing.
> I think for the ease of installation alone rasterio is a blessing.
How does rasterio simplify installation, when you still have to deal with gdal's C extension dependencies?
Whenever I see a GIS application, I'm always curious to see how they solve the gdal problem.
The best is always no GDAL, but that gets pretty hard to do quite quickly as GDAL contains some really great algorithms for dealing with non-trivial raster processing.
Any chance you might consider adding NetCDF support? Used in so many places, contains time, but currently requires a behemoth like geoserver to share as a service
It's not :-)