Hey HN,I shared GribStream here a while ago when it was just a NOAA forecast API. It’s grown into something bigger: a full weather data archive you can query just like a database, by time (with as-of/time-travel!), location, or more interestingly by specific weather conditions.Use built-in expressions directly in your API requests to calculate derived metrics like wind chill, dew point, or...
You can check the example here: https://github.com/GribStream/python-client
https://github.com/GribStream/python-client
A few weeks ago there was a request for GribStream to support this particular model https://www.nco.ncep.noaa.gov/pmb/products/rap/ to use it on Skew-T Log-P diagrams and so after banging my head against jpeg2000 compressed data within grib2 files I finally got it :).I'm sharing a python example to show how to create these charts.A Skew-T Log-P diagram is a type of thermodynamic chart used...
Having all historical weather forecasts is useful to backtest how accurate the model is because you can compare what was being forecasted at the time with what actually happened.
It is also useful to backtest derived models. If you want to figure out how accurate your model is forecasting something like solar power generation while using one of these weather datasets as features for cloud cover and solar radiation, you will want to run the model with the forecast that was current at that point in time. Otherwise you would be "cheating" by using data you wouldn't have had if you ran the model at that time.
In other words, you'd want to "see" the weather as it was being predicted at the time in order to find out what your model would predict for solar power generation.
I hope that makes sense.
But open-meteo free access is only for non-commercial use. GribStream.com allows any use.
Also, can open-meteo query forecasts 10 days out at hourly resolution for 150.000 coordinates in a single request and take just 8 seconds? At what price?
I'll do a benchmark soon.
Out of the top of my head, re-slicing into grib files for the response is probably a big lift but some of the other formats like maybe netCDF or geoTIFF or just compressed array of floats might be a nice MVP.
I tried to tie the pricing with the amount of processing that the API needs to do, which is closely related to the number of grib2 files that the API needs to download and process in order to create the response. And it doesn't change as much wether I extract 1 point or 1000 points. But I thought I had to draw the line somewhere or nobody would ever pay for anything because the freetier is enough.
But I might make it same price for maybe chunks of 5000 or more points.
From the line of business I come from the main usage is actually to extract scattered coordinates (think weather where specific assets are, like hotels or solar panels or wind farms) and not whole boundaries at full resolution but it makes a lot of sense that for other types of usage that is not the case.
It is definitely in the roadmap to be able to select based on lat/lon bounds and even shapes. Also to return data not as timeseries but the gridded data itself, either as grib2 or netCDF or parquet or a plain matrix of floats or png or even mp4 video.
I basically scrape all the grib index files to know all the offsets into all variables for all time. I store that in clickhouse.
When the API gets a request for a time range, set of coordinates and a set of weather parameters, first I pre-compute the mapping of (lat,lon) into the 1 dimensional index in the gridded data. That is a constant across the whole dataset. Then I query the clickhouse table to find out all the files+offset that need to be processed and all of them are queued into a multi-processing pool. And then processing each parameter implies parsing a grib file. I wrote a grib2 parser from scratch in golang so as to extract the data in a streaming fashion. As in... I don't extract the whole grid only to lookup the coordinates in it. I already pre-computed the index, so I can just decode every value in the grid in order and when I hit and index that I'm looking for, I copy it to a fixed size buffer with the extracted data. When You have all the pre-computed indexes then you don't even need to finish downloading the file, I just drop the connection immediately.
It is pretty cool. It is running in very humble hardware so I'm hoping I'll get some traction so I can throw more money into it. It should scale pretty linearly.
I've tested doing multi-year requests and the golang program never goes over 80Mb of memory usage. The CPUs get pegged so that is the limiting factor.
Grib2 complex packing (what the NBM dataset uses) implies lots of bit-packing. So there is a ton more to optimize using SIMD instructions. I've been toying with it a bit but I don't want to mission creep into that yet (fascinating though!).
I'm tempted to port this https://github.com/fast-pack/simdcomp to native go ASM.
Fixed!
Thank you!
The grib2 files are the storage. They are sorted by time in the path and so that is used like a primary index. And then grib2 is just a binary format to decode to extract what you want.
I originally was going to write this as a plugin for Clickhouse but in the end I made it a Golang API cause then I'm less constrained to other things. Like, for example, I'd like to create and endpoint to live encode the gribfiles into MP4 so the data can be served as Video. And then with any video player you would be able to playback, jump to times, etc.
I might still write a clickhouse integration though because it would be amazing to join and combine with other datasets on the fly.
This is an API that will do streaming extraction and only have you download what you actually need.
When you make the http request to the API, the API will be processing up to terabytes of data only to respond to you with maybe a few Kb of csv.
Can you describe how it worked? Who knows, maybe I could make it happen.
I've been thinking of creating a more involved "demo" to entice traffic/links/SEO. Maybe something like this could be it.
Besides using the usual index files to only do http range requests for weather parameters of interest, GribStream also avoids creating big memory buffers to decode/decompress the whole grid. It does the decoding in a streaming fashion and only accumulates the values that are being looked for so it can do so very efficiently. It doesn't even finish downloading the partial grib file, it early aborts. And it also skips ahead many headers and parts of the grib2 format that are not really required or that can be assumed for being constant in the whole dataset. In other words, it cuts all possible corners and the parse is (currently) specifically optimized for the NBM and GFS datasets.
Although I intend to support several others, like the Rapid Refresh (RAP) model.
And the fact that this process runs close to the data (AWS), it can do so way faster than you can run it anywhere else.
This is a little different though.
This dataset is is part of the AWS Open Data program and so it is freely available from S3. By running the API within AWS then you get a massive latency and bandwidth advantage.
So with GribStream you are pushing the computation closer to where the big data is and only downloading the small data. And GribStream uses a custom grib2 parser that allows it to extract the data in a streaming fashion, using very little memory.
It makes a huge difference if you need to extract timeseries of a handful of coordinates for months at a time.
Cheers!
Hello! I'd like share about my sideproject https://gribstream.comIt is an API to extract weather forecasting data from the National Blend of Models (NBM) https://vlab.noaa.gov/web/mdl/nbm and the Global Forecast System (GFS) https://www.ncei.noaa.gov/products/weather-climate-models/gl... . The data is freely available from AWS S3 in grib2 format which can be great but also really hard (and...
A free tier to help you get started Full history of weather forecasts Extract timeseries for thousands of coordinates, for months at a time, at hourly resolution in a single http request taking only seconds. Supports as-of/time-travel, indispensable for proper backtesting Automatic gap filling of any missing data with the next best (most recent) forecast.
It currently leverages the National Blend of Models (NBM) with more datasets to come.
High efficiency and low latency access to GRIB data I'm eager to hear your feedback and suggestions!