Show HN: Gribstream.com – Historical Weather Forecast API

67 points · ElPeque · 1 days ago

Hello! I'd like share about my sideproject https://gribstream.com

It is an API to extract weather forecasting data from the National Blend of Models (NBM) https://vlab.noaa.gov/web/mdl/nbm and the Global Forecast System (GFS) https://www.ncei.noaa.gov/products/weather-climate-models/gl... . The data is freely available from AWS S3 in grib2 format which can be great but also really hard (and resource intensive) to work with, especially if you want to extract timeseries over long periods of time based on a few coordinates. Being able to query and extract only what you want out of terabytes of data in just an http request is really nice.

What is cool about this dataset is that it has hourly data with full forecast history so you can use the dataset to train and forecast other parameters and have proper backtesting because you can see the weather "as of" points in time in the past. It has a free tier so you can play with it. There is a long list of upcoming features I intend to implement and I would very much appreciate both feedback on what is currently available and on what features you would be most interested in seeing. Like... I'm not sure if it would be better to support a few other datasets or focus on supporting aggregations.

Features include:

- A free tier to help you get started - Full history of weather forecasts - Extract timeseries for thousands of coordinates, for months at a time, at hourly resolution in a single http request taking only seconds. - Supports as-of/time-travel, indispensable for proper backtesting of derivative models - Automatic gap filling of any missing data with the next best (most recent) forecast.

Please try it out and let me know what you think :)

33 comments

drusenko · 23 hours ago

I was initially very excited because this data is not nearly as it should be, especially historical forecasts. However, your pricing model seems to seriously limit the potential uses.

I would imagine that most people who have a serious interest in weather forecasting and would be target users of this service don’t think in terms of number of points but rather in lat/lon bounds, resolution, and number of hours & days for the predictions. I imagine they would also like to download a GRIB and not a CSV.

Your pricing for any large enough area to be useful presumably gets somewhat prohibitive, eg covering the North Pacific (useful for West Coast modeling) at 0.25 deg resolution might be ~300k data points per hour if I am doing my math right?

Show replies

Upitor · 1 days ago

In my experience, NWP data is big. Like, really big! Data over HTTP calls seems to limit the use case a bit; have you considered making it possible to mount storage directly (fsspec), and use fx the zarr format? In this way, querying with xarray would be much more flexible

Show replies

greggsy · 1 days ago

There used to be a really nice website that allowed you to scroll and scale years worth of historical weather for any location. Also allowed you to prepare average annual graphs.

It fell into disrepair after Flash was killed off, and the maintainer wasn't able to commit time to porting it over to a new platform.

I figure that the commodification of weather data is the real reason why it hasn't been replaced with a viable alternative.

Show replies

alexose · 1 days ago

This is great! I've always wanted to backtest forecast models, but I was stuck on where to find good historical data.

Show replies

antasvara · 1 days ago

Just a heads up that there is a similar Python package for this that is free called Herbie [1], though the syntax of this one looks a little easier to use.

[1]: https://herbie.readthedocs.io/en/stable/

Show replies