News for hackers. Stuff that matters.
Re:IBM to Acquire the Weather Company by Alex3917 on Wednesday October 28, 2015 @12:42PM (#10465554)
Hopefully IBM can fix the weather data industry. It seems ridiculously broken right now that if I want to get a "prediction" of how much it's going to rain next week there are a million free sources, but if I want to see how much it actually rained in the past week then I need to pay thousands of dollars or else figure out how to parse a bunch of obscure and undocumented formats.
Re:IBM to Acquire the Weather Company by cwal37 on Wednesday October 28, 2015 @01:00PM (#10465714)
Hey buddy, I use this resource[1] all the time (and they're actually housed into the building next to me here at work). It's the long-term climate records from monitoring stations across the country, the United States Historical Climatology Network, and you can get daily data going back over a hundred years in some cases.

For example, I've used it to crank out random temperature histograms in the past[2], and even plotted some long-term averages to decide on temperature and precipitation ranges when going on camping trips. It's really useful if you're willing to work with the data a tiny bit. You're getting csv files, so it's not terrible to work with in any way via Python or R, or whichever data-friendly language/tool you want to use.

[1] http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.ht...

[2] http://btus.us/decadal-histograms-of-monthly-yosemite-temper...

Re:IBM to Acquire the Weather Company by Alex3917 on Wednesday October 28, 2015 @01:38PM (#10465979)
So apparently in order to use this I need to enter a NESDIS ID. When I Google that, the first result I get informs me that a NESDIS ID is "an 8-character identifier assigned by NESDIS that becomes the RAWS identification number for use with the GOES satellite." Clicking on 'glossary' in the breadcrumbs then fails to take me to an actual glossary.

I realize that it's theoretically possible to get this working. But it's not easy. What I want is a way to enter a postal code and get a JSON response with the precipitation for the last week. Without having to convert to scientology, learn GeoDjango, license GIS software, read an 800 page manual on parsing a bytestream from a satellite, etc.

Re:IBM to Acquire the Weather Company by cwal37 on Wednesday October 28, 2015 @01:56PM (#10466121)
Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled down the station stuff I need by hand (since you can get all the station's data at once), or by some lightly mechanized webscraping. Your comment sounded significantly more chill than what it seems like you might need.

It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this.

Does this page[1] not contain what you need?

[1] http://cdiac.ornl.gov/ftp/ushcn_daily/

Re:IBM to Acquire the Weather Company by tcdent on Wednesday October 28, 2015 @02:10PM (#10466194)
*.txt.gz is not exactly a modern data format.

I share the parent's sentiment; finding easy access to good weather data is hard.

Do you want your side project to be an experience in learning how to parse obscure formats and coordinating obsolete dependencies, or HTTP GET some JSON and immediately do something with the data?

Re:IBM to Acquire the Weather Company by Eridrus on Wednesday October 28, 2015 @02:33PM (#10466349)
You're shitting me, right?

They're literally giving you all of the data in a plaintext format with documentation, and you're still complaining?

Sure, having everything handed to you on a silver platter is nice, but why should they do your project for you?

Re:IBM to Acquire the Weather Company by spiralpolitik on Wednesday October 28, 2015 @05:02PM (#10467379)
Imagine what is going to happen when they discover the entire financial system is built on copying CSV files around by FTP.
Re:IBM to Acquire the Weather Company by kbenson on Wednesday October 28, 2015 @05:24PM (#10467543)
Don't forget the emails back and forth and manual fixes and resubmissions from either end when there's a problem causing the CSV to not parse!
Re:IBM to Acquire the Weather Company by mikestew on Wednesday October 28, 2015 @04:20PM (#10467103)
> *.txt.gz is not exactly a modern data format.

You know what you get with a format that's not "modern"? A huge, mature set of tooling that works. For instance, without trying because just responding to this whinging is barely worth the wear on my keyboard, I'd be disappointed if OS X wouldn't cheerily open the file, and unarchive it into the original directory structure simply by typing "open weather_data.txt.gz" from a terminal window. In other words, one could script it. I'd show you how, but holding your hand leaves me with only one hand for typing.

> HTTP GET some JSON and immediately do something with the data

Well, that's the magic of JSON: any code I write just wondrously knows what to do it without me telling it how to parse it. How can poor ol' ASCII text possibly compete?

Re:IBM to Acquire the Weather Company by karlshea on Wednesday October 28, 2015 @09:02PM (#10468561)
You don't even need the terminal, just double click the file.
Re:IBM to Acquire the Weather Company by mikestew on Wednesday October 28, 2015 @09:20PM (#10468617)
Can't script a double-click in Finder. :-) (Well, there's AppleScript...). But, yeah, point taken.
Re:IBM to Acquire the Weather Company by cwal37 on Wednesday October 28, 2015 @02:32PM (#10466341)
Not modern doesn't have to mean difficult to parse, it depends on your personal outlook. You can also download individually customized csv files if you want, which my original link led to. csv and txt files seem eminently reasonable to me when dealing with a huge historical cache of data that's being put out by a public institution for free.
Re:IBM to Acquire the Weather Company by mvf0 on Wednesday October 28, 2015 @02:41PM (#10466397)
I could write a node server that converts all that fixed length ASCII data into JSON faster than you guys can complain.
Re:IBM to Acquire the Weather Company by nl on Wednesday October 28, 2015 @06:42PM (#10467928)

  wget -O - http://cdiac.ornl.gov/ftp/ushcn_daily/state01_AL.txt.gz | gunzip -c |  in2csv -s schema.txt | csvjson
Fixed that for you. And in less space than it took for you to complain.
Re:IBM to Acquire the Weather Company by ska on Wednesday October 28, 2015 @03:23PM (#10466666)
What entitled nonsense. *.txt.gz is not a problem for anyone of even minimum competence.
Re:IBM to Acquire the Weather Company by tcdent on Wednesday October 28, 2015 @04:27PM (#10467141)
Wow, you guys really missed the point.

Why can't we ask for nice things, even if they are "free"?

How are you going to get a kid interested in programming historical weather analysis when he's got to figure out how to configure an entire stack just to query your dataset. If you really want my opinion, this should be an AI problem, not yet another tedious MVC application.

Furthermore, when I jump into projects, I am most definitely appreciative that there is a responsive system on the other end, no matter how much I'm getting paid. I'm not some noob and yes I have imported gigabytes of minimally-structured data into my own applications, but I don't want to have to do that every time. Especially just to tinker.

If you're not thinking about how your software development experience can be made quicker, more direct, and less painful, I pity you. Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day.

Re:IBM to Acquire the Weather Company by kaishiro on Wednesday October 28, 2015 @04:46PM (#10467282)
"Shit on JSON all you want, but I'll take a populated native object and a blinking cursor over FTP'd text files any day."

I'm not sure this particular hill is worth dying for. Remember, text was the universal transport long before JSON was (and probably will be long after).

Re:IBM to Acquire the Weather Company by rgovind on Wednesday October 28, 2015 @04:10PM (#10467028)
Try this http://www.forecastwatch.com/

I once bought data from them for couple of hundred dollars.

Re:IBM to Acquire the Weather Company by jonknee on Wednesday October 28, 2015 @12:50PM (#10465629)
Wunderground has lots of data available. They may have actually been part of this deal as they were owned by The Weather Channel.

http://www.wunderground.com/history/airport/KSFO/2015/10/28/...

Re:IBM to Acquire the Weather Company by ISL on Wednesday October 28, 2015 @12:59PM (#10465699)
International Business Machines on Wednesday announced an agreement to acquire The Weather Co.'s B2B, mobile, and cloud-based web properties, including WSI, weather.com, Weather Underground and The Weather Company brand.

Source: http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weat...

Re:IBM to Acquire the Weather Company by acomjean on Wednesday October 28, 2015 @01:40PM (#10465994)
I use weather underground for my personal server (developer accounts are free up to 500 calls a day...). We used them for work to get weather data for lots of locations hourly ("drizzle" plan). Very reliable.

api: http://www.wunderground.com/weather/api/d/docs

National weather service, http://weather.gov is my goto weather forecaster, but at the time I wrote the code, getting hourly data based on lat/long for several thousand locations (some outside the US) was problematic. and XML..

Re:IBM to Acquire the Weather Company by hangtime79 on Wednesday October 28, 2015 @01:01PM (#10465722)
Re:IBM to Acquire the Weather Company by free2rhyme214 on Wednesday October 28, 2015 @07:54PM (#10468285)
Aww I love this app except their ads. I'm not sure what IBM's plans for this is.
Re:IBM to Acquire the Weather Company by rconti on Wednesday October 28, 2015 @08:24PM (#10468425)
That's what I love about wunderground. I actually have an Ambient Weather ws-1001-wifi weather station at my house, and I'm part of wunderground's weather station network. Whenever I pop open the weather app on my iPhone, it shows the weather in my backyard which I love.
Re:IBM to Acquire the Weather Company by Redoubts on Wednesday October 28, 2015 @06:00PM (#10467739)
That doesn't look free, unless you want to scrape.

http://www.wunderground.com/weather/api/d/pricing.html

Re:IBM to Acquire the Weather Company by rdtsc on Wednesday October 28, 2015 @03:12PM (#10466583)
Part of IBM now as well.
Re:IBM to Acquire the Weather Company by deuill on Wednesday October 28, 2015 @12:50PM (#10465630)
Have you tried using http://forecast.io/? You can get historical forecasts via the JSON API[0], which is, from what I can tell, free for low-volume use.

[0]: https://developer.forecast.io/docs/v2#time_call

Re:IBM to Acquire the Weather Company by tertius on Wednesday October 28, 2015 @01:18PM (#10465862)
I think he's trying to get historical data of actual weather, not forecasts.
Re:IBM to Acquire the Weather Company by drewbug on Wednesday October 28, 2015 @01:41PM (#10466003)
> Please note that we only store the best data we have for a given location and time: in the past, this will usually be observations from weather stations (though we may fall back to forecasted data if we don't have any observations)
Re:IBM to Acquire the Weather Company by Redoubts on Wednesday October 28, 2015 @06:19PM (#10467836)
That looks really nice.
Re:IBM to Acquire the Weather Company by subnaught on Wednesday October 28, 2015 @01:04PM (#10465750)
For the most part, the Weather Company is merely redistributing data collected by the federal government, specifically the National Oceanic and Atmospheric Administration. You can easily access historical rainfall data here: http://www.nws.noaa.gov/oh/hads/
Re:IBM to Acquire the Weather Company by dublinben on Wednesday October 28, 2015 @01:05PM (#10465756)
The National Weather Service makes historical weather data available for free in a standardized format.[0] I'm sure there's already a parser written for any language you might be using.

[0]http://w2.weather.gov/climate/index.php?wfo=mtr

Re:IBM to Acquire the Weather Company by lelandbatey on Wednesday October 28, 2015 @03:43PM (#10466823)
That's true, but that data is for only a few locations.
Re:IBM to Acquire the Weather Company by anderiv on Wednesday October 28, 2015 @11:56PM (#10469185)
Weather Underground makes nearly all climatological data available in easy-to-consume formats (csv mostly). Using your rainfall example, here's how I'd get a listing of rainfall by day for the month of September, from my local National Weather Service monitoring site:

  $ curl -s "http://www.wunderground.com/history/airport/KMSP/2015/09/31/MonthlyHistory.html?format=1" | grep -v Precip | cut -d "," -f 20
Re:IBM to Acquire the Weather Company by vegardx on Wednesday October 28, 2015 @01:44PM (#10466024)
You can actually get all that for free from Norwegian Meteorological Institute, both forecasts and observations. The only caveat is that it's all in Norwegian. Looks like that is a little intentional[1].

[1] http://om.yr.no/verdata/free-weather-data/

Re:IBM to Acquire the Weather Company by alphataraxian on Wednesday October 28, 2015 @03:50PM (#10466879)
Free quality controlled historical data available from NOAA as well:

https://www.ncdc.noaa.gov/data-access/land-based-station-dat...

Re:IBM to Acquire the Weather Company by arikrak on Wednesday October 28, 2015 @01:54PM (#10466103)
I once found a link that showed useful charts for historical weather data including averages and highs and lows. I can't find it right now, but Googling for it brings up a similar site: https://weatherspark.com
Re:IBM to Acquire the Weather Company by feider on Wednesday October 28, 2015 @03:24PM (#10466679)
Sorry for non value adding cynicism, but when has a big corp fixed any industry? Especially IBM? Sure, more competition is welcome, but I would not get my hopes up.
Re:IBM to Acquire the Weather Company by Keats on Wednesday October 28, 2015 @02:34PM (#10466356)
I scraped http://en.tutiempo.net/climate when I was writing the following article: https://blog.wearewizards.io/comparing-the-weather-of-places...

There is also a link to the python script I made to scrape it

Re:IBM to Acquire the Weather Company by exelius on Wednesday October 28, 2015 @02:02PM (#10466156)
I wouldn't be too hopeful; this situation is the entire reason IBM made this purchase.

IBM is in the business of providing data about a whole lot of things. I would expect the cost for access to this data to go up, not down -- the majority people who care about how much it rained in the last week only care because they can make more money by knowing that.

Re:IBM to Acquire the Weather Company by hangtime79 on Wednesday October 28, 2015 @04:28PM (#10467153)
I couldn't disagree more. I bet we actually see a service exposed via Bluemix in the not distant future.
Re:IBM to Acquire the Weather Company by chinathrow on Wednesday October 28, 2015 @01:04PM (#10465752)
Those formats are very well documented and standardised world wide. Commercial use is another thing though...
Re:IBM to Acquire the Weather Company by vram22 on Wednesday October 28, 2015 @03:21PM (#10466649)
Check out products like ForecastWatch from Intellovations. Not sure, but might be of help. Not affiliated, just had conversations with the founder a while ago.

http://www.intellovations.com/products/

Re:IBM to Acquire the Weather Company by PretzelFisch on Wednesday October 28, 2015 @02:57PM (#10466494)
There is Noaa http://w2.weather.gov/climate/ you can get past results there.
Re:IBM to Acquire the Weather Company by huhtenberg on Wednesday October 28, 2015 @12:47PM (#10465604)
Wolfram|Alpha has the data you are looking for.
Re:IBM to Acquire the Weather Company by feider on Wednesday October 28, 2015 @03:28PM (#10466715)
...and operates like North Korea when it comes to openness.