News for hackers. Stuff that matters.
Re:Announcing Google Refine 2.0, a power tool for data wrang... by thejefflarson on Wednesday November 10, 2010 @09:37PM (#1892644)
So I'm on ProPublica's web team -- the organization mentioned in the first video -- and we deal with the types of messy data Refine is made for on a day to day basis.

We've been using it pretty much daily for about 5 months now and cleaning messy government data used to be time consuming and destructive, with google Refine it's so easy and fast to join, cleanup and do rudimentary analysis on said data.

It especially shines when you have to merge many disparate data sets into one. My colleague, Dan Nguyen, did just that for our Dollars for Doctors app:

http://projects.propublica.org/docdollars/

and he scraped the data from reports like this:

http://www.pfizer.com/responsibility/working_with_hcp/paymen...

(one company even put the disclosures up as a flash movie).

Of course we could write scripts, use grep/awk/sed or import it into a database, but Refine is really it. I encourage you to give it a try if you have questionable data you'll need to clean.

Re:Announcing Google Refine 2.0, a power tool for data wrang... by brown9-2 on Thursday November 11, 2010 @08:04AM (#1893691)
Would you and your team be willing to write up a quick intro or howto or article about how you're using the tool? Some real-life scenarios and examples might be very useful.

Also, you guys do great work - keep it up!

Re:Announcing Google Refine 2.0, a power tool for data wrang... by thejefflarson on Thursday November 11, 2010 @11:50AM (#1894324)
I think we have 2 posts in the hopper about it, keep an eye on the nerd blog:

http://www.propublica.org/nerds

Re:Announcing Google Refine 2.0, a power tool for data wrang... by nswanberg on Wednesday November 10, 2010 @10:09PM (#1892729)
Thanks for your work.

How did you deal with the Flash content? Decompile the source code? Did you encounter tabular PDF data? If so, did you find a good solution?

Also, have you or your colleagues had any contact with the Wolfram Alpha team? It seems like your organizations have similar data curation goals.

http://blog.stephenwolfram.com/2010/10/the-emerging-computat...

Re:Announcing Google Refine 2.0, a power tool for data wrang... by thejefflarson on Wednesday November 10, 2010 @10:15PM (#1892744)
I didn't grab the flash content, but if I remember correctly, it was a flash movie that wrapped a PDF that Dan then OCRed and cleaned up with Refine. The coolest part was that the pdf was in grid form, so Dan wrote an ImageMagick script that split it into individual cells and then OCRed each cell (for better results).

EDIT: We haven't had any contact with Wolfram|Alpha but maybe we should reach out.

Re:Announcing Google Refine 2.0, a power tool for data wrang... by poincare on Wednesday November 10, 2010 @09:52PM (#1892697)
How do you guys make money?

Edit: I see that you are a nonprofit organization from your "About" page.

Re:Announcing Google Refine 2.0, a power tool for data wrang... by thejefflarson on Wednesday November 10, 2010 @10:15PM (#1892745)
Yeah, non-profit, mostly foundation grants.