Re:Search Big Data in Five Minutes with Pig, Wonderdog and E...
by rjurney on Tuesday July 10, 2012 @07:57PM
(#4226949)
Lets be clear: Pig powers Yahoo and LinkedIn. Large enterprises. There's no question whether it is production ready.
[ Parent
| on HN ]
Re:Search Big Data in Five Minutes with Pig, Wonderdog and E...
by squarecog on Wednesday July 11, 2012 @01:09AM
(#4227816)
And Twitter.
We unit test our pig scripts, it's pretty straightforward given MockStorage class we have contributed (you can find it in pig trunk). Granted, we've long been separating load statements from actual logic, which allows us to fairly easily mock up data to feed into the Pig flows. It would be harder if your loads and flows are in the same file.
Re:Search Big Data in Five Minutes with Pig, Wonderdog and E...
by matterhayes on Wednesday July 11, 2012 @03:28PM
(#4230976)
At LinkedIn we collected together a bunch of UDFs we developed for pig and released it as the DataFu library. We used PigUnit to unit test each of the UDFs:
https://github.com/linkedin/datafu/tree/master/test/pig/data...