codeordie

I think it really depends on the initial state of the code, and on who is doing the refactoring.

If this code base was in good shape to start with, this is probably just illustrating that there is a point of diminishing returns on refactoring.

It's not bad, and Apache Spark adds SQL on top of some other nosql DBs, but they are still hard to use.

Cassandra for example: you can't just put a secondary index on a table to enable sorting on it. There are secondary indexes, but they aren't useful for that (they don't perform well on high cardinality data).

Want a different sort? Make a new table. And duplicate the data, since there are no JOINs.

I find this really cumbersome.

Ha - same thing happened to me!

I think you are spot on - I've done a lot of working from home, but it's by far the most satisfying when there's a chat room (or IRC) system setup. It makes it fun, I feel more connected, and I think we are more productive that way.

For scraping, how does this compare to pjscrape? (http://nrabinowitz.github.com/pjscrape/)