It's easy to poke at Java for being a hog when in reality its just poor coding and operating practices that lead to bloated runtime behavior.
After spending 4 days trying to diagnose a problem with hbase given the two errors "No region found" and "No table provided" and finally figuring out it was due to a version mismatch I now believe it is the culture.
At the very least you should be printing a WARN when you connect to an incompatible version.
I've never seen another community that would actually have a builder for the configuration for the factory for the settings for a class.
If you want bad logging, look at most PHP projects...
For really thoughtful logging, the Apache HTTP client and HikariCP are good Java examples.
Both were written in Java.
And don't get me started on Forte (developed by Sun itself, no less). It was even slower and more memory-hungry than JBuilder.
Sorry, but you just made my day. :P
You could work around it by using shared memory regions and the like but then you're doing a lot of extra work.
With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily.
Also, depending on the dataset in question you might find that things shrink. The latest HotSpots can deduplicate strings in memory as they garbage collect. If your dataset has a lot of repeated strings then you effectively get an interning scheme for free. I don't know if G1 can really work well with over 1TB of heap, though. I've only ever heard of it going up to a few hundred gigabytes.
The JVM has crashed on me in the past (as in hard crash, not a Java exception). Less often than the C++ programs I write do? Yes, but I of course I wouldn't test a program on a 1TB dataset before ironing out all the kinks.
>The latest HotSpots can deduplicate strings in memory as they garbage collect
Obviously when working with huge datasets I would implement some kind of string deduplication myself. Most likely even a special string class an memory allocation scheme optimized for write-once, read-many access and cache friendliness.
Or I would use memory mapping for the input file and let the OS's virtual memory management sort it out.
But the real impact is if you want to be mutating that data set. The default behaviour of "tear down the entire process on error" can of course be worked around even if you ignore data corruption errors, but not having to do things by hand is the point of managed runtimes in the first place.
IIRC, the images for the site were closer to 7-8TB, but I don't know how typical that is for other types of sites, and caching every image on the site in memory is pretty impractical... just the same... damn.
The reason is so if you fuck up a scaling script for example you can't launch 1000 machines and take all the capacity and then bitch that you won't pay for it.
It's a stop gap.
However, aside from the hard limit of 100 S3 buckets, all other limits are configurable at the request of your AWS rep
https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3...
If practice in recent decades has taught us anything, it's that performance is found in intelligently using the cache. In a multi-core concurrent world, our tools should be biased towards pass by value, allocation on the stack/avoiding allocating on the heap, and avoiding chasing pointers and branching just to facilitate code organization.
EDIT: Or, as placybordeaux puts it more succinctly in a nephew comment, "VM or culture? It's the culture."
EDIT: It just occurred to me -- Programming suffers from a worship of Context-Free "Clever"!
Whether or not a particular pattern or decision is smart is highly dependent on context. (In the general sense, not the function call one.) The difficulty with programming, is that often context is very involved and hard to convey in media. As a result, a whole lot of arguments are made for or against patterns/paradigms/languages using largely context free examples.
This is why we end up in so many meaningless arguments akin to, "What is the ultimate bladed weapon?" That's simply a meaningless question, because the effectiveness of such items is very highly dependent on context. (Look up Matt Easton on YouTube.)
The analogy works in terms of the degree of fanboi nonsense.
Still really cool to see something like this, I didn't even know you could get close to 2TB of ram in a single server at any kind of scale.
http://www.cray.com/blog/the-power-of-512-terabytes-of-share...
http://www.enterprisetech.com/2014/10/06/ibm-takes-big-workl...
Or significantly higher if you don't restrict yourself to single-system-image, shared memory machines - there are at least 2 1300-1500 TB systems on the Top 500 list.
Do a little research before implying that there's no way that Java can address gigantic heaps.
Scala _beats_ Java in most of the benchmarks: http://benchmarksgame.alioth.debian.org/u64q/scala.html