News for hackers. Stuff that matters.
Re:The JVM is not that heavy by gregschlom on Wednesday February 01, 2017 @02:33PM (#13544134)
Also worth noting that the JVM itself only weighs a couple of megabytes. The bulk of the size comes from the Java runtime (ie: the "standard libraries"), and there are lots of things that your app may not need there (XML parsing, serialization, etc...)

A couple of years ago I wrote a simple tool (https://github.com/aerofs/openjdk-trim) that allows you to filter out what you don't need. We were able to get the size of OpenJDK from 100MB down to around 10MB.

Note that the work of determining which classes you need is entirely manual. In our case I used strace to check what classes where being loaded.

Re:The JVM is not that heavy by necubi on Wednesday February 01, 2017 @02:42PM (#13544239)
This is officially supported in JDK 9. It's part of an ongoing project called "jigsaw" which is introducing modules to the JVM. The first step, shipping in 9, is to modularize the JDK [0].

Already today you can build a custom JDK in the early access release.

[0] http://openjdk.java.net/jeps/200

Re:The JVM is not that heavy by kodablah on Wednesday February 01, 2017 @05:11PM (#13545877)
Not that well. The java.base module is still huge. Also, Zulu has nice pre-packaged JDK9 downloads[0] so you don't have to build.

0 - http://zulu.org/zulu-9-pre-release-downloads/

Re:The JVM is not that heavy by the8472 on Thursday February 02, 2017 @01:47AM (#13548634)
I think jlink can strip at the class level, not just the module level.
Re:The JVM is not that heavy by derefr on Wednesday February 01, 2017 @03:00PM (#13544465)
> I wrote a simple tool that allows you to filter out what you don't need

It'd be a short hop from here to a tool that basically does for JDK-platform apps what Erlang's releases do for the ERTS platform: builds a new JRE (as a portable executable, not an installer) that actually contains the app and its deps in the JRE's stdlib, such that you just end up with a dir containing "a JRE", plus a runtime "boot config" file that tells the JRE what class it should run when you run the JRE executable.

With such a setup, your Java program could actually ship as an executable binary, rather than a jar and an instruction to install Java. Nobody would have to know Java's involved! :)

Re:The JVM is not that heavy by gregschlom on Wednesday February 01, 2017 @03:25PM (#13544732)
Funny you mention that, that's exactly why we wrote this tool. We were having a ton of support issues with our Windows users having to install Java, incompatible Java versions, needing admin privileges to install the JRE, etc...

We fixed this whole class of issues by doing exactly what you suggest: bundling the JRE and writing our own launcher binary.

Re:The JVM is not that heavy by mirekrusin on Thursday February 02, 2017 @07:57AM (#13550182)
Does java license interfere with this kind of setup?
Re:The JVM is not that heavy by merb on Friday February 03, 2017 @04:07AM (#13558686)
not when you bundle the openjdk
Re:The JVM is not that heavy by yoden on Wednesday February 01, 2017 @04:33PM (#13545469)
https://docs.oracle.com/javase/8/docs/technotes/guides/deplo...

Not only is it a short hop, it already exists :P

Re:The JVM is not that heavy by andor on Wednesday February 01, 2017 @04:42PM (#13545562)
Nice. Here's the manual page, the relevant flag is "-native":

http://docs.oracle.com/javase/8/docs/technotes/tools/unix/ja...

Re:The JVM is not that heavy by radicalbyte on Thursday February 02, 2017 @12:45AM (#13548405)
The problem with visiting that site is that Oracle have started litigating against users of their JVMs.

In that way the JVM can be "heavy".

Re:The JVM is not that heavy by pjmlp on Thursday February 02, 2017 @02:28AM (#13548791)
With users that use commercial features without paying for them.

It is quite easy to know which features those are when they require a flag named -XX:+UnlockCommercialFeatures, you just don't use them by mistake.

Re:The JVM is not that heavy by geodel on Thursday February 02, 2017 @08:49AM (#13550433)
Oracle gives free license to use commercial features in development[1] in section B. It is very easy to some script for JVM startup parameters and get that script deployed in production by mistake.

1. http://www.oracle.com/technetwork/java/javase/terms/license/...

Re:The JVM is not that heavy by pjmlp on Thursday February 02, 2017 @11:44AM (#13551915)
Quite right, but unless I am too cynical I doubt those deployments were actually done by mistake, when I relate how many companies try to "alleviate" their costs.
Re:The JVM is not that heavy by andor on Wednesday February 01, 2017 @04:36PM (#13545502)
Avian [0] is an embeddable VM in the way you suggest, I haven't used it though.

In the future, we'll hopefully have the Substrate VM [1] for Java and other Truffle-supported languages. It's embeddable and also does reachability analysis to exclude unused library code. For now, it seems to be closed source.

[0] http://readytalk.github.io/avian/

[1] http://lafo.ssw.uni-linz.ac.at/papers/2015_CGO_Graal.pdf

Re:The JVM is not that heavy by marklgr on Thursday February 02, 2017 @02:06AM (#13548702)
> what Erlang's releases do for the ERTS platform

That's the difference between an industrial-strength platform like Erlang, and a dev-centric deployment nightmare like Python. Java is normally on the enterprise side of the spectrum, but unfortunately it didn't get deployment right for quite a long time, even though it appears it's getting there lately.

Re:The JVM is not that heavy by pjmlp on Thursday February 02, 2017 @02:27AM (#13548787)
It started on Java 8 with adoption of having a packing story as part of the JDK tooling, but I don't know how good it is.
Re:The JVM is not that heavy by eru on Thursday February 02, 2017 @07:09AM (#13549956)
Careful there---it's a slippery slope all the way to unikernels.

https://mirage.io/

Re:The JVM is not that heavy by jjtheblunt on Wednesday February 01, 2017 @04:05PM (#13545167)
Isn't that _exactly_ what GoLang does for its binaries, insofar as shipping them statically linked?
Re:The JVM is not that heavy by dvlsg on Wednesday February 01, 2017 @04:14PM (#13545263)
dotnetcore has it as an option, as well.
Re:The JVM is not that heavy by mahmud on Wednesday February 01, 2017 @08:02PM (#13547114)
It's called "treeshaking", and is a classic option in Lisp delivery tools.
Re:The JVM is not that heavy by pjmlp on Thursday February 02, 2017 @02:30AM (#13548801)
Or virtual image pruning in Smalltalk tools.
Re:The JVM is not that heavy by hashhar on Thursday February 02, 2017 @08:36AM (#13550368)
.NET Core does have this and allows distributing the runtime, your classes as a standalone executable. It is obviously larger than just the class files and dlls but works great for portability.
Re:The JVM is not that heavy by deepsun on Wednesday February 01, 2017 @06:28PM (#13546497)
Proguard is de-facto standard for Android Java ecosystem, what's the difference OpenJDK-Trim has? Except that Proguard determines used/unused classes automatically, by doing static code analysis, so you need to have proguard config to omit removing stuff that is used dynamically, of course.
Re:The JVM is not that heavy by gregschlom on Wednesday February 01, 2017 @07:45PM (#13547012)
We were already using Proguard for obfuscation and we tried to use it to reduce the size of the JDK as well. If my memory serves well the main problem was that the results weren't good enough results because Proguard was being conservative and keeping a lot of stuff that we know we didn't need, but static analysis indicated that it could potentially be used.

Also, Proguard's config is pretty complicated and the results are hard to understand. Our approach (openjdk-trim) is dumb simple: unpack the java runtime jars, use rsync to filter out entire directories we don't need, pack it back.

It's a simple, brute-force approach compared to Proguard's advanced static analysis, but in this case it gives better results. Maybe a good example of "worse is better".

Re:The JVM is not that heavy by ksec on Thursday February 02, 2017 @03:13AM (#13548959)
That is why i am looking forward and VERY excited for TruffleRuby, SubstrateVM, Graal, along with C Extention.

I think once there is official way to trim JDK down, making deployment super easy and fast ( Single executable File ), Java will pick up stream again.

The only problem and hesitation we have...is Oracle.

Re:The JVM is not that heavy by ominous on Thursday February 02, 2017 @03:43AM (#13549056)
Not to sound rude, but the expression you were aiming to use is maybe "pick up steam". Random example: http://idioms.thefreedictionary.com/pick+up+steam
Re:The JVM is not that heavy by grw_ on Thursday February 02, 2017 @06:06AM (#13549664)
OT, but this expression seems like a corruption of 'build up steam' and 'pick up speed'. 'Pick up steam' doesn't really make sense in original context- something a steam train would do while stationary and preparing to move :)
Re:The JVM is not that heavy by aduffy on Wednesday February 01, 2017 @09:28PM (#13547574)
-Xverbose:class JVM arg could also have sufficed in case you don't want to use strace.
Re:The JVM is not that heavy by jandrese on Wednesday February 01, 2017 @02:40PM (#13544215)
Would it not be possible to scan an app to determine exactly what libraries it could load? I think you can avoid halting problem issues by simply doing a dumb search for the library load routines in the code, at the possible (but not very likely) problem of picking up spurious data in a binary blob section.
Re:The JVM is not that heavy by necubi on Wednesday February 01, 2017 @02:45PM (#13544264)
There are tools that do this (for example, proguard). The issue is that java can dynamically load code, so static analysis isn't sufficient. Typically for areas where code size matters (like on Android), you use static analysis plus a manual list.
Re:The JVM is not that heavy by astrodust on Wednesday February 01, 2017 @05:26PM (#13546011)
It would be great to have a runtime for JRuby that's really stripped down to just the necessities. The spin-up time of the jruby environment is painful and it seems predominantly the fault of the JVM's baggage.
Re:The JVM is not that heavy by geocar on Wednesday February 01, 2017 @06:22PM (#13546466)
I don't know. 10MB still sounds really too big.

Why is it so big? What do we gain?

I don't have a system with 10MB of cache, so I imagine Java can't run faster than memory...

Re:The JVM is not that heavy by cbsmith on Wednesday February 01, 2017 @11:44PM (#13548171)
Huh? What?

Several points:

* Who said the 10MB is all used at once?

* I don't know your hardware, but there is very, very good chance you are actually quite wrong about <10MB of cache. These days, most magnetic disks have more cache than that, and if you are using SSD's, there's a boatload more cache than that in there.

* If you were referring strictly to CPU cache, then I'm even more confused, because the entire existence of that stuff is predicated on it being faster than memory, so... (and even still, if your total CPU cache isn't 10MB, it likely isn't that much smaller).

* It's not like the whole package would sit in RAM the whole time anyway. By your same assertion, I could say that one of my CPU registers is only 64-bits wide, so I imagine all programs larger than 64-bits can't run faster than L3 cache...

I'm not sure why you'd say it is too big. The article page is 1.4 MB alone... and it still needs to leverage a general purpose runtime/JIT that is orders of magnitude larger to do its single fixed purpose.

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @04:34AM (#13549274)
> Who said the 10MB is all used at once?

The parent was suggesting that this was all that was actually needed out of the 100mb or so downloadable. If you think the JVM is smaller, how small is it exactly?

> If you were referring strictly to CPU cache, then I'm even more confused, because the entire existence of that stuff is predicated on it being faster than memory, so... (and even still, if your total CPU cache isn't 10MB, it likely isn't that much smaller).

http://www.intel.co.uk/content/www/uk/en/processors/core/cor...

I don't have anything with 10MB cache.

> It's not like the whole package would sit in RAM the whole time anyway. By your same assertion, I could say that one of my CPU registers is only 64-bits wide, so I imagine all programs larger than 64-bits can't run faster than L3 cache...

If you get into L1, you get about 1000x faster.

http://tech.marksblogg.com/benchmarks.html

> I'm not sure why you'd say it is too big.

Maybe I have a different perspective? If a 600kb runtime is 1000x faster, I want to know what I get by being 10x bigger. I'm quite surprised that there are so many responders defending it given that these benchmarks were just on Hacker News a few days ago.

Re:The JVM is not that heavy by adrianN on Thursday February 02, 2017 @07:51AM (#13550146)
Unless you linearly scan the whole binary all the time, your CPU makes sure that only the stuff you're currently using is in the cache, so only the data your hot loop is touching.

You could easily see that your assumption is wrong by observing that a typical C application is not 1000 times faster than a typical Java application.

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @11:09AM (#13551539)
> Unless you linearly scan the whole binary all the time, your CPU makes sure that only the stuff you're currently using is in the cache, so only the data your hot loop is touching.

Cache fills optimize for linear scans, and have nothing to do with eviction.

> You could easily see that your assumption is wrong by observing that a typical C application is not 1000 times faster than a typical Java application.

What assumption are you talking about?

Where do you find your typical applications? Spark is supposed to be one the fastest Java-implementations of a database system, and it's 1000x slower than the fastest C-implementation database systems, but this is clearly a problem limited by memory.

What about problems that are just CPU-bound? C is at least 3x faster than Java for those[1], so just by being "a little bit faster" (if 3x is a "little" faster) then as soon as we introduce latency (like memory, or network, or disk, and so on) this problem magnifies quickly.

[1]: https://benchmarksgame.alioth.debian.org/u64q/compare.php?la...

Re:The JVM is not that heavy by cbsmith on Thursday February 02, 2017 @05:28PM (#13555473)
> Spark is supposed to be one the fastest Java-implementations of a database system, and it's 1000x slower than the fastest C-implementation database systems, but this is clearly a problem limited by memory.

Wow.. so much wrong, I'm not sure how to unpack it all.

a) Spark is Scala, not Java, though both do use the JVM, so I'll give you that.

b) Spark is not a database system, though it is a framework for manipulating data

c) Spark is generally considered to be much faster than Hadoop, and does it's job well, but I'm not sure it qualifies as the fastest anything.

d) By any reasonable interpretation, the fastest Java database system is definitely not Spark. You will find that benchmarks of Java database systems generally don't even include Spark (as an example https://github.com/lmdbjava/benchmarks/blob/master/results/2...)

e) Fast is an ambiguous term... usually you are looking at things like latency, throughput, efficiency, etc. I'm not sure which you mean here.

f) If you know anything at all about runtimes, you'd know that if you've found a Java based system that is 1000x slower than a C based system, either your benchmark is extremely specialized, broken, or you are comparing apples & oranges.

Look, Java certainly has some overhead to it, and sometimes it significantly impacts performance. Before you get too excited about attributing it to runtime size, you might want to look at the size of glibc...

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @07:56PM (#13556675)
> By any reasonable interpretation, the fastest Java database system is definitely not Spark

What database would you recommend for solving the taxi problem using the JVM?

> Spark is Scala, not Java, though both do use the JVM, so I'll give you that.

What does JVM stand for? I was under the impression that we were talking about it's size (10mb v. 100mb).

> You will find that benchmarks of Java database systems generally don't even include Spark

And? What are we talking about here?

> If you know anything at all about runtimes, you'd know that if you've found a Java based system that is 1000x slower than a C based system, either your benchmark is extremely specialized, broken, or you are comparing apples & oranges.

Why?

We're talking about business problems, not about microbenchmarks.

If this is a business problem, and I solve it in 1/1000th the time, for roughly the same cost, then what exactly is your complaint?

> Fast is an ambiguous term... usually you are looking at things like latency, throughput, efficiency, etc. I'm not sure which you mean here.

It's not ambiguous. I'm pointing to the timings for a specific, and realistic business problem.

> Look, Java certainly has some overhead to it, and sometimes it significantly impacts performance. Before you get too excited about attributing it to runtime size, you might want to look at the size of glibc...

Does Java include glibc?

What exactly is your point here?

Re:The JVM is not that heavy by cbsmith on Thursday February 02, 2017 @11:30PM (#13557694)
> What database would you recommend for solving the taxi problem using the JVM?

You have me at a disadvantage here... The only taxi problem that comes to mind is a probability problem that I'd not likely use a database for at all...

> If this is a business problem, and I solve it in 1/1000th the time, for roughly the same cost, then what exactly is your complaint?

If you came to the conclusion that your business problem runs 1000x faster because of differences in the runtime... you've made a mistake. It is far more likely your benchmark is flawed, or there are significant differences in the compared solutions beyond just the runtimes.

Seriously, I've spent a career dealing with situations exactly like that: "hey, this is 1000x slower than what we are doing before... can you fix that?". Once you are dealing with optimized runtimes, while there can be important differences between them, there just isn't that much room left for improvement.

> It's not ambiguous. I'm pointing to the timings for a specific, and realistic business problem.

The problem is perhaps not ambiguous to you, but you haven't described it in terribly specific terms. More importantly though, you haven't described what you mean by "faster"? That's the ambiguity.

> Does Java include glibc?

> What exactly is your point here?

C programs do. Lots of very efficient, high performance C programs.

Re:The JVM is not that heavy by geocar on Saturday February 04, 2017 @10:47AM (#13567325)
> The only taxi problem that comes to mind

It's the problem that I linked to previously.

http://tech.marksblogg.com/benchmarks.html

Finding good benchmarks is hard: Business problems are a good one because these are the ways experts will solve problems using these tools, and we can discuss the choice of tooling, whether this is the right way to solve the problem, and even what the best tools for this problem is -- in this case, GPU beats CPU, but what's amazing is just how close a CPU-powered solution gets by turning it into a memory-streaming problem (which the GPU needs to do anyway).

> If you came to the conclusion that your business problem runs 1000x faster because of differences in the runtime...

I haven't come to any conclusion.

There are a lot of differences between a JVM-powered business solution and a KDB-powered business solution, however one striking difference is the cache-effect.

However the question remains: What exactly do we get by having a big runtime? That we get to write loops?

Re:The JVM is not that heavy by cbsmith on Saturday February 04, 2017 @12:54PM (#13568010)
> what's amazing is just how close a CPU-powered solution gets by turning it into a memory-streaming problem (which the GPU needs to do anyway).

Yes, it turns out the algorithmic approach you use to solve the problem tends to dwarf other factors.

> There are a lot of differences between a JVM-powered business solution and a KDB-powered business solution, however one striking difference is the cache-effect.

Wait, you looked at those benchmarks and came to the conclusion that the language runtimes were the key to the differences?

> However the question remains: What exactly do we get by having a big runtime? That we get to write loops?

There is absolutely no intrinsic value in a big runtime.

Now, one can trivially make a <1KB read-eval-print runtime. So I'll answer your question with a question: why do people not use <1KB runtimes?

Re:The JVM is not that heavy by geocar on Thursday February 09, 2017 @02:35AM (#13605329)
> Wait, you looked at those benchmarks and came to the conclusion that the language runtimes were the key to the differences?

At the risk of repeating myself: I don't have any conclusions.

> There is absolutely no intrinsic value in a big runtime.

And yet there is cost. It is unclear if that cost is a factor.

> Now, one can trivially make a <1KB read-eval-print runtime. So I'll answer your question with a question: why do people not use <1KB runtimes?

Because they are not useful.

We are looking at a business problem, think about the ways people can solve that problem, and cross-comparing the tooling used by those different solutions.

Is there really nothing to be gained here?

The memory-central approach clearly wins out so heavily (and the fact we can map-reduce across cores or machines as our problem gets bigger) is a huge advantage in the KDB-powered solution. It's also the obvious implementation for a KDB-powered solution.

Is this Spark-based solution not the typical way Spark is implemented?

Could a 10mb solution do the same if it can't get into L1? Is it worth trying to figure out how to make Spark work correctly if the JVM has a size limit? Is that a size limit?

There are a lot of questions here that require more experiments to answer, but one thing stands out to me: Why bother?

If I've got a faster tool, that encourages the correct approach, why should I bother trying to figure these things out? Or put perhaps more clearly: What do I gain with that 10mb?

That CUDA solution is exciting... There is stuff to think about there.

Re:The JVM is not that heavy by cbsmith on Monday February 13, 2017 @10:09AM (#13635758)
> At the risk of repeating myself: I don't have any conclusions.

For someone who doesn't have any conclusions, you're making a lot of assertions that don't jive with reality.

> And yet there is cost. It is unclear if that cost is a factor.

It's a factor... just not the factor you think it is.

> Because they are not useful.

I think you grokked it.

> The memory-central approach clearly wins out so heavily (and the fact we can map-reduce across cores or machines as our problem gets bigger) is a huge advantage in the KDB-powered solution. It's also the obvious implementation for a KDB-powered solution.

KDB is a great tool, but you are sadly mistaken if you think the trick to its success is the runtime. That its runtime is so small is impressive, and a reflection of its craftsmanship, but it isn't why it is efficient. For most data problems, the runtime is dwarfed by the data, so the efficiency that the runtime organizes and manipulates the data dominates other factors, like the size of the runtime. This should be obvious, as this is a central purpose of a database.

> There are a lot of questions here that require more experiments to answer, but one thing stands out to me: Why bother?

Yes, you almost certainly shouldn't bother.

Spark/Hadoop/etc. are intended for massively distributed compute jobs, where the runtime overhead on an individual machine is comparatively trivial to inefficiencies you might encounter from failing to orchestrate the work efficiently. They're designed to tolerate cheap heterogenous hardware that fails regularly, so they make a lot of trade-offs that hamper getting to anything resembling peak hardware efficiency. You're talking about a runtime fitting in L1, but these are distributed systems that orchestrate work over a network... Your compute might run in L1, but the orchestration sure as heck doesn't. Consequently, they're not terribly efficient for smaller jobs. There is a tendency for people to use them for tasks that are better addressed in other ways. It is unfortunate and frustrating.

Until you are dealing with such a problem, they're actually quite inefficient for the job... but that inefficiency is not a function of JVM.

Measuring the JVM's efficiency with Spark is like measuring C++'s efficiency with Firefox.

> If I've got a faster tool, that encourages the correct approach, why should I bother trying to figure these things out? Or put perhaps more clearly: What do I gain with that 10mb?

If you read the documentation, the gains should be clear. If you are asking the question, likely the gains are irrelevant to your problem. I would, however, caution you to worry less about the runtime size and more about the runtime efficiency. The two are often at best tenuously related.

Re:The JVM is not that heavy by adrianN on Thursday February 02, 2017 @11:33AM (#13551806)
If your assumption that a 10MB JVM kills the cache were true, then the alioth benchmarks you have posted wouldn't show a speed difference of ~3. I suggest you learn a bit more about how CPUs work and what benchmarks mean before posting bold claims.
Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @07:57PM (#13556677)
Why not? Those problems fit into cache.
Re:The JVM is not that heavy by cbsmith on Monday February 06, 2017 @07:34PM (#13585821)
Because they are 333x slower than you'd expect.
Re:The JVM is not that heavy by cbsmith on Thursday February 02, 2017 @05:14PM (#13555330)
> I don't have anything with 10MB cache.

The link you provided was to three distinct models of i7 processors... all with 8MB of L3 cache. I would argue that 8MB isn't much smaller than 10MB, but I will understand if you disagree. However, even the slowest of those processors also has 1MB of L2 cache and 256KB of L1 cache, not to mention other "cache-like" memory in the form of renamed registers, completion queues, etc. At most, we're talking <800KB shy of 10MB in cache.

> If you get into L1, you get about 1000x faster.

I think you are making my point for me.

> Maybe I have a different perspective? If a 600kb runtime is 1000x faster, I want to know what I get by being 10x bigger.

You are assuming that at all times all of that 10MB must be touched by the processor at once. You can have a 10MB runtime where most of the cycles are being spent on a hotspot <4KB of data.... Having a hot spot that is orders of magnitude smaller than the full runtime is totally unsurprising. It's particularly true when your runtime has a JIT in it. With a JIT, most of the time, the bytes that are being executed aren't part of that 10MB, but rather are generated by it. Are you going to penalize your 600KB runtime for the size of the source code? ;-)

Re:The JVM is not that heavy by gregschlom on Wednesday February 01, 2017 @07:50PM (#13547051)
10MB for a platform that allows you to run code on all three major operating systems without too much trouble and in a performant way is a huge win, in my opinion. Not many alternatives come close to that.
Re:The JVM is not that heavy by pjmlp on Thursday February 02, 2017 @02:33AM (#13548811)
Actually all languages with a rich runtime and standard library that isn't just a thin POSIX layer like C or C++ (although C++ has been improving their library story).

Still your point holds.

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @04:30AM (#13549251)
Q/KDB is 600kb, also runs code on all three major operating systems (and a few minor ones). It's also about 1000x faster than Java/Spark[1].

1000x slower doesn't sound like a huge win to me; it sounds like a huge cost, so my question is what do we gain by making our programs 1000x slower?

[1]: http://tech.marksblogg.com/benchmarks.html

Re:The JVM is not that heavy by adrianN on Thursday February 02, 2017 @07:54AM (#13550163)
That benchmark is literally comparing Apples to Oranges. It's not even the same hardware.
Re:The JVM is not that heavy by taneq on Wednesday February 01, 2017 @08:56PM (#13547414)
You don't need the entire executable file in cache in order to run the program.

For comparison, a C++ wxWidgets 3.0 application isn't going to be much smaller than 10MB in release mode if you statically link it. Much as I hate to admit it, 10MB just isn't that big in an age of terabyte SSDs and systems with 32GB of RAM.

Re:The JVM is not that heavy by nv-vn on Wednesday February 01, 2017 @09:54PM (#13547691)
Not that the majority of users have that. Even eschewing those outside of wealthy countries, most users are on mobile devices (laptops, tablets, cell phones). Of those who have desktop PCs, very few have terabyte SSDs and even fewer have 32gb of RAM. For most people, RAM is probably somewhere between 4-8GB.
Re:The JVM is not that heavy by krallja on Wednesday February 01, 2017 @10:04PM (#13547739)
10MB fits comfortably within 4GB of RAM.
Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @04:37AM (#13549288)
I don't think it's healthy to think of that 4GB module as "RAM".

It's connected to your CPU by a serial communications interface so access is not uniform or timely, and if the CPU needs any of it, it stops what it's doing while it waits.

The "cache ram" (L1 and to a lesser extent L2) actually acts like the RAM that we learn about in Knuth, so that when we discuss algorithms in terms of memory/time costs, this is the number we should be thinking about. Algorithms that are performant on disk/drum are modern solutions for what you're calling "RAM".

Re:The JVM is not that heavy by hota_mazi on Wednesday February 01, 2017 @11:50PM (#13548202)
> I don't know. 10MB still sounds really too big.

Can't tell if sarcastic or... o_O.

In the unlikely case you're actually serious, you really need to rethink your perception of memory costs in 2017.

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @04:27AM (#13549233)
No, I'm really quite serious.

KDB[1] is about 1000x faster than Spark[2], and is only about 600kb (and most of that is shared library dynamic linker stuff that makes interfacing with the rest of the OS easier). A big part of why it's fast is because it's small -- once you're inside cache memory everything gets faster.

That's the real cost of memory in 2017. So what did we gain for paying it?

[1]: https://news.ycombinator.com/item?id=13481824

[2]: http://tech.marksblogg.com/billion-nyc-taxi-rides-spark-2-1-...

Re:The JVM is not that heavy by ipsi on Thursday February 02, 2017 @05:45AM (#13549576)
You're comparing completely, utterly different results here, and it's really hurting any point you're trying to make.

You're comparing KDB running on 4x Intel Xeon Phi 7210 CPUs, totaling 256 physical CPUs.

Compared to the best result for Java/Spark, which was running on 11x m3.xlarge instances on AWS. That's only 44 CPUs, plus it's running on AWS, not 100% dedicated hardware, so it's tough to tell what sort of an impact the virtualization + EBS has on performance. Plus, from the AWS page: "Each vCPU is a hyperthread of an Intel Xeon core except for T2 and m3.medium", which does not do anything good for the results.

Yes, technically, KDB was 199.80x faster (not 1000!) than Java/Spark, when it was given vastly superior, dedicated hardware without virtualization, and when tackling a problem that the hardware setup is optimized for. Note that the author calls this out by saying "This isn't dissimilar to using graphics cards" when talking about the setup he was using for the KDB benchmarks.

To get a sensible idea of the relative difference in performance, you would have to compare KDB and Java/Spark both running on the Xeon Phis, and/or running both on 11x m3.xlarge AWS instances - and even then, if Java/Spark does poorly on the Xeon Phi test, that might just mean that the Java/Spark developers haven't optimized for that particular setup.

Re:The JVM is not that heavy by srpeck on Thursday February 02, 2017 @11:27AM (#13551740)
Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @11:22AM (#13551675)
> You're comparing completely, utterly different results here, and it's really hurting any point you're trying to make.

Then argue with the point you think I could be making instead of the point that you think I'm making[1]

[1]: http://philosophy.lander.edu/oriental/charity.html

> you would have to compare KDB and Java/Spark both running on the Xeon Phis, and/or running both on 11x m3.xlarge AWS instances - and even then, if Java/Spark does poorly on the Xeon Phi test...

If Spark can solve the business problem in less real-time in another way, I think that would be worth talking about, but it's my understanding that a bunch of mid/large machines connected to shared storage is the typical Spark deployment, and the hardware costs are similar to the Phi solution.

So my larger question still stands: What is the value in this approach, if it's not faster or cheaper?

Re:The JVM is not that heavy by tobz on Thursday February 02, 2017 @12:27PM (#13552357)
If "this approach" is using Java/Spark, instead of something that is a smaller binary, then there are some easy answers to your questions:

- people don't want to write C (or K, or whatever yields a small binary)

- the cost of switching languages is not worth the speed-up

- it's already fast enough

I don't think you're wrong, overall, that, specifically, kdb can be much faster than an equivalently sized Spark cluster, but simply being faster does not invalidate other approaches, which is what you seem to be arguing for.

Re:The JVM is not that heavy by geocar on Thursday February 02, 2017 @07:59PM (#13556698)
I'm not arguing for anything: I'm asking what do we get for this cost.

It sounds like you're suggesting we get:

* Not having to write in SQL (note KDB supports SQL92)

Maybe something else? I'm not sure I understand.

Re:The JVM is not that heavy by guard-of-terra on Wednesday February 01, 2017 @07:36PM (#13546950)
Locales? Timezones? Unicode? There's a lot of stuff that is there to be used from time to time, does not mean it hits your processor cache often.

BTW, libruby-2.3 is 2,5M, just the shared object file, and it tries to use all aforementioned stuff from the underlying UNIX.

Re:The JVM is not that heavy by ausjke on Thursday February 02, 2017 @09:08AM (#13550536)
Java8 has profiles already, Java9 will be able to customize further. I'm not a Java guy, but node.js after npm-install the suddenly increased base is hundreds MB easily, I somehow now feel Java/whatever is better organized and manageable. After learning node.js for a product for a few months, I'm actually returning to PHP7, which has nearly identical OOP as Java.
Re:The JVM is not that heavy by kazinator on Thursday February 02, 2017 @06:47PM (#13556183)
Couple of megabytes without a run-time is huge. What's hiding in there; a VirtualBox image with a Linux kernel + initrd? Or maybe a high definition splash screen in PNG form?