News for hackers. Stuff that matters.
Re:Gcc compile error with 2GB of data by dexen on Thursday June 09, 2011 @06:12PM (#2638981)
tl;dr:

the error is returned by the linker rather than compiler itself. It is not a bug, just size limitation of the default memory model. Linux x86_64 provides `large model' -- as pointed out by VJo, but it is not supported by GCC before 4.6, http://stackoverflow.com/questions/6296837/gcc-compile-error...

Re:Gcc compile error with 2GB of data by joeyh on Thursday June 09, 2011 @08:14PM (#2639262)
What a great technical problem, except for this bit:

> Btw, I didn't try to hide behind "this is scientific computing -- no way to optimize". It's just that the basis for this code is something that comes out of a "black box" where I have no real access to

Black box != science.

Re:Gcc compile error with 2GB of data by esrauch on Thursday June 09, 2011 @08:56PM (#2639355)
A research assistantship I held was based on classified data; all of the published work had to be approved by the DoD and the actual data we used wasn't allowed to be published which made our results entirely unreproducable.
Re:Gcc compile error with 2GB of data by jedbrown on Friday June 10, 2011 @06:19AM (#2640331)
I'm certain the methods being used here are just wrong. It looks like it's code generated by a symbolic algebra system. Chances are that using a more scalable methodology would eliminate the code size problem. For example, symbolic manipulations often have exponential space complexity in the depth of the expression tree, but automatic differentiation can do the required operation in polynomial time and space (often linear or log-linear).

Unfortunately, "no way to optimize" is basically a statement that the questioner is not interested in an algorithmic way to make the problem go away.

Re:Gcc compile error with 2GB of data by roel_v on Friday June 10, 2011 @05:38AM (#2640264)
It seems you don't understand the context of research projects. When you're assigned to write a program that processes data x coming group another group, it's valid to say that that input data is a 'black box' to you. That still doesn't mean that the whole research is 'black box' or somehow not 'real science'.
Re:Gcc compile error with 2GB of data by eru on Thursday June 09, 2011 @08:50PM (#2639339)
I wish that was true.
Re:Gcc compile error with 2GB of data by sevenproxies on Thursday June 09, 2011 @10:02PM (#2639501)
Science is a blackbox. We make theories about the world we live in based upon the results of experiments. Richard Feynman commented method of thinking about science on one of his videos.

http://www.youtube.com/watch?v=o1dgrvlWML4

Re:Gcc compile error with 2GB of data by jerf on Thursday June 09, 2011 @10:06PM (#2639510)
Science studies a black box. The science itself shouldn't be a black box. What is the added value of a black box representation of a black box?
Re:Gcc compile error with 2GB of data by cbr on Thursday June 09, 2011 @10:52PM (#2639590)
> What is the added value of a black box representation of a black box?

The ability to make predictions.

Re:Gcc compile error with 2GB of data by jerf on Friday June 10, 2011 @12:20AM (#2639789)
How? You handed me a black box. You wrote some code, and you fed it some data, and you made some prediction, and all I really have is your word for it that your prediction worked, but I sure can't do anything with it. I don't know what it is or how it works, how to extend it, how to build on it, anything.

That's not science by any useful definition of the term.

Re:Gcc compile error with 2GB of data by shareme on Friday June 10, 2011 @03:15AM (#2640077)
wht do you call a machine so complicated that we do not know its source code?

Its called the human body and the black box is known as Medical Science..:)

Re:Gcc compile error with 2GB of data by robin_reala on Friday June 10, 2011 @05:13AM (#2640230)
We know its source code. We just don’t understand it.

http://www.ornl.gov/sci/techresources/Human_Genome/home.shtm...

Re:Gcc compile error with 2GB of data by __rkaup__ on Thursday June 09, 2011 @05:20PM (#2638838)
Is including all that data in the object code really necessary?
Re:Gcc compile error with 2GB of data by dekayed on Thursday June 09, 2011 @06:02PM (#2638953)
Generally not. My last project involved building a framework in which scientists could run calculations for certain types of risk. A prototype built by some of said scientists involved code like seen in the SO question. After looking at the types of calculations being done, we figured out that most of the equations that were used were similar and could be generalized. We also moved all the coefficients and parameters for all the equations into configuration files. I suspect that a thorough evaluation of this code would reveal something similar as in my case.
Re:Gcc compile error with 2GB of data by kelnos on Thursday June 09, 2011 @06:01PM (#2638952)
From some of the comments, it's suggested that the problem is too much code, and that the data (of which there is a lot, granted) isn't contributing to the problem. I'm not sure if that's the case though.

If the data is part of the problem, I wonder if he can write a new linker script to rearrange the sections in the file so all code is below the signed 32-bit (~2GB) boundary. Though that raises the question... will it be able to address the data? Does initialized data access use 32- or 64-bit offsets in the small/medium models?

At any rate, it seems gcc 4.6 supports the x86_64 "large model", which should solve the problem without code/data changes.

Re:Gcc compile error with 2GB of data by kodablah on Thursday June 09, 2011 @05:56PM (#2638942)
Absolutely not. This appears to be generated code from a data set to avoid what most developers would do at runtime.
Re:Gcc compile error with 2GB of data by juiceandjuice on Thursday June 09, 2011 @09:49PM (#2639475)
Those expressions look a lot like an alternating series to me. You should be able to generate an expression to produce them fairly easily.
Re:Gcc compile error with 2GB of data by bsiemon on Thursday June 09, 2011 @06:06PM (#2638962)
This sorta reminds me of the silly things people do with JavaScript.
Re:Gcc compile error with 2GB of data by nddrylliog on Thursday June 09, 2011 @07:36PM (#2639185)
Such as... an MP3 decoder? https://github.com/nddrylliog/jsmad
Re:Gcc compile error with 2GB of data by malkia on Thursday June 09, 2011 @08:18PM (#2639277)
I suggested to try and use LuaJIT.

It has very good double precision floating point, and it looks like SSE/SIMD were not used directly from the code.

LuaJIT should garbage collect unused code, if not it always regenerates it. It should be able to compile and execute tons of code as he has (and it looks like all his stuff is generated, but for C++).

Re:Gcc compile error with 2GB of data by premchai21 on Thursday June 09, 2011 @09:39PM (#2639445)
Last I checked, the LuaJIT allocator on AMD64 platforms uses only a small part of the address space for the Lua heap, partly so that more efficient type-punned representations can be used internally. I don't remember what the limit is exactly, but it's only a few GB (and beyond that the GC starts having trouble anyway). I don't know whether this applies to the machine-code JIT output, or to external cdata arrays, but it's something to watch out for here.