News for hackers. Stuff that matters.
Re:Use OpenGL to get more RAM by tluyben2 on Sunday April 24, 2016 @06:37AM (#11559219)
It used to be standard to use your VRAM as RAM where possible. If you were not coding a game but something like a business tool, you would just stuff parts of the application in VRAM so you could swap it out from there instead of from HD or, before that, floppy. This was without OS support mostly though. For work I haven't needed this technique since the end of 90s I think.
Re:Use OpenGL to get more RAM by astrodust on Sunday April 24, 2016 @06:43AM (#11559230)
"Used to be standard" is highly debatable. The only people I ever heard of doing that were the crazy sort, enthusiasts pushing the limits of their rig, not average people.
Re:Use OpenGL to get more RAM by tluyben2 on Sunday April 24, 2016 @07:22AM (#11559312)
That depends how far back you go. You are talking PC probably; most 'desktop' computers before that had so little memory this was common place and even mostly impossible to do 'useful' business applications without it.
Re:Use OpenGL to get more RAM by astrodust on Sunday April 24, 2016 @05:51PM (#11561523)
Citation needed because honestly, I've never heard of this and I think you're confusing it with something else. I've been using computers since the introduction of the original IBM PC and have had to spend a lot of time dealing with memory issues over the years.

"Extended memory" drivers were a common thing back then but that was to circumvent the 640KB/1MB limit of DOS and they had nothing to do with video card memory. In that era few people even knew how much memory was on their video card because it was rarely a concern, most of the time it was just barely enough to hold a single frame buffer, typical of most VGA and "Super-VGA" cards.

Extended memory is that which is beyond the boundaries of the original IBM PC design: https://en.wikipedia.org/wiki/Extended_memory

It's not video card memory in any way. If I recall correctly the video card memory was often mapped into the 640KB-1024KB memory zone, but using that for system memory was futile since you'd be unable to see anything on screen.

Re:Use OpenGL to get more RAM by tluyben2 on Monday April 25, 2016 @02:55AM (#11562828)
I mentioned before PC in my comment to your comment right? You are still going on about PCs; I was not referring to PCs but rather MSX, CPC, Amiga etc. They had distinct RAM & VRAM (in the case of the MSX the cool machines had 128kb-256kb RAM & 128kb VRAM and you could not map one to the other => it was really different memory) and when you used text input screens you were wasting 112kb of memory if you didn't do this. The memory was slower to access from the CPU but it was much much faster than disks.

Not confused, just not talking about PCs in general (although I have heard cases of this happening there but that might be just indeed demo groups); I still do this on those old computers today for hobby purposes (the last work I did was somewhere end 90s on a legacy system which counted on video memory and which had to be ported so I had to remove & test that 'feature') ; there just isn't another way to get memory if you don't want to cheat (and solder in 1024kb).

Edit: I don't know if it was used in practice because there were so many different VGA cards, but [1] shows that it was common to have 256kb on a card which, depending on the era, was significant. And like said above, for business applications in text based environments (screen 0) you can use a large portion of video memory for storage while you still see something. But I can see the issue there; with other computers we didn't have that issue as you could trust a certain amount to be there for sure.

[1] http://www.osdever.net/FreeVGA/vga/vgamem.htm#detect

Re:Use OpenGL to get more RAM by Kristine1975 on Sunday April 24, 2016 @10:56AM (#11559863)
Related: https://news.ycombinator.com/item?id=10874127

VRAM is also where many "online" development suites—the BASICs and Pascals of the time—expected you to write exception traces. Rather than trying to "break into" a debugger (i.e. cram a debugger into the same address space as your software), you'd simply have your trap-handler persist the stack-trace to VRAM, switch your tape drive over to the devtools disk, and reboot. The "monitor" would load, notice that there's a stack-trace in VRAM, parse it, read the pages it mentions from your program (now on the slave tape) and display them.

Re:Use OpenGL to get more RAM by ordinary on Sunday April 24, 2016 @09:31AM (#11559620)
Hogging tabs in long-running Firefox instances has taught me that Windows (even 64 bit versions) still cannot address more than 3 GB of memory to 32 bit applications. It used to be that was because 1 GB was reserved for the GPU, among other things. I suspect (but do not know for sure) that this is still the case.
Re:Use OpenGL to get more RAM by thristian on Sunday April 24, 2016 @10:22AM (#11559763)
Originally, the top 2GB of the 32-bit address-space was reserved for the kernel, and the bottom 2GB available for each running program. This worked fairly well until programs started legitimately using 2GB of RAM, so Microsoft came up with a new scheme where only the top 1GB of address-space was reserved for the kernel, and the bottom 3GB was available for running programs.

Unfortunately, they couldn't turn it on universally; there were a lot of programs that "knew" that malloc() would never return a pointer with the top bit set, so they used it as a flag for their own purposes. Such programs would break horribly in 3GB mode when pointers could have the top bit set. Also, restricting kernel-space meant the kernel had less room to do stuff; if you were running an app that was heavy on kernel-resources but light on RAM usage, 3GB mode would be a downgrade[1].

So applications have to opt-in to 3GB mode by setting a special flag in the executable, and 32-bit versions of Windows had to be put into 3GB mode at startup to support it at all.

Of course, the real solution to "3GB mode at startup" is just to run a 64-bit OS, and the real solution to "2GB is not enough address-space" is to run 64-bit applications.

[1]: https://blogs.technet.microsoft.com/askperf/2007/03/23/memor...

Re:Use OpenGL to get more RAM by ryandrake on Sunday April 24, 2016 @11:15AM (#11559916)
> Unfortunately, they couldn't turn it on universally; there were a lot of programs that "knew" that malloc() would never return a pointer with the top bit set, so they used it as a flag for their own purposes. Such programs would break horribly in 3GB mode when pointers could have the top bit set.

Fascinating story. Never knew this. This dedication to backwards-compatibility is something Microsoft was known for. Reminds me of how they took great care when writing Windows to ensure perfect compatibility with old, horrible DOS programs, going so far as re-creating DOS bugs that these broken programs relied on. I suppose they were just accepting reality: if a user upgraded to Windows and some third party program broke, the user would blame Microsoft, not the incompetent application developer.

Re:Use OpenGL to get more RAM by adrusi on Sunday April 24, 2016 @11:56AM (#11560073)
Both Windows and Linux are distinguished for their pathological conviction to never break userspace. Linux has its warts because of this, but since it's 10 years younger than DOS and since, as a Unix kernel, it was always very clear what the kernel should and should not do, it doesn't have quite the trail of hacks seen in Windows.

Windows has broken legacy programs though. In Windows 7 a lot of old programs stopped working in the consumer editions of the OS. I don't know if they've continued making their "professional" editions that provide comparability. I suppose that's a reasonable strategy to encourage developers to update their software without killing user productivity, which could workout for the best in the long run.

Re:Use OpenGL to get more RAM by digi_owl on Sunday April 24, 2016 @08:55PM (#11562018)
Linux the kernel yes. but Linux user space is notoriously against dealing with backwards compatibility.

As for Windows, best i can tell the problem was Win16 on 64-bit Windows. My father ran into that regarding some old solitaire programs he had been keeping around.

Re:Use OpenGL to get more RAM by cyphar on Monday April 25, 2016 @11:33PM (#11569255)
Nit: There's no such thing as the "Linux userspace". The GNU/Linux userspace is a collection of many different pieces of software (including the GNU tooling and all of the FreeDesktop stuff).
Re:Use OpenGL to get more RAM by digi_owl on Tuesday April 26, 2016 @03:24PM (#11574780)
Yes, but outside of the kernel, and maybe the GNU tools, there seems to be very little willingness to entertain the need for backwards compatibility.
Re:Use OpenGL to get more RAM by cyphar on Tuesday April 26, 2016 @07:22PM (#11576576)
My issue was with the term "Linux userspace". I agree with the sentiment that many components of a modern GNU/Linux system don't seem to like the idea of backwards compatibility.
Re:Use OpenGL to get more RAM by armitron on Sunday April 24, 2016 @11:45AM (#11560026)
Or you could have separate virtual address spaces for kernel / userland.
Re:Use OpenGL to get more RAM by Dylan16807 on Sunday April 24, 2016 @12:16PM (#11560175)
You could, but:

"Just for comparison, Linux defaults to 3 GB for user mode and 1 GB for kernel mode. There are experimental patches to implement a separate virtual address space for the kernel, thus granting 4 GB for user mode AND 4 GB for kernel mode. Unfortunately, there is a 10-20% performance hit because every system call requires an address space switch (and TLB cache flush)."

Take it with a grain of salt, that quote is a decade old.

Re:Use OpenGL to get more RAM by lokedhs on Sunday April 24, 2016 @09:17PM (#11562090)
I wonder if this is something that is limited to a 32-bit kernel. Solaris gives each process the full 4 GB in userspace, and I think that a 10-20% performance degradation compared to Linux would be noticeable.
Re:Use OpenGL to get more RAM by userbinator on Sunday April 24, 2016 @11:17PM (#11562382)
It is slower. Here's an old benchmark but presumably Linux and Solaris had the same shared vs. separate address space difference back then, and the results show it:

https://pdfs.semanticscholar.org/4489/18f01d17628cb94ac69f27...

They don't call it "Slowlaris" for nothing...

Re:Use OpenGL to get more RAM by the8472 on Sunday April 24, 2016 @12:57PM (#11560367)
32bit executables can use 4GB address space on windows if they have the LARGEADDRESSAWARE flag set, which has been the case for firefox.exe since 2010[1].

It may appear that they use less than 4GB if you look at the working set or private bytes figures instead of virtual size because they reserve more address space than they allocate. Or it might crash before reaching that limit due to large allocations failing.

But 64bit builds have been available for years on their ftp server, they just weren't supported. But since last december they are[2], so there really shouldn't be any need to run 32bit builds on a 64bit system.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=556382#c34 [2] https://blog.mozilla.org/futurereleases/2015/12/15/firefox-6...

Re:Use OpenGL to get more RAM by acqq on Sunday April 24, 2016 @03:16PM (#11560929)
> there really shouldn't be any need to run 32bit builds on a 64bit system

Maybe the 64-bit version doesn't support the Adobe Flash plugin?

The another problem with 64-bit versions is that all the pointers are twice as big. So if you have only 8 GB of physical ram, you won't necessarily be able to handle much more than with a 32-bit application. But with 16 GB or more the 64-bit version must be better.

Re:Use OpenGL to get more RAM by the8472 on Sunday April 24, 2016 @04:30PM (#11561235)
> Maybe the 64-bit version doesn't support the Adobe Flash plugin?

64bit flash works for me.

> The another problem with 64-bit versions is that all the pointers are twice as big.

You're assuming pointers dominate the footprint. if you're seeing crashes due to address space exhaustion that's more likely due to large strings/blobs of data and not a huge amount of pointers.

Or to put it differently, do you really think OOM crashes are preferable to some background applications or memory-mapped files being paged out?

Re:Use OpenGL to get more RAM by acqq on Sunday April 24, 2016 @06:17PM (#11561617)
> You're assuming pointers dominate the footprint.

No, I just wrote:

>> if you have only 8 GB of physical ram, you won't necessarily be able to handle much more than with a 32-bit application

That means, I absolutely know applications that won't be better on a 8 GB RAM computer as 64-bit applications, and they were written using standard C++ libraries. Typically, such applications weren't optimized for cases when they use a lot of memory. If somebody competent spends enough time and energy, even such application can be made to use memory better but it's not something that "just happens."

And sometimes 32-bits application is a good trade-off. See for example

https://blogs.msdn.microsoft.com/ricom/2009/06/10/visual-stu...

"A 64 bit address space for the process isn’t going to help you with page faults except in maybe indirect ways, and it will definitely hurt you in direct ways because your data is bigger."

What I'm trying to suggest is that the topic is more nuanced as "64 doublegood 32."

Re:Use OpenGL to get more RAM by brianwawok on Sunday April 24, 2016 @05:14PM (#11561403)
I don't think all pointers are twice as big. How does Java do compressed OOPS?
Re:Use OpenGL to get more RAM by alblue on Monday April 25, 2016 @05:56AM (#11563312)
Java has a couple of different options for dealing with objects but is able to manage the full set of pointers and know that (for example) code doesn't try to access memory+1 for looping etc.

Under 4G worth of size pointers are simply zero extended, so that the higher order bits are just zero and the lower order bits stores in 32 bits.

Above 4G but below 32G it switches to "compressed oops" mode. This right-shifts the address by 3 bits and ensures/mandates that all objects are aligned to 8 bits (2^3). When converting back again to a memory address it multiplies by 8. This effectively multiplies the top address space by a factor of 8 and so you can store an object in the same 32 bits but can be wider.

It's possible to shift the alignment (-XX:ObjectAlignmentInBytes=16) which will then right shift the address by 4 instead giving you 64Gb memory with 32 bit addressing.

Although it's possible to go higher the advantages tend to scale out; effectively you end up padding the object space more and more so simple objects take up larger and larger slots.

The bottom line is: by controlling where pointers are used, Java can use a different addressing scheme by unpacking and repacking object pointers on the fly. And for some operations (equality testing) it doesn't even need to unpack the pointers in the first place.

Re:Use OpenGL to get more RAM by ordinary on Monday April 25, 2016 @02:34AM (#11562778)
I knew Mozilla was working on it, and at some point abandoned the process, but I wasn't aware they had picked it up again. Thanks for the pointer!
Re:Use OpenGL to get more RAM by gruez on Sunday April 24, 2016 @10:27AM (#11559783)
Its reserved for the kernel, not the GPU specifically. Normally the top half of the 32 bit address space was reserved for kernel use, and the bottom half for application use, but there is a flag in the PE header that tells windows to only reserve 1 GB for the kernel (and 3 GB for the application)
Re:Use OpenGL to get more RAM by Dylan16807 on Sunday April 24, 2016 @12:11PM (#11560151)
You can adjust the boot options on 32 bit windows to reserve 1GB for the kernel, or any number between 1GB and 2GB. It's irrelevant for 64 bit windows.

The flag in the PE header says that the application can handle addresses over 0x80000000. It don't change reservations.

Re:Use OpenGL to get more RAM by Dylan16807 on Sunday April 24, 2016 @12:08PM (#11560129)
On 64 bit windows, they get 2GB by default but can opt in to 4GB. Nothing is capped at 3GB.