Re:The Problems That Attract the Smartest People
by wsgeorge on Monday April 03, 2023 @05:57AM
(#35420932)
On start up it does load a fraction of the weights, so RAM usage is low, but as you run inferences on the model, it tends to eat up the required amount of RAM. This was from my initial testing.
Re:The Problems That Attract the Smartest People
by smcameron on Monday April 03, 2023 @05:58AM
(#35420942)
It mmaps the data so the page any particular bit of data is on is only loaded into memory if something on that page is accessed (at least that is my understanding).
Re:The Problems That Attract the Smartest People
by enlyth on Monday April 03, 2023 @07:25AM
(#35421689)
Looks like it will be reverted because it introduced some regressions and as far as I understand, there was no magic in the end, but there was a ton of drama around it.
Most of the drama is contained in this thread, if you like reading this sort of stuff: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
I saw people saying it only loads the weights necessary, but all the weights are necessary (unless the architecture is heavily modified). Does it stream or load only the currently necessary layers in a way?