Re:Tiny Language Models Come of Age
by whimsicalism on Sunday October 08, 2023 @04:49PM
(#37814275)
Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude.
Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.
Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.