News for hackers. Stuff that matters.
Re:Pixie – A small, fast, native Lisp by copx on Tuesday January 17, 2017 @04:08PM (#13421206)
> And it's small. Currently the interpreter, JIT, GC, and stdlib clock in at about 10.3MB once compiled down to an executable.

Oh how the definition of "small" has changed. I actually would like to know how they managed to make something like this so big.

To compare, LuaJIT is about 400 KB, and that includes the Lua standard library, a JIT almost certainly more advanced than Pixie's current one, an incremental GC, and a C FFI.

Neither compilers (well, except C++ ones), nor stuff you usually find in standard libraries, nor a GC should require much code to implement, relatively speaking (e.g. compared to a WYSIWYG word processor). These things are usually small. The compilers for almost every language were < 1 MB in size for the longest time.

I am not saying that Pixie being 10 MB in size is a problem. We have a lot more bandwidth and disk space nowadays, 10 MB is nothing. My point is that a "JIT, GC, and stdlib" package weighing this much cannot claim to be "small" for what it does.

Re:Pixie – A small, fast, native Lisp by haberman on Tuesday January 17, 2017 @07:34PM (#13422965)
I agree with your point completely. I just want to add that throwing out raw numbers like "10.3MB" or "400 KB" is not very precise. Binaries can vary immensely based on whether they have debug info, string tables, etc. or whether these have been stripped away.

I wrote a size profiling tool that can give much more precise measurements (like size(1) on steroids, see: https://github.com/google/bloaty). Here is output for LuaJIT:

    $ bloaty src/luajit -n 5
         VM SIZE                     FILE SIZE
     --------------               --------------
      74.3%   323Ki .text           323Ki  73.8%
      12.5%  54.5Ki .eh_frame      54.5Ki  12.4%
       7.6%  33.2Ki .rodata        33.2Ki   7.6%
       2.2%  9.72Ki [Other]        12.9Ki   2.9%
       2.1%  9.03Ki .eh_frame_hdr  9.03Ki   2.1%
       1.2%  5.41Ki .dynsym        5.41Ki   1.2%
     100.0%   435Ki TOTAL           438Ki 100.0%
And for Pixie:

    $ bloaty pixie/pixie-vm -n 5
         VM SIZE               FILE SIZE
     --------------         --------------
      57.5%  4.39Mi .text    4.39Mi  44.7%
      33.7%  2.58Mi .data    2.58Mi  26.3%
       0.0%       0 .symtab  1.31Mi  13.4%
       0.0%       0 .strtab   978Ki   9.7%
       8.8%   688Ki [Other]   595Ki   5.9%
       0.0%       8 [None]        0   0.0%
     100.0%  7.64Mi TOTAL    9.82Mi 100.0%
In this case, neither binary had debug info. Pixie does appear to have a symbol table though, which LuaJIT has mostly stripped.

In general, I think "VM size" is the best general number to cite when talking about binary size, since it avoids penalizing binaries for keeping around debug info or symbol tables. Symbol tables and debug info are useful; we don't want people to feel pressured to strip them just to avoid looking bad in conversations about binary size.

Re:Pixie – A small, fast, native Lisp by sabauma on Tuesday January 17, 2017 @09:38PM (#13423542)
Its worth noting that the design of the RPython JIT will always result in a large amount of static data in the resulting binary. The RPython translator basically generates a bytecode representation of most of your interpreter and bakes that into the binary. You can probably expect at least a 2x size increase in the size of your binary. As a reference point, after stripping [Pycket](https://github.com/pycket/pycket)'s binaries are 6.1Mi without the JIT and 16Mi with the JIT.
Re:Pixie – A small, fast, native Lisp by stcredzero on Tuesday January 17, 2017 @04:31PM (#13421433)
To compare, LuaJIT is about 400 KB, and that includes the Lua standard library, a JIT almost certainly more advanced than Pixie's current one, an incremental GC, and a C FFI.

Back in the day, Smalltalk was criticized as bloated because one ended up with stripped binary+image of about 2MB. Someone around that time got SqueakVM+image down to just under 400k. There were specialized Smalltalks in company R&D that got their image down to 45k.

Re:Pixie – A small, fast, native Lisp by oska on Tuesday January 17, 2017 @06:35PM (#13422572)
The Oberon Operating System and compiler was 131 KB. [1]

[1] http://users.cms.caltech.edu/~cs140/140a/Oberon/system_faq.h...

Re:Pixie – A small, fast, native Lisp by beamatronic on Tuesday January 17, 2017 @05:02PM (#13421817)
Don't forget, even 2 MB was still a somewhat inconvenient size when floppy disks were only at 1.44 MB capacity.
Re:Pixie – A small, fast, native Lisp by stcredzero on Tuesday January 17, 2017 @06:05PM (#13422359)
Floppy disks were well on their way out, and people were still complaining about a 2 MB footprint.
Re:Pixie – A small, fast, native Lisp by timClicks on Wednesday January 18, 2017 @01:28AM (#13424504)
I thought Smalltalk's hayday was early-mid 80s. That would imply 5.25" floppies would still be in use and 3.5" floppies overtaking them circa 1988.
Re:Pixie – A small, fast, native Lisp by stcredzero on Wednesday January 18, 2017 @02:06PM (#13429285)
I thought Smalltalk's hayday was early-mid 80s.

My involvement with Smalltalk started in the 90's. Then it still had high penetration in the Fortune 500, and it was a niche language for complex financial programs. It was also used in Energy.

Re:Pixie – A small, fast, native Lisp by throwaway7645 on Tuesday January 17, 2017 @04:39PM (#13421535)
Interesting historical note!
Re:Pixie – A small, fast, native Lisp by jlarocco on Tuesday January 17, 2017 @05:11PM (#13421919)
Well, for comparison "Hello World" in Common Lisp, using SBCL on x64 Linux creates a ~70 Mb binary, but nobody would ever claim it's small :-)

Unfortunately, for an arbitrary CL program it's impossible to tell for sure how much of the CL compiler and standard library it will need at run time, so SBCL takes the easy route and just includes everything. Some of the commercial Lisp compilers are a lot smarter at stripping things out and and can create significantly smaller executables.

I usually avoid the issue altogether by not building binaries and running most things from the REPL or using a "#!/usr/bin/lisp --script" shebang line.

Re:Pixie – A small, fast, native Lisp by gkya on Tuesday January 17, 2017 @05:22PM (#13422013)
SBCL does not create binaries per se, but dumps the program image. While it practically is a binary, it's not equal to an executable that a conventional compiler/linker produces.
Re:Pixie – A small, fast, native Lisp by SolarNet on Tuesday January 17, 2017 @05:51PM (#13422247)
Except without a better way to package a binary it will be compared as one.
Re:Pixie – A small, fast, native Lisp by jlarocco on Tuesday January 17, 2017 @08:13PM (#13423171)
That distinction isn't important here, though. It is the reason the resulting file is so big, but for all practical purposes it's a binary that statically links the full CL runtime.
Re:Pixie – A small, fast, native Lisp by throwaway91111 on Tuesday January 17, 2017 @09:27PM (#13423485)
I wonder why tree shaking is so ineffective. Is that because any symbol could be accessed dynamically at runtime?
Re:Pixie – A small, fast, native Lisp by lispm on Wednesday January 18, 2017 @01:27AM (#13424502)
SBCL does treeshaking? I thought it just dumps the image.
Re:Pixie – A small, fast, native Lisp by throwaway91111 on Wednesday January 18, 2017 @11:35AM (#13427635)
Well that would explain it. Any idea why they don't tree shake?
Re:Pixie – A small, fast, native Lisp by lispm on Friday January 20, 2017 @02:39PM (#13446421)
Because no one wrote one. There were some attempts but nothing which would work now. Delivering small applications does not seem to be a focus of SBCL users.

https://www.snellman.net/blog/archive/2005-07-06.html

There is also the possibility to compress the image. See :compression option when saving an image:

http://www.sbcl.org/manual/#index-g_t_0040sbext_007bsave_002...

Commercial Lisp implementations like LispWorks and Allegro CL have a treeshaker for delivery, but that's not too surprising, since these products are for developers of applications and the users pay for such features.

Re:Pixie – A small, fast, native Lisp by jfoutz on Tuesday January 17, 2017 @10:12PM (#13423672)
eval is the root of all evil. String to symbol lets you introduce anything.
Re:Pixie – A small, fast, native Lisp by gkya on Tuesday January 17, 2017 @10:41PM (#13423780)
In lisp eval does not take a string as its input, but a form, i.e. a linked list, containing the program to execute. Maybe you confuse it with JavaScript or Python eval (or the like) where the function both reads in a string and runs the program it parsed from that string. There are diverse functions to convert a string to a symbol in different lisps.
Re:Pixie – A small, fast, native Lisp by jfoutz on Tuesday January 17, 2017 @10:55PM (#13423838)
Perhaps i'm rusty.

in this case

    (funcall #'+ 1 2 3)
or the more scheme-ish

    (apply '+ '(1 2 3))
i think the symbol plus gets evaluated to #<function+> and then is called with 1 2 3 - CL will do apply, of course, not sure about scheme and funcall. I'm totally willing to accept my mental model is wrong, i don't have an interpreter handy.

So if that's true, you run into this problem:

    (funcall (string->symbol "+") 1 2 3)
    (apply (string->symbol "+") '(1 2 3))
maybe string->symbol does not introduce in to the function namespace, but i bet there's a way or a flag to do it.

so anyway, at the end, you run into this problem:

    (funcall (string->symbol "any-random-function") 1 2 3)
    (apply (string->symbol "any-random-function") '(1 2 3))
this makes tree shaking hard.

you could, of course, look for the handful of ways to bind symbols to functions, maybe do constant folding and include a minimal set if the compiler can prove what's actually needed.

but in general i think tree shaking is hard when you can load things like that dynamically.

Re:Pixie – A small, fast, native Lisp by lispm on Wednesday January 18, 2017 @03:02AM (#13424852)
(funcall #'+ ...) does not do a symbol lookup and does not need to retrieve the function from a symbol in Common Lisp. (funcall '+ ...) would retrieve the function from the symbol.

Treeshakers are generally used for application delivery in Lisp. When you do that, then one usually limits the amount of use of runtime dynamics. Typically you can tell the treeshaker what to remove or what to keep: the compiler, the symbol table, debugging info, etc etc.

Allegro CL and LispWorks have extensive facilities for this.

http://franz.com/support/documentation/10.0/doc/delivery.htm

http://www.lispworks.com/documentation/lw70/DV/html/delivery...

Re:Pixie – A small, fast, native Lisp by kazinator on Wednesday January 18, 2017 @05:28AM (#13425353)
More briefly, in a half decent implementation, the + symbol is involved in the processing of (funcall #'+ ...) in the same ways and at the same times as it is involved in the plain call (+ ...).
Re:Pixie – A small, fast, native Lisp by lispm on Wednesday January 18, 2017 @01:29AM (#13424511)
Some CL compilers even don't save an image. An extreme example is mocl. mocl applications for a 64bit iPad start at 1 MB size.

Other examples of CL compilers not saving an image are ECL and WCL.

The ECL manual had the terminology a bit wrong, but it actually does not dump an image.

https://common-lisp.net/project/ecl/static/manual/ch34.html#...

WCL:

https://github.com/wadehennessey/wcl

> For example, the executable for a Lisp version of the canonical ``Hello World!'' program requires only 20k bytes on 32 bit x86 Linux.

Re:Pixie – A small, fast, native Lisp by ww520 on Tuesday January 17, 2017 @10:08PM (#13423657)
10.3MB definitely is not small. I was expecting couple hundreds of K, or 1M.
Re:Pixie – A small, fast, native Lisp by nine_k on Tuesday January 17, 2017 @04:18PM (#13421302)
I wonder if it's a stripped binary.
Re:Pixie – A small, fast, native Lisp by vram22 on Tuesday January 17, 2017 @05:24PM (#13422030)
Yes, at least on Unixes, stripping could reduce the sizes of binaries by quite a bit.