News for hackers. Stuff that matters.
Re:Brian Kernighan adds Unicode support to Awk by ducktective on Saturday August 20, 2022 @02:50PM (#32534296)
I became aware of this while watching Professor Brailsford's interview with him (Computerphile channel): https://www.youtube.com/watch?v=GNyQxXw_oMQ

(Around 7-8 minute mark)

Update:

At 24-25 minute mark, he talks about the technologies he is inquiring to write his new book with (he mentions troff and groff).

He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!"

So there we go...We had the opportunity to read the mind of the developer of awk and unix and co-author of the literal "C Programming Language", confronting with the absolute state of the tooling of the modern world.

Re:Brian Kernighan adds Unicode support to Awk by maxnoe on Saturday August 20, 2022 @05:39PM (#32535361)
The Problem is that TeXLive still defaults to doing a full install.

A full install means installing ~4000 packages, including their source files (tens of thousands of tex files) and built documentation (thousands of PDF files) and hundreds of free fonts (otfs, ttfs, texs own format).

This is huge (>7GB, not just the 5 GB claimed here).

However, you don't need 99 % of this for any given document.

Not installing the source files and documentation PDFs will alone reduce the size by roughly half.

Only installing the packages you really need from a minimal installation gives you a few hundred megabytes at most for even complex documents.

It's a bit annoying to get the list of packages needed though, since there is not really any working dependency management.

I wrote a python wrapper around the tex live installer [1] to make this easy for CI jobs, see e.g. [2].

On a side note: I'd recommend luatex over xetex.

- [1] https://github.com/maxnoe/texlive-batch-installation/

- [2] https://github.com/pep-dortmund/toolbox-workshop/blob/8b00f0...

Re:Brian Kernighan adds Unicode support to Awk by dhosek on Saturday August 20, 2022 @09:51PM (#32536705)
I did some digging about a year ago and the biggest uses of space are the docs and, surprisingly, fonts: https://www.reddit.com/r/LaTeX/comments/oqmb3w/installed_mac...

It’s mostly a death by a thousand cuts thing.

MikTeX, which is available for Mac and Linux as well as Windows will do on-demand downloading of packages and is probably my recommendation for a new installation.

A lot of the drawbacks to TeX stem from to trying to meet the combined requirements of computing platforms of the 1970s–1990s. All input files effectively live in a single namespace so you cannot have two input files with the same name (the same applies to coding where you have a single namespace for all your commands as well).

Re:Brian Kernighan adds Unicode support to Awk by ddmitriev on Sunday August 21, 2022 @01:04AM (#32537550)
Eons (>15 years) ago, I sold software that relied on having a working TeX installation, and to make it easier on my users, my installer provided a very very trimmed down version of MiKTeX as an optional download. The beautiful thing about MiKTeX is that its autoinstall feature meant that one could delete 90% of the packages, and the worst that would happen is that referencing something I deemed unessential would prompt a just-in-time download.

I recall I managed to cut MiKTeX down to about 12MB. Never heard a single complaint about it.

Re:Brian Kernighan adds Unicode support to Awk by m_mueller on Sunday August 21, 2022 @01:19AM (#32537611)
Why hasn’t something like this become the standard way to use LaTeX? Boggles my mind.
Re:Brian Kernighan adds Unicode support to Awk by bzxcvbn on Sunday August 21, 2022 @02:28AM (#32537877)
Latex is stuck in the past. Academic publishers even more so. It's 2022, and I can't use slightly non-standard letters (e.g. non-Latin characters) in my papers without jumping through hoops. It's anyone's guess as to which ones will work and which won't. The language was designed in the 80's and it shows, many commonly used packages are severely outdated and nobody cares.
Re:Brian Kernighan adds Unicode support to Awk by m_mueller on Monday August 22, 2022 @11:02AM (#32552443)
I guess noone can receive fake brownie points (paper impact factor) by fixing that stuff.
Re:Brian Kernighan adds Unicode support to Awk by jcelerier on Saturday August 20, 2022 @05:54PM (#32535471)
On archlinux there's the texlive-core package which does not ship the PDF docs (most of the size). It should install 500mb (most of which are fonts..) and already provide enough to build normal documents, including lualatex for unicode support
Re:Brian Kernighan adds Unicode support to Awk by phire on Saturday August 20, 2022 @08:20PM (#32536296)
> It's a bit annoying to get the list of packages needed though, since there is not really any working dependency management.

Sounds like someone should just create a wrapper that catches file opens of uninstalled dependencies and automatically downloads them on the fly.

Re:Brian Kernighan adds Unicode support to Awk by sanxiyn on Saturday August 20, 2022 @08:32PM (#32536341)
This has been done and yes people should switch to this. "Tectonic automatically downloads support files so you don't have to install a full LaTeX system in order to start using it. If you start using a new LaTeX package, Tectonic just pulls down the files it needs and continues processing."

https://tectonic-typesetting.github.io/

Re:Brian Kernighan adds Unicode support to Awk by bradrn on Saturday August 20, 2022 @08:49PM (#32536428)
MikTeX also does this: https://docs.miktex.org/manual/autoinstall.html. As far as I’m aware, TeX Live is unique in forcing the user to install all packages at once.

[EDIT: And in fact, if a cousin comment is to be believed, TeX Live also allows this kind of installation! https://news.ycombinator.com/item?id=32535521]

Re:Brian Kernighan adds Unicode support to Awk by svat on Saturday August 20, 2022 @11:51PM (#32537261)
TeX Live allows a minimal installation, but what MikTeX has that TeX Live doesn't (AFAIK) is the on-demand automatic fetching and installing of packages. For example, if you `\usepackage{foo}` then with a minimal TeX Live you have to install it manually (with `tlmgr install foopkg`), while MiKTeX will either install it automatically or pop up a prompt asking you whether you want to install foopkg.
Re:Brian Kernighan adds Unicode support to Awk by woile on Sunday August 21, 2022 @07:50AM (#32539258)
I can't believe I read this comment literally 10 minutes after installing endless GBs of xetex haha. Thanks for the info!
Re:Brian Kernighan adds Unicode support to Awk by JadeNB on Saturday August 20, 2022 @05:54PM (#32535480)
TeXLive also comes with installation schemes that will give you (if I remember the names correctly) bare, medium, and full installations, if you prefer not to pick packages yourself. Alternately, although I don't use it myself, I'm sure you could use MikTeX, which is much better about on-demand package installation. (Or even Overleaf, if you don't want to put anything on your local device!)
Re:Brian Kernighan adds Unicode support to Awk by jhoechtl on Sunday August 21, 2022 @05:04AM (#32538499)
Development and re-juvenile of groff is recently taking a good path. While groff is still single-pass and doesn't do paragraph-at-once type setting, it is a much more pleasant experience than maybe like five years ago.
Re:Brian Kernighan adds Unicode support to Awk by samatman on Saturday August 20, 2022 @04:35PM (#32534977)
> the absolute state of the tooling of the modern world

Hah, TeX Live is... not that.

It's been enormous since I installed it off a CD in the 90s. The idea, and it works, is that you can just compile anyone's stuff out of the TeX ecosystem.

There is just... a lot... in it. You don't need a package manager if you install the whole universe locally. Like I said: not what I would call a modern approach to tooling.

On the other hand, I have latex files from the mid-Noughties and, I don't even need to check: they'll compile if I want them to.

But yeah, if you want just a little piece of TeX here and there, you're off the beaten track. That's not how TUG rolls.

Re:Brian Kernighan adds Unicode support to Awk by rob74 on Sunday August 21, 2022 @01:01AM (#32537535)
Old approach: "we don't have a lot of disk space, we have to be careful what we install".

More modern approach: "hey, we have a lot of disk space, we can install everything locally".

Even more modern approach: "hey, we have fast Internet access, we only have to install stuff when we need it".

Re:Brian Kernighan adds Unicode support to Awk by mistrial9 on Sunday August 21, 2022 @01:55PM (#32542650)
all three of those ways are important today; modern is overrated
Re:Brian Kernighan adds Unicode support to Awk by rdlw on Saturday August 20, 2022 @06:00PM (#32535521)
TeX Live can also be configured to install the bare minimum TeX ecosystem (or just TeX+LaTeX), which only takes a few minutes to download and install but results in hunting down dependencies and manually installing them whenever you want to use a new package.

It also seems quite slow to update, and a recent (?) name change of `tools' to `latex-tools' seems to have broken multicol, which drove me to MikTeX. Internet connection required, but far less headache.

Re:Brian Kernighan adds Unicode support to Awk by mlyle on Saturday August 20, 2022 @03:27PM (#32534559)
> He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!"

Man, I think once you're Kernighan there should be like a 1gigabit/sec symmetric circuit wherever you go just in case you use it to do something else useful.

Re:Brian Kernighan adds Unicode support to Awk by asddubs on Sunday August 21, 2022 @07:17PM (#32545460)
on the other hand, I've done lots of excellent coding in places with awful internet. fast enough to be able to look things up when you really have to, but too annoying to meaningfully distract yourself
Re:Brian Kernighan adds Unicode support to Awk by JdeBP on Sunday August 21, 2022 @04:36AM (#32538371)
... or a fast car with a load of magnetic tapes (or memory cards, per M. Munroe) in the boot fuelled up and ready to roll. (-:
Re:Brian Kernighan adds Unicode support to Awk by sargstuff on Sunday August 21, 2022 @10:02PM (#32546510)
Ah, thought goto usage was considered harmful.
Re:Brian Kernighan adds Unicode support to Awk by rustqt6 on Saturday August 20, 2022 @06:26PM (#32535693)
You put it jokingly? But distinguished people should definitely get some state managed perks like politicians do.
Re:Brian Kernighan adds Unicode support to Awk by kergonath on Sunday August 21, 2022 @12:18AM (#32537378)
No they should not. They need to live in the same world we do, using the same tools we have. The example being politicians should make one think: do we really need more of this greasy pole and lust for power and freebies in computing? The whole elite mindset is corrosive.

Besides, where would you put the bar for “distinguished”? Who would take the decision?

Re:Brian Kernighan adds Unicode support to Awk by Aeolos on Sunday August 21, 2022 @02:59AM (#32537990)
Other “distinguished” people, obviously.

Which should make the problems of any such system obvious: it inevitably becomes yet another “in club” for people with connections and power.

Re:Brian Kernighan adds Unicode support to Awk by mlyle on Sunday August 21, 2022 @11:28AM (#32541114)
> No they should not. They need to live in the same world we do, using the same tools we have.

Grand prizes with honorariums and perks of various kinds are a longstanding tradition with value.

I was mostly kidding, but: Kernighan totally can afford a big internet connection wherever he is. But he chooses not to buy it. The social benefit from him having one likely outweighs the personal benefit from him buying it.

Re:Brian Kernighan adds Unicode support to Awk by z3t4 on Sunday August 21, 2022 @03:13PM (#32543418)
You should start a fiber/ISP company
Re:Brian Kernighan adds Unicode support to Awk by mlyle on Sunday August 21, 2022 @04:52PM (#32544293)
It would be more of a grand prize with an internet access component. :P

Of course, if I still operated internet access and Kernighan was in my area, I'd definitely comp him a circuit. :P

Re:Brian Kernighan adds Unicode support to Awk by sargstuff on Sunday August 21, 2022 @10:05PM (#32546530)
? Atomatons ?
Re:Brian Kernighan adds Unicode support to Awk by scoot on Saturday August 20, 2022 @04:02PM (#32534780)
THis isn't about bandwidth, it's about the size of modern binaries.
Re:Brian Kernighan adds Unicode support to Awk by mananaysiempre on Saturday August 20, 2022 @04:14PM (#32534854)
TeX distributions are enormous, but the binaries themselves are actually not big—the overhead imposed by Knuth’s obnoxious license, while nonzero (you can’t modify the original Pascal-in-WEB source, only patch it, so a manual source port to a different language is painful enough that nobody tried, we’re all using an automatic Pascal-to-C translation with lipstick on it), is not huge, and aside from a smattering of utilities that’s it for the binary part.

It’s just that the distros also include oodles of (plain text!) macro packages for everything under the sun. There are some legitimately large things such as fonts, but generally speaking a full TeXLive or MiKTeX distribution is bloat by ten thousand 100-kilobyte files, like a Python distribution with the whole of PyPI included.

If you know what you want, you can probably fit a comprehensive LaTeX workbench in under 50M, but it takes an inordinate amount of time.

Re:Brian Kernighan adds Unicode support to Awk by svat on Saturday August 20, 2022 @07:24PM (#32536033)
I agree with your main point about the binaries being small and a reasonable LaTeX distribution fitting in under 50 MiB (I think starting with a minimal MiKTeX and letting it install what you need a few times will leave you there as well), but regarding

> a manual source port to a different language is painful enough that nobody tried

there are in fact quite a few, many of which are complete: https://tex.stackexchange.com/questions/507846/are-there-any...

Re:Brian Kernighan adds Unicode support to Awk by inglor_cz on Saturday August 20, 2022 @04:10PM (#32534829)
Pardon me, but why have modern binaries grown so big? When I wrote my thesis in TeX, the entire installation would fit in some 30 megabytes or so. It was actually right in the uncomfortable middle. Far too big to carry around on a set of diskettes, but a CD would be waste of space.
Re:Brian Kernighan adds Unicode support to Awk by mlyle on Saturday August 20, 2022 @04:24PM (#32534912)
You can fit a decent TeX distribution in <100MB.

But if you want to have every macro package that everyone everywhere likes, you're going to use some space.

Re:Brian Kernighan adds Unicode support to Awk by Turing_Machine on Saturday August 20, 2022 @04:33PM (#32534968)
I've often wondered why there isn't a dependency system for TeX that lets you get only the packages you need... feed it a document and tell it to automatically download and install any missing packages.

There may be some technical reason why this isn't practical. Anyone know, offhand?

Re:Brian Kernighan adds Unicode support to Awk by kccqzy on Saturday August 20, 2022 @04:43PM (#32535019)
Totally practical and supported. See MikTeX: https://miktex.org/kb/just-enough-tex

But the reason I don't use it is because I don't always have Internet when I need to write my document. I'd rather download all the packages and their documentation beforehand.

Re:Brian Kernighan adds Unicode support to Awk by bombcar on Saturday August 20, 2022 @08:24PM (#32536310)
When stuck on a plane without wifi the only thing I could find on my computer to read were the Tex manuals. Good stuff.
Re:Brian Kernighan adds Unicode support to Awk by rdlw on Saturday August 20, 2022 @05:53PM (#32535465)
MikTeX also supports downloading individual packages, though I wish there was an option to download, say, the most commonly used 1GB or something.
Re:Brian Kernighan adds Unicode support to Awk by mickmcq on Saturday August 20, 2022 @06:18PM (#32535647)
There is a TeX distribution that does what you say, TinyTeX. For some reason it is obscure and only really used by R and R Studio users. That may be because it is used to render R Markdown documents to pdf.
Re:Brian Kernighan adds Unicode support to Awk by lr1970 on Saturday August 20, 2022 @07:10PM (#32535962)
> He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!"

He can try "Tectonic" [0] - a modern XeTeX based TeX/LaTeX distribution that installs a minimum system and then downloads and installs dependencies on-demand. Tectonic is written in C and Rust [1].

[0] https://tectonic-typesetting.github.io/en-US/

[1] https://github.com/tectonic-typesetting/tectonic

Re:Brian Kernighan adds Unicode support to Awk by xvilka on Saturday August 20, 2022 @09:42PM (#32536668)
It's sad that Tectonic conversion to Rust[1] was never finished. For now it's just a wrapper around C and C++ code. By far, it was the most promising thing in this distribution.

[1] https://github.com/tectonic-typesetting/tectonic/issues/459

Re:Brian Kernighan adds Unicode support to Awk by jfk13 on Saturday August 20, 2022 @03:32PM (#32534590)
Sounds like he was looking at downloading a complete TeX Live distribution; XeTeX itself isn't anything like that size (by a couple orders of magnitude, at least).
Re:Brian Kernighan adds Unicode support to Awk by zimpenfish on Saturday August 20, 2022 @05:24PM (#32535247)
MacTeX is 4.7GB which matches the 5GB he's talking about and...

"MacTeX installs TeX Live, which contains TeX, LaTeX, AMS-TeX, and virtually every TeX-related style file and font. [...] MacTeX also installs the GUI programs TeXShop, LaTeXiT, TeX Live Utility, and BibDesk. MacTeX installs Ghostscript, an open source version of Postscript."

Which is, as you say, considerably more than just "XeTeX".

(Also those are universal binaries containing both Intel and ARM versions which probably adds some heft.)

Re:Brian Kernighan adds Unicode support to Awk by JadeNB on Saturday August 20, 2022 @05:55PM (#32535485)
> (Also those are universal binaries containing both Intel and ARM versions which probably adds some heft.)

Heh, I remember when "universal binaries" meant "PowerPC and Intel". Different universes ….

Re:Brian Kernighan adds Unicode support to Awk by jhbadger on Saturday August 20, 2022 @06:12PM (#32535609)
In the early/mid 1990s there were even "fat binaries" that had 68000 (the original Mac platform) and PowerPC binaries back when PowerPC was the new thing.
Re:Brian Kernighan adds Unicode support to Awk by ducktective on Saturday August 20, 2022 @03:35PM (#32534608)
I think distros package it like TeX-full, TeX-minimal etc...The one having documentation files is a couple of GiB on Ubuntu...

I wonder what distro or editor he is using...

Re:Brian Kernighan adds Unicode support to Awk by naves on Saturday August 20, 2022 @03:54PM (#32534733)
Two years ago, he used macOS on a 13" MacBook Air and an iMac, as per his conversation with Lex Fridman: https://youtu.be/O9upVbGSBFo?t=2523
Re:Brian Kernighan adds Unicode support to Awk by amyjess on Saturday August 20, 2022 @09:03PM (#32536504)
It's not just the tooling of the modern world.

I remember installing teTeX (the predecessor of TeX Live) back in 2003 and I despaired at the file size because I knew it would take days to download on the shitty dialup connection I had at the time. Yeah, it wasn't 5 GB big back then, but it was certainly big by 2003 standards.

Re:Brian Kernighan adds Unicode support to Awk by cfiggers on Saturday August 20, 2022 @05:49PM (#32535434)
Watching this interview inspired me to start playing around with groff. It has a very steep learning curve... And being as old/niche as it is, I've found it very hard to find any active community to get newbie questions answered. If anybody knows where I could find that sort of thing, I'd be very grateful.
Re:Brian Kernighan adds Unicode support to Awk by neutrono on Saturday August 20, 2022 @08:24PM (#32536311)
NB: I'm new to HN, so please excuse any formatting errors in this comment.

Well, I've been using groff on and off for a few years, I can give it a shot. Most "macros" as in TeX, are invoked by having the first character of a line be a `.` and then as in TeX, refer to a name. Now, macros in groff are weird. Macros are not delimited by `{}` delimiters as in TeX, but either by having an environment block like so:

    .TS
    .TE
Incidentally, `.TS` and `.TE` is the environment for tbl(1), the pre-processor for tables. Or by invoking another macro, as in this example for headings and paragraphs in the -ms macro set. (The -ms macro set is called as such because that's how you get it into your document, you supply the -ms flag at build time to groff to get the -ms macros). See groff_ms(7) for more information than I will supply here.

    .NH 1
    Some heading here
    .PP
    Start of a paragraph.
Sometimes you want to start a line of input with a period. To do that you just use the character '\&' before the period. The `\&` character is a non printing character, so use it anywhere you want, really. I imagine you want to play with math typography, in that case there is EQN. EQN works by having an environment block, `.EQ` and `.EN` and in that block you type in the EQN mini-language.

    .EQ
    f ( x ) ~=~ x sup {2} - 1
    .EN
EQN does not care about precedence or anything, so use `{}` to delimit expressions. I think there is some kind of EQN document rescued from Bell Labs that tells you about how the language works. Check troff.org for it, or email the mailing list for groff, which is also pretty good for newbie questions, it is not really all that high traffic.

Groff also does pictures, so you can for example create a box with some text in it (it is capable of more, like strokes, macros, color, and I believe rotation[not sure]). To start a picture, `.PS` and `.PE` will do so, and again, a small language as input in that block.

Now, to print anything you should know the design of the original troff system was heavily based on the UNIX philosophy, which means the output you get from running the groff program prints the output to stdout. Just redirect the output to a file to get a PDF or a PS document.

Obviously, there is much I have not covered in this comment, such as macros, registers, and other exciting topics such as refer(1), but this should give you a quick start.

Re:Brian Kernighan adds Unicode support to Awk by cfiggers on Saturday August 20, 2022 @10:21PM (#32536857)
This is helpful! Thank you for taking the time to write it up.

I didn't know about '\&' as a way to "short circuit" groff from interpreting something as a macro that should be literal (such as a line-initial period). That's great to know.

I've found a couple of resources that have been helpful. Mostly, I've been referring to this[0] page on gnu.org. I've also learned from this[1] article on Linux journal and Luke Smith's YouTube playlist[2] on groff.

To practice, I'm trying to re-typeset a paper I wrote in school that follows the Chicago Manual of Style[3]. I formatted the original in MS Word using a template my school provided, so I have something to visually compare against. I'm using the -ms macro set, mostly because that's what Luke Smith demos in his videos so that's what I got started with.

I've already got a reasonable imitation of the original document, which is fun and a bit rewarding. But there's a couple of very small details that I'm finding difficult to crack. Are you very well-versed with -ms? In particular, right now I'm trying to figure out how to add one blank line of space in footnotes AFTER the horizontal line, but BEFORE the first footnote begins. I suspect I'll need to redefine the -ms macro that controls the trap for footnotes...? But I haven't figured out which macro that is just yet.

[0] "The GNU Troff Manual", https://www.gnu.org/software/groff/manual/groff.html

[1] "Typesetting with groff Macros", https://www.linuxjournal.com/article/4375

[2] "groff/troff for Minimalist Document Complication", https://www.youtube.com/playlist?list=PL-p5XmQHB_JRe2YeaMjPT...

[3] "The Chicago Manual of Style Online", https://www.chicagomanualofstyle.org/home.html

Re:Brian Kernighan adds Unicode support to Awk by neutrono on Saturday August 20, 2022 @11:26PM (#32537151)
Sadly I don’t know how to futz about with traps, I just haven’t gotten to that point yet in my use of groff.

First I would point you to the manpage, but, we all know that manpages are a mess, so it might not be there. I will however point you, perhaps not /super/ helpful, but alas, I know not of any better solution. If you go to /usr/share/groff/current/tmac/s.tmac and search for «module fn», that should be the code for the footnotes. Aside, the -ms name comes from this, -m means macro, and whatever letter comes after it is the macro set, hence why -man will be called an.tmac in groff source code, or in this case, s.tmac.

I am not sure where documentation is on traps in groff, but I assume it’s buried withing some manpage or info document somewhere. Software is crap, eh? :D

Of course, you could ask in the groff mailing list[1], they are usually pretty helpful in these matters. Which course of action you chose is of course up to you, I will investigate for myself as well to educate myself more on traps.

[1]: mailto:[email protected]

Re:Brian Kernighan adds Unicode support to Awk by ChrisRR on Tuesday August 23, 2022 @04:32AM (#32562040)
> He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!"

I'm struggling to understand this sentence. Who are they? What's a boonie? What's not finished?

Re:Brian Kernighan adds Unicode support to Awk by detaro on Tuesday August 23, 2022 @04:34AM (#32562046)
"in the boonies" = "in the wilderness/far away from everything/...". I.e. they probably have only crappy mobile internet and don't want to download large packages.
Re:Brian Kernighan adds Unicode support to Awk by stjohnswarts on Saturday August 20, 2022 @04:37PM (#32534988)
He could master it in a week if he set his head to it. I don't have one doubt of that. He just doesn't really need to.
Re:Brian Kernighan adds Unicode support to Awk by dredmorbius on Sunday August 21, 2022 @01:54PM (#32542639)
I just wanted to say: that's an excellent interview and I strongly recommend others listen to it.