News for hackers. Stuff that matters.
Re:On “Learning to Summarize” by simonebrunozzi on Saturday September 12, 2020 @06:13PM (#24456329)
I initially thought this would be about "humans" learning to summarize longer texts in a smart way. It's a topic dear to my heart; in fact, I even launched a little side project a few years back, called MNMN [0], with the aim of providing summarization of interesting articles or discussions on the web. I started with my own itch to scratch: Hacker News homepage, which I religiously read pretty much every day.

Here's an example of how the Hacker News home page would be summarized: [1].

Instead, this "learning to summarize" article is about another interesting topic, which is how you teach an AI (GPT-2 and GPT-3) how to summarize text. It's also - bear with me - something dear to my heart, as I used to teach compilers at a CS course in an Italian university in 2004-2006, and I developed an interest for languages in general (not just computer ones).

This one below is the central point of the article, and it is indeed a crucial part of having success with an AI (note: LM stands for Language Model):

> IMO there are two almost unrelated ideas going on in OpenAI’s preference learning work.

> First, the idea of collecting binary preference annotations on LM samples, and (in some way) tuning the LM so its samples are better aligned with the preferences.

> Second, a specific method for tuning the sampling behavior of LMs to maximize an (arbitrary) score function defined over entire samples.

We are, IMHO, at the cusp of a true revolution in linguistics. Can't wait to see what happens in the coming 18-24 months. I expect to be blown away on at least a few fronts.

[0]: https://github.com/simonebrunozzi/MNMN

[1]: https://github.com/simonebrunozzi/MNMN/blob/master/Weekly-Su...

Re:On “Learning to Summarize” by Zakuzaa on Sunday September 13, 2020 @07:26AM (#24460117)
Wow, I am trying to do something similar [0]. I have waited for someone to build something like this; giving it a go myself.

[0] https://thebulletpoints.substack.com/

Re:On “Learning to Summarize” by DenisM on Sunday September 13, 2020 @02:35AM (#24458880)
How much traction/revenue would you need to keep mnmn alive? Like, if you made $1000 per day you would probably keep it up?
Re:On “Learning to Summarize” by simonebrunozzi on Thursday September 17, 2020 @08:44PM (#24511754)
My initial analysis concluded that I could keep it alive, JUST for Hacker News "summaries", for something in the ballpark of $100,000/year. I'm sure you could do it cheaper (like 60-70k), but that budget includes consistent quality, which requires at least one full-time editor, and at least one part-time sw dev.
Re:On “Learning to Summarize” by Zakuzaa on Sunday September 13, 2020 @07:23AM (#24460106)
$1000 per day?
Re:On “Learning to Summarize” by DenisM on Sunday September 13, 2020 @12:33PM (#24461977)
Yes, that amounts to $250,000 year.
Re:On “Learning to Summarize” by Zakuzaa on Sunday September 13, 2020 @10:23PM (#24466121)
Hmm doesn't sound that much now that I think.
Re:On “Learning to Summarize” by binarymax on Sunday September 13, 2020 @01:02PM (#24462238)
It’s interesting that the title seems like a mix on Learning to Rank, in which a preferred ordering is learned for relevant ranking of search results...but doesn’t mention this anywhere in the article! As noted, you don’t want to do RL if you can avoid it - so I’m wondering if learning decision trees (common in LTR with LambdaMART) can be helpful with summarization to best fit the annotator preferred passages over others. Perhaps the title name is just a coincidence but there is value in exploring, for example, random forests...which are way easier (and faster) than things like RL or CNNs
Re:On “Learning to Summarize” by supernova87a on Saturday September 12, 2020 @08:10PM (#24457051)
Wow, that article had nothing to do with the title, for a normal person reading it.

It should really get a much more technical title, befitting the dissertation it leads to.