News for hackers. Stuff that matters.
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @01:10PM (#29463408)
Got you, fam.

https://encyclopedia.marginalia.nu/wiki-start.html

(I built this mostly for myself, because I find it incredibly difficult to read text that has hyperlinks and images in the text)

Re:Show HN: I made a modern web UI for Wikipedia by ciphol on Monday December 06, 2021 @04:45PM (#29466114)
That is very nice looking. But I want the hyperlinks because the information I want is often spread out over more than one page (often Wikipedia achieves more concision than a traditional encyclopedia by hyperlinking rather than defining the terms it uses). And I want references because otherwise the information is unreliable. And I often want images because they sometimes convey information better than any text ("an image is worth a thousand words").
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @04:53PM (#29466199)
Regular wikipedia is still there if those are your preferences. I don't know, I've never check a single reference in over a decade of using Wikipedia. I don't have access to those references. Maybe you do, if so that's cool.
Re:Show HN: I made a modern web UI for Wikipedia by dsizzle on Monday December 06, 2021 @01:56PM (#29464024)
I'm with you on links being distracting, but it seems you... nuked them entirely, which maybe goes too far haha? I guess you can always go back to source if you want that.
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @02:02PM (#29464101)
I just moved them to the bottom of the article.

In part this is because I feel inline links on wikipedia often lets you overestimate how well you comprehend a subject. It's as though the fact that there is a link over a word makes you think you could go check what that is, and therefore that you probably understand the word.

Actually looking up some term mid sentence is also probably not particularly helpful for understanding the text either.

I concede that this is a somewhat unusual decision, but I designed the pages for the singular purpose of reading and comprehending the article. Everything else is secondary. It's a bit of a design experiment, but I think it's worked out pretty well. I don't fall into skim-reading these articles like I must admit I often do with regular Wikipedia articles.

Re:Show HN: I made a modern web UI for Wikipedia by dsizzle on Monday December 06, 2021 @03:01PM (#29464826)
Oh, ha, I didn't even notice that. Ok, I like that!

I was actually looking at the footnote links, and those do indeed seem to be missing entirely -- as well as the whole references section. Arguably that's more important than links to "definition" wiki pages, so it forms the basis for the knowledge (vs mostly just definitions).

Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @03:13PM (#29464973)
Yeah, I removed those entirely, as I very rarely use those. More often than not the links are pay walled and the references are books that may or may not even be in print. Maybe it's useful if you have a university library at hand but I don't so I simply live with the fact that some of the things I've read may be wrong.

If you want to look up sources it's better to use the live wikipedia at any rate, since it will be more up to date than this snapshot.

Re:Show HN: I made a modern web UI for Wikipedia by dsizzle on Monday December 06, 2021 @03:23PM (#29465111)
Ehh, I would say it's useful to see the reference even if you don't follow the link, but I'm sure it also depends on the nature of the topic. Seems easy to just preserve that section, but, hey, it's your site. Maybe there's a cost aspect you're considering? On that note, I'm curious how much it costs to basically mirror all of Wikipedia, if you wouldn't mind sharing?
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @03:34PM (#29465267)
I do a lot of these design experiments. I have an idea such as "what if wikipedia was designed only with reading comprehension in mind" and then I see what taking it to the most ridiculous extreme does. Sometimes it pans out, like I'm fairly happy with this wikipedia mirror, sometimes it doesn't--like I have no logos or branding anywhere on my sites, that turned out to be a bit confusing.

My mirror clocks in at 21 Gb. I'm self hosting so there's no real money cost besides the one-time investment in hardware. A really chap person could probably host this off a raspberry pi attached to something like a Corsair Voyager and serve multiple requests per second.

There's some technical trickiness in actually storing the files in a file system, since most filesystems don't deal well with having millions of files in the same folder. So I've had to build a four-tier directory structure based on the hash of the file name to be able to store the files. They're stored in a structure like 31/444/781/225/foobar.gz.

I did some experiments storing them as BLOBs in a database, but I couldn't get it to work.

Re:Show HN: I made a modern web UI for Wikipedia by dsizzle on Monday December 06, 2021 @03:59PM (#29465630)
Interesting. What are the bandwidth considerations for this? I had the impression that many ISPs make self-hosting impractical on normal household plans, but then I've never looked into it seriously.
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @04:11PM (#29465788)
Dunno, I have unlimited uploads and downloads @ 100 mbit, and I host both this Wikipedia mirror as well as a search engine and some other stuff.

The bandwidth usage is pretty inconsequential since most of my pages clock in at just a few kilobytes.

Re:Show HN: I made a modern web UI for Wikipedia by JasonFruit on Monday December 06, 2021 @02:46PM (#29464613)
I love what you've done with the links, and will use this, I think. Thank you for your excellent work!
Re:Show HN: I made a modern web UI for Wikipedia by fallat on Monday December 06, 2021 @01:18PM (#29463503)
This is kind of incredible.

Have you considered images just take up full width?

Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @01:25PM (#29463584)
Half of cutting the images is a space saving thing.

Wikipedia is a pretty big website, and (like the rest of my web presence) I'm hosting the mirror on a single small consumer hardware server. I'm keeping all the wikipedia pages as compressed HTML, and if you look at the sources you'll see it's very bare bones, most pages are like 50 Kb, yet the total is still 21 Gb. I don't think I have the space or bandwidth for selfhosting all of wikipedia's images.

Part of this experiment is to illustrate just how ridiculously large page loads on a lot of pages. Even text-heavy ones like Wikipedia.

If you compare the HTML payload alone of these two renditions of largely the same text, it's a 50x difference. The full Wikipedia page load exceeds 1 Mb, mine isn't even 10 Kb.

https://en.wikipedia.org/wiki/Scamander

https://encyclopedia.marginalia.nu/wiki/Scamander

That's kinda ridiculous.

Re:Show HN: I made a modern web UI for Wikipedia by fallat on Monday December 06, 2021 @11:41PM (#29469126)
Gotchya. All good then. Excellent work and insight :)

Have you attempted to convert them to plain text and serve those? Or is there no real savings there?

Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Tuesday December 07, 2021 @02:52AM (#29470019)
Would probably lose more than you'd gain doing that. The HTML right now is so clean you can read it without a browser, almost looks like it might be hand written, while still preserving things like formulas and tables (which do need a browser).
Re:Show HN: I made a modern web UI for Wikipedia by kingcharles on Monday December 06, 2021 @02:45PM (#29464605)
Thanks, bestie!
Re:Show HN: I made a modern web UI for Wikipedia by 0des on Monday December 06, 2021 @01:16PM (#29463473)
You complete me <3
Re:Show HN: I made a modern web UI for Wikipedia by marginalia_nu on Monday December 06, 2021 @01:18PM (#29463498)
I should say it's a bit janky in some places, but for most articles it works reasonably well.