News for hackers. Stuff that matters.
Re:Creepy Crawlies by chuckadams on Sunday August 30, 2026 @10:36AM (#49499053)
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
Re:Creepy Crawlies by rcxdude on Sunday August 30, 2026 @11:26AM (#49499533)
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
Re:Creepy Crawlies by charcircuit on Monday August 31, 2026 @12:44AM (#49505688)
Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.
Re:Creepy Crawlies by kees99 on Sunday August 30, 2026 @11:01AM (#49499260)
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.

Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.

[0] https://git.zx2c4.com/cgit/about/

Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @11:33AM (#49499595)
he said its fast enough for now actually...
Re:Creepy Crawlies by Zariel on Sunday August 30, 2026 @10:39AM (#49499074)
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
Re:Creepy Crawlies by teo_zero on Sunday August 30, 2026 @10:43AM (#49499104)
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
Re:Creepy Crawlies by kijin on Sunday August 30, 2026 @11:10AM (#49499351)
Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @04:03PM (#49502245)
no its only difficult for you. and only at this moment... any minute now you will see the way. btw, the OP is just saying "its OK for now". And the OP is just telling us: this is what has been happening... maybe its difficult for OP also, but they didnt say that. they just said their current setup cant handle it. old tech...
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @06:58PM (#49503666)
remember, its only difficult for you. at the end of the day its just 7gb of data...
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @04:06PM (#49502261)
it can be solved in a few days, just the OP said its ok for now
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @11:38AM (#49499648)
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @04:00PM (#49502219)
anyway, I have solved this exact scenario many times in the last 30 years... thanks for the dv
Re:Creepy Crawlies by mik1998 on Sunday August 30, 2026 @11:06AM (#49499312)
The cache gets evicted between the times the old commits are revisited.
Re:Creepy Crawlies by oowa on Sunday August 30, 2026 @04:02PM (#49502239)
using the current solution and a cache pattern doesnt solve it, but it is an "affordable" solvable problem. but first we need to ignore OP said its ok for now...
Re:Creepy Crawlies by skybrian on Sunday August 30, 2026 @12:47PM (#49500307)
Apparently the current implementation is very inefficient by modern standards:

https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j

Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.

Re:Creepy Crawlies by edward28 on Monday August 31, 2026 @02:03PM (#49512752)
Maybe it's time for AI to rewrite it in rust.
Re:Creepy Crawlies by throwawayffffas on Monday August 31, 2026 @06:45AM (#49508123)
My first thought was put varnish in front of it as well. But given how wide the url scheming is, the valid urls seem to be in the quintillions, if i counted correctly, its unlikely a naive caching solution would work.

But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.