tamnd

Fixed: https://github.com/tamnd/tomo-labs/blob/main/docs/content/ex...

I built these tools because I'm tired of recent tools being built for AI rather than for humans. More are coming soon!

It is a sibling project of https://github.com/tamnd/kage, in case you missed it.

Sorry, the docs come first, and I will set up Brew and Scoop later. In the meantime, you can use the pre-built images from the releases or use Go to build from source.

Possible, but currently I disable all large files, including videos.

For video downloading, I suggest wrapping around yt-dlp. It's an awesome tool.

If there's more demand for that, maybe I will implement a more relaxed version.

Currently, all of that is broken. At one point, I had a traumatic experience where an archived HTML file kept redirecting to the live site, even though I already had all the content rendered, so I ended up disabling all JavaScript entirely.

Good news for you: here is the command to clone Apple Docs:

```bash bin/kage clone https://developer.apple.com/documentation/ \ --scope-prefix /documentation/ \ --out /Users/apple/data/apple-docs \ --chrome "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ --max-pages 0 --max-depth 0 \ --workers 3 --browser-pages 3 --asset-workers 6 \ --render-timeout 60s --settle 2s --timeout 30s \ 2>&1 | tee -a /Users/apple/apple-docs.log ```

Adjust it to your needs :)

I smoke-tested it, and all the content and CSS work, but I stripped all the JS, so the sidebar won't work.

If you run into any problems, feel free to create new issues in the repo. It helps me prioritize and know what should be fixed.

It should be fixed by https://github.com/tamnd/kage/pull/12

Thanks for nice trick.

Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.

This could be a nice code golf project. It only needs a webview, a ZIM reader, and a way to append data to an existing binary and read it back.

I did something like that a very long time ago (Of course, I have forgotten)

For sharing, better use the html folder or zim format, Kage supports both of them.

I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.

Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple:

https://github.com/tamnd/kage/blob/main/Dockerfile

Btw, let me think the way to only enable this when running inside Docker.

Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too.

I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large.

By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.

Thanks ;)

Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file.

The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.

This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))

You could use python -m http.server instead. I haven't tried it yet, but it should work.

Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.