News for hackers. Stuff that matters.
Re:HTTrack Website Copier – Offline browser by boulos on Saturday July 10, 2021 @01:29AM (#27791341)
Ahh, the good ole days. I used httrack nearly 20 years ago to make the CD copies of the osha.gov site (e.g., [1]). Back then, for ADA and internet access compliance, government websites had to also be made available for offline use if possible.

I haven’t followed httrack since, but it seems like scrapy and similar are much better replacements.

[1] https://forum.httrack.com/readmsg/3556/index.html

Re:HTTrack Website Copier – Offline browser by xroche on Saturday July 10, 2021 @05:03AM (#27792141)
Glad the project helped a bit a few people :) I don't have much time unfortunately to enhance the engine nowadays, and the code is dirty and broken beyond any repair. Yet I'm still puzzled to see how many people are still using the project today.

You'll probably find better approaches, and while I never tried scrapy, it seems to be using a javascript engine for hard cases, which was something I thought about (but this was way above my skills at that time).

The hard parts remains however, if you want a functional site: you need to rewrite links, or use an external proxy-like mechanism. Having a fully functional offline, file-based site, is the real tricky part. Cases will remain unsolvable, as the inside code logic can produce whatever external link resource based on randomness, time, etc.

The approach in httrack was both ugly and pragmatic: attempting to recognize link/files patterns within javascript and fetch/replace what can be replaced with local links. Javascript producing html will typically be analyzed with really dumb - yet sometimes effective - js parsers. (parental advisory: don't look at the parsers code, your eyes would melt)

And obviously this is not going to solve all cases and will even break pages with tricky js

Re:HTTrack Website Copier – Offline browser by boulos on Saturday July 10, 2021 @12:10PM (#27794370)
Let me be clear: Thanks, Xavier!

httrack was extremely helpful and there really was no equal. The “modern” web requires a live JS engine, but as you point out, even the “old” web had server-side logic that couldn’t be captured.

In that light, I think httrack has stood up pretty well and nobody expects you to go rewrite it or clean it up. If someone today has a mostly static site they want to archive without writing custom code, I would still recommend httrack (it’s more controllable than wget or similar). I just assume that those sites are mostly gone :(.

Re:HTTrack Website Copier – Offline browser by robtherobber on Saturday July 10, 2021 @07:59AM (#27792775)
I'm using this software on and off, but it's especially useful when clients plan to redo their websites and I want to make sure I have a copy of the pages offline but don't have access to server backups or things like that.

But, generally speaking, being able to preserve "the internet" by saving whole websites offline should be something we give more attention to.

Just read this recently: https://www.theatlantic.com/technology/archive/2021/06/the-i...

Re:HTTrack Website Copier – Offline browser by enqk on Saturday July 10, 2021 @07:59AM (#27792776)
I still use it to mirror websites :) After all I also witnessed its creation! Dirty code can still be super useful
Re:HTTrack Website Copier – Offline browser by fouc on Saturday July 10, 2021 @04:19AM (#27791961)
The good ole days when web pages were web pages!
Re:HTTrack Website Copier – Offline browser by dzign on Saturday July 10, 2021 @10:04AM (#27793412)
And the information was under a unique domain...
Re:HTTrack Website Copier – Offline browser by wilsonfiifi on Saturday July 10, 2021 @01:15AM (#27791283)
I’v recently been using Monolith[0] and I find it’s creation of a single html file much more convenient. It’s also written in Rust so I’m sure that will make the source code a bit more accessible for some.

[0] https://github.com/Y2Z/monolith

Re:HTTrack Website Copier – Offline browser by kybernetikos on Saturday July 10, 2021 @05:11AM (#27792172)
It doesn't look like it follows links though, so it's much more of an alternative to "Save As -> mhtml" than to HTTrack.
Re:HTTrack Website Copier – Offline browser by chowderman on Saturday July 10, 2021 @10:46AM (#27793654)
There is also a similar program called HyperFiler[0]* that bundles web pages into single HTML files with a few more options such as a headless chromium transport option, built in minifiers, page sanitizers, and an option to grayscale the output pages, among other options. It's TypeScript based and has an programmatic API to customize the bundling process as well.

[0] https://github.com/chowderman/hyperfiler

* disclaimer: I created HyperFiler

Re:HTTrack Website Copier – Offline browser by AnyTimeTraveler on Saturday July 10, 2021 @03:11AM (#27791687)
Cool! Thanks for sharing! I've been on the lookout for replacements for archiving recipies on cooking sites and this tool works great.
Re:HTTrack Website Copier – Offline browser by Gys on Saturday July 10, 2021 @03:39AM (#27791830)
There is also SingleFile as a browser extension: https://github.com/gildas-lormeau/SingleFile
Re:HTTrack Website Copier – Offline browser by gildas on Saturday July 10, 2021 @09:30AM (#27793229)
Note that SingleFile can also crawl websites when run from the CLI (see the --crawl-* options)
Re:HTTrack Website Copier – Offline browser by robtherobber on Saturday July 10, 2021 @08:00AM (#27792785)
I also recommend this wholeheartedly.
Re:HTTrack Website Copier – Offline browser by Minor49er on Saturday July 10, 2021 @02:26AM (#27791541)
I used to use HTTrack pretty often to save entire sites. But I learned that wget can take care of my copying needs even more easily most of the time. Something like this usually does the trick:

  wget -E -r -k -p --span-host http://mycoolhomepage.com
Re:HTTrack Website Copier – Offline browser by mattowen_uk on Saturday July 10, 2021 @07:26AM (#27792645)
Wow, I didn't think people still use this. I have a copy on my PC, and wheel it out every so often, when I find a great small site that quite obviously has been abandoned by it's owner so could vanish any moment, and Wayback doesn't have a full copy. I could switch to something better, but I know how HTTrack works, and it works well enough for me.
Re:HTTrack Website Copier – Offline browser by slumdev on Saturday July 10, 2021 @08:24AM (#27792893)
You know what might be a great add-on?

A tool to crawl all of the links within a website and submit each one of them to Wayback...

Re:HTTrack Website Copier – Offline browser by bagpuss on Saturday July 10, 2021 @09:25AM (#27793200)
https://archivarix.com/

is an interesting solution to GP's issue, works better than httrack and can also pull down multiple timestamps of archived site

Re:HTTrack Website Copier – Offline browser by bradknowles on Monday July 12, 2021 @03:17AM (#27807886)
The ArchiveTeam has tools that do that. In fact, they have a number of different tools, some of which I believe are closely related to httrack.

Disclaimer: I tried to get involved with ArchiveTeam to help get some websites that I care about properly archived and saved to The Way Back Machine, but they weren’t allowing new members to sign up. They were happy to talk to me about what needed to be done and one of their members set things up to archive everything that was available, but I wasn’t able to help in that process.

Re:HTTrack Website Copier – Offline browser by notRobot on Friday July 09, 2021 @10:27PM (#27790649)
HTTrack is one of my favourite pieces of software - it makes it super easy to create offline mirrors of websites and browse them later. It's sorta like wget on steroids for that use case.
Re:HTTrack Website Copier – Offline browser by mosselman on Saturday July 10, 2021 @02:43AM (#27791612)
Wget has the —mirror flag which makes it much like httrack for minor scrapes. Httrack is faster because it can work in parallel.
Re:HTTrack Website Copier – Offline browser by icythere on Saturday July 10, 2021 @01:41AM (#27791396)
I used httrack to transform the public version of my wordpress blogs to a static site. It often crashed but as long as I had a copy of its local data(base) it's just fine to restart it.

I really like the tool. I doubt if that is helpful today, bc. of the raise of the Javascript stuff...

Re:HTTrack Website Copier – Offline browser by daniel_iversen on Friday July 09, 2021 @11:05PM (#27790803)
I looked into it the other day but AFAIR it easily breaks down if a site is using Cloudflare to protect itself from abuse, which I imagine could be quite a fair few sides these days.
Re:HTTrack Website Copier – Offline browser by Scoundreller on Friday July 09, 2021 @11:29PM (#27790897)
I wonder if there’s a way to integrate it with my own browser so it can mirror a website over days/weeks by using my regular behaviour to avoid suspicion.
Re:HTTrack Website Copier – Offline browser by luismedel on Saturday July 10, 2021 @04:02AM (#27791912)
You can set the user agent, randomize link retrieval order and set a longer delay between requests to avoid detection.
Re:HTTrack Website Copier – Offline browser by klkvsk on Saturday July 10, 2021 @11:17AM (#27793938)
I used to use HTTrack, but then I found Cryotek WebCopy[1]. It's basically same, but with more user friendly GUI and some more features useful for modern web. Like for example, it can fetch resources by URLs contained in JS code or from special attributes like data-src. And it's free, but available only for Windows, though.

[1] https://www.cyotek.com/cyotek-webcopy

Re:HTTrack Website Copier – Offline browser by DerekBickerton on Saturday July 10, 2021 @10:03AM (#27793408)
The SingleFile[0] Firefox addon is handy too, if you just want to archive a webpage with all the images, styling etc intact all enclosed in a single HTML file.

[0] https://addons.mozilla.org/en-US/firefox/addon/single-file/

Re:HTTrack Website Copier – Offline browser by Gys on Saturday July 10, 2021 @03:48AM (#27791868)
Similar project (cross platform commandline): https://github.com/wabarc/wayback
Re:HTTrack Website Copier – Offline browser by cubano on Saturday July 10, 2021 @12:28AM (#27791118)
HTTrack is a powerful package, but unless it can now handle JS-embedded links properly, it still can't take you all the way to. perfect local site mirroring.
Re:HTTrack Website Copier – Offline browser by gengear on Saturday July 10, 2021 @12:41AM (#27791163)
Used it in undergraduate to mirror courses. Very good. doesn't work with js links or SPA. But a nice to have tool if you are into self hosting.
Re:HTTrack Website Copier – Offline browser by cyberge99 on Saturday July 10, 2021 @12:20AM (#27791079)
How does this differ from Save As in a browser?
Re:HTTrack Website Copier – Offline browser by okareaman on Saturday July 10, 2021 @12:26AM (#27791110)
There's no comparison. It's way more powerful than that, but careful what you point it at because it will download gigabytes of everything and might annoy the website owner.
Re:HTTrack Website Copier – Offline browser by judge2020 on Saturday July 10, 2021 @12:23AM (#27791101)
Seems to be more powerful, eg. it can spider an entire website and save all linked pages and assets.
Re:HTTrack Website Copier – Offline browser by EvilGretzky on Saturday July 10, 2021 @12:02AM (#27791017)
What makes this better than DarcyRipper?
Re:HTTrack Website Copier – Offline browser by judge2020 on Saturday July 10, 2021 @12:24AM (#27791105)
For one, that software's website seems to be down - http://darcyripper.com/