News for hackers. Stuff that matters.
Re:Show HN: CLI tool for saving web pages as a single file by mikaelmorvan on Friday August 23, 2019 @04:43AM (#20775603)
The main problem with your code is that you only handle simple web1 site.

What about javascript execution ? If you replay your capture, you have no idea of what you will see on general Web2 website.

The only way I know to capture a web page properly is to "execute" it on a browser.

Gildas, the guy behind SingleFile (https://github.com/gildas-lormeau/SingleFile) is well aware of that and his approach realy works everytime.

Try on a Facebook post, a Tweet, ... It just works.

Re:Show HN: CLI tool for saving web pages as a single file by lucideer on Friday August 23, 2019 @07:35AM (#20776421)
The capture includes JS, so this should work for most JS-dependent sites, with the exception of scripts loading other additional assets.

Tbh, often those are superfluous, or egregious examples of bad web dev, so it seems a reasonable solution for most cases.

SingleFile is a different approach, but it's a lot more involved/less convenient than a cli, and loading in something like WebDriver on the cli for this would be overkill, unless you're doing very serious archival work.

Re:Show HN: CLI tool for saving web pages as a single file by mikaelmorvan on Friday August 23, 2019 @08:51AM (#20776955)
superfluous, or egregious examples of bad web dev?? Do you know what Web 2.0 is? Do you know what are React, Angular, and the other JS Frameworks?

When you create a modern webapp, a lot of data are retrieved from servers as Json and formated in the browser in Javascript. Even sometimes Css is generated on browser-side. Even more, on webapp where user login is taken into account, the display is modified accordingly.

That's the web of 2019. The approach consisting of geting remote files and launching them in a browser is really naive.

Speaking of SingleFile, it as a cli version and can handle full web 2.0 webapp without any problem. And of course, the Web 1.0 webapps work as well.

Re:Show HN: CLI tool for saving web pages as a single file by lucideer on Friday August 23, 2019 @09:21AM (#20777247)
With the exception of actual XHR requests (which should ideally be for dynamic resources, and as such somewhat outside the remit of saving a webpage), I was referring specifically to JS loading JS, etc. solutions. React, Angular do not recommend/advise you to do this. This isn't a requirement in Web 2.0 or Web 5.0 or anything else.

In terms of React at least, fetch requests are not a part of the framework in any way and any present would typically be done in custom code in lifecycle methods. Even Redux, is—by default—client-side only. Stores are in-memory, actions populating them would make fetch requests with React/Redux-independent logic.

Other JS frameworks are, typically, the same. And all of that is just considering dynamic XHR. Loading scripts is much less typical, and never required. The most common application of this I've seen is the GA snippet, which mainly does it to ensure the load is async without relying on developer implementation: it's 100% unnecessary to do it this way.

So yes, unless you're distributing a tracking snippet that you expect non-devs to be blindly pasting into their wordpress panels and still have it work efficiently, generally speaking use of this method is never necessary, and commonly a red flag for poor architecture.

Re:Show HN: CLI tool for saving web pages as a single file by wolco on Friday August 23, 2019 @09:18AM (#20777214)
Sometimes we spent too much time in our own ecosystem. In 2019 most sites globally still use php and jquery.

Web 2.0 refers to the use of ajax. This refers to the early 2000s sajax, jquery..

If you want to separate angular, react, vue maybe it's web 3.0.. but wasn't web 3.0 referred to as mobile?

Re:Show HN: CLI tool for saving web pages as a single file by tinsx on Friday August 23, 2019 @09:02AM (#20777047)
I think that's exactly what that person means by superfluous and egregious examples of bad web development; SPAs, javascript frameworks of that nature. :p
Re:Show HN: CLI tool for saving web pages as a single file by mikaelmorvan on Friday August 23, 2019 @09:07AM (#20777099)
Yes, the debate between building a SPA with rich features or Web old pages with good SEO is eternal :) We see more and more an hybrid approach that can be called web 1.5 :)
Re:Show HN: CLI tool for saving web pages as a single file by baby on Friday August 23, 2019 @11:41AM (#20778721)
So any ajax and it won't work?
Re:Show HN: CLI tool for saving web pages as a single file by lucideer on Friday August 23, 2019 @04:09PM (#20782058)
Any ajax won't work offline.

Ajax will* still work fine with an internet connection as long as those ajax endpoints don't require cookies and don't linkrot.

* not 100% sure how the tool handles relative URLs embedded in source : if it's not clever enough though, this is very fixable via PR (as in its not an architectural limitation)

Re:Show HN: CLI tool for saving web pages as a single file by baby on Friday August 23, 2019 @04:40PM (#20782317)
What's the point of saving a webpage if it won't work offline?
Re:Show HN: CLI tool for saving web pages as a single file by chiefalchemist on Friday August 23, 2019 @09:01AM (#20777040)
I hear ya but another way to look at this is...

The main problem with too many websites is they've become too much about technology and have left visitors, as well as the spirit and intent of the internet behind.

Re:Show HN: CLI tool for saving web pages as a single file by UI_at_80x24 on Friday August 23, 2019 @09:23AM (#20777260)
It stopped being about information and started being entertainment.

I agree with you. BadSite: "we want you to experience this.."

GoodSite: "we want you to learn this..."

Re:Show HN: CLI tool for saving web pages as a single file by chiefalchemist on Friday August 23, 2019 @11:38AM (#20778705)
To clarify myself a bit. Many of the new technologies are great...for building applications; applications where UI and UX are essential.

But for run-of-the-mill websites? Such tools are not only overkill, they break the spirit of the internet.

Re:Show HN: CLI tool for saving web pages as a single file by h1d on Friday August 23, 2019 @08:08AM (#20776633)
Is there a CLI version of it?
Re:Show HN: CLI tool for saving web pages as a single file by dotneter on Friday August 23, 2019 @08:14AM (#20776684)
Re:Show HN: CLI tool for saving web pages as a single file by chiefalchemist on Friday August 23, 2019 @09:05AM (#20777076)
Does it also capture tracking crap? Wouldn't it be great to grab a URL via CL and read it local without anyone knowing?

If it does capture the tracking shite, would disconnecting from the internet be a good enough blocker?

Re:Show HN: CLI tool for saving web pages as a single file by gildas on Friday August 23, 2019 @09:12AM (#20777157)
By default, it will capture everything that would be interpreted/displayed. However, you can also run it with Selenium (and Firefox) instead of Puppeteer, and install your favorite extensions to block all the unwanted resources (there's a switch for that).
Re:Show HN: CLI tool for saving web pages as a single file by lucasverra on Tuesday September 03, 2019 @05:29AM (#20864918)
Puppeteer and FF will be a thing officialy by Chromium https://github.com/GoogleChrome/puppeteer/tree/master/experi...
Re:Show HN: CLI tool for saving web pages as a single file by flatroze on Friday August 23, 2019 @04:57PM (#20782482)
Everything has its limits