News for hackers. Stuff that matters.
Re:Curl vs. Wget by JonathonW on Wednesday March 02, 2016 @05:51PM (#11213703)
For my usage:

* Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't.

* Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line tool is more unix-y by default (outputs to stdout) and it's more flexible in terms of options. It's also present by default on more systems-- of note, OSX ships curl but not wget out of the box. Its backing library (libcurl) is also pretty nifty, but not really relevant to this comparison.

This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow.

Re:Curl vs. Wget by manyxcxi on Wednesday March 02, 2016 @09:35PM (#11214675)
> This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow.

Totally agree. I love curl for testing API request/responses manually. It's usually a huge part of navigating my way around a new API that doesn't have a client library available for whatever language I'm using at that time.

I also use it for weird requests that need special headers or authentication.

Wget is the first thing I turn to when I'm downloading anything remote from the command line or scraping some remote content for analysis.

Re:Curl vs. Wget by majewsky on Thursday March 03, 2016 @03:45AM (#11215710)
Random plug: Another invaluable tool for API analysis is http://mitmproxy.org.
Re:Curl vs. Wget by mhils on Thursday March 03, 2016 @09:41AM (#11217028)
Thanks for mentioning it! :) One of the authors here, happy to answer any questions.
Re:Curl vs. Wget by hobarrera on Thursday March 03, 2016 @08:41AM (#11216726)
Wow, this looks extremely interesting, I'll definitely keep this in mind!
Re:Curl vs. Wget by mschuster91 on Thursday March 03, 2016 @03:07AM (#11215601)
Use the Chrome Postman extension together with Postman Interceptor. It's really awesome.
Re:Curl vs. Wget by hobarrera on Thursday March 03, 2016 @08:38AM (#11216714)
I'd just like to mention HttpRequester here, which is a very similar FLOSS addon for Firefox.
Re:Curl vs. Wget by jsingleton on Thursday March 03, 2016 @05:30AM (#11216024)
Yes, wget is fantastic for mirroring www and ftp sites and I use it a lot for that purpose. It's magic [0]. I hadn't realized that it didn't support compression though, which might explain why it's so slow in some cases. Not normally a problem as it just runs in the background on a schedule.

Curl supports gzip and deflate. It would be great if support for sdch and br were added too. Brotli is in Firefox 44 and can be enabled in Chrome Canary with a flag. SDCH has been in Chrome for a while and is used on Google servers.

[0] http://www.litkicks.com/WgetMagic

Re:Curl vs. Wget by stephengillie on Thursday March 03, 2016 @05:43AM (#11216060)
The Win64 latest version of curl doesn't seem to support gzip nor deflate. I have to remove those options when copying from Chrome developer tools and pasting into a script. I'd report a bug but their page doesn't seem to have an obvious link.
Re:Curl vs. Wget by detaro on Thursday March 03, 2016 @07:59AM (#11216548)
Re:Curl vs. Wget by voltagex_ on Thursday March 03, 2016 @10:37AM (#11217446)
I wonder if the Win64 version is built without zlib or similar.
Re:Curl vs. Wget by cathexis on Thursday March 03, 2016 @02:14AM (#11215474)
Pretty much. If I want to save a file: wget. If I want to do _anything_ else: curl. Yes you can write files with curl, no I don't use that functionality very often. I don't think of them as "end user" vs "developer" use cases so much as them being two great tools for different tasks. I do wish that -s was the curl default, since that stderr progress output is pretty lame.
Re:Curl vs. Wget by girvo on Wednesday March 02, 2016 @09:39PM (#11214684)
Interestingly, I've somewhat replace my usual curl calls with HTTPie, at least on my mac; for distributed scripts it's still curl all the way.
Re:Curl vs. Wget by eli on Wednesday March 02, 2016 @06:07PM (#11213798)
Doesn't curl -C continue an incomplete download? Regardless agree that wget is better for just downloading a file.
Re:Curl vs. Wget by toupeira on Wednesday March 02, 2016 @06:41PM (#11213998)
-C requires you to specify the byte offset, and you also need --append or >> so it's not quite the same.
Re:Curl vs. Wget by hmpc on Wednesday March 02, 2016 @06:52PM (#11214056)
Actually no. From the manpage:

Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.

Re:Curl vs. Wget by foota on Thursday March 03, 2016 @03:43AM (#11215701)
Stupid question, but how do things like this resume from where they left off? Wouldn't the server need to be cooperating in this? Is that build into HTTP?
Re:Curl vs. Wget by laumars on Thursday March 03, 2016 @04:11AM (#11215778)
Yes the server would need to support it. The request is made via a HTTP header (Range[1] IIRC).

Also, I wouldn't consider that a stupid question. :)

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#Rang...

Re:Curl vs. Wget by hobarrera on Thursday March 03, 2016 @08:42AM (#11216738)
This is also how download accelerators worked (back in the late nineties, early naught's), by having different connections work at several ranges to maximize bandwidth usage.
Re:Curl vs. Wget by laumars on Thursday March 03, 2016 @10:43AM (#11217497)
Funny enough, it was writing a download accelerator that taught me about the Range HTTP request header.
Re:Curl vs. Wget by zo1 on Thursday March 03, 2016 @04:07AM (#11215772)
You specify a byte "range" when requesting a file via HTTP from the server. Not all servers support this, but most do these days.

#Edit. More info here: https://en.wikipedia.org/wiki/Byte_serving

Re:Curl vs. Wget by viperscape on Thursday March 03, 2016 @06:13AM (#11216146)
Just looked that up, that's a rather limited usefulness. The multi homed usage makes sense but feels out of place really.
Re:Curl vs. Wget by joosters on Thursday March 03, 2016 @06:57AM (#11216289)
Why? How could it be more useful? HTTP byte ranges are incredibly flexible, since you one request can specify many byte ranges at once (it's almost too flexible, since a dumb server can easily be overwhelmed by a maliciously complicated request)

It handles the basic case of fetching the remainder of an interrupted download, and can also support partial downloads e.g. for supporting a video stream with the user jumping to different places in the movie.

Re:Curl vs. Wget by raverbashing on Thursday March 03, 2016 @12:14AM (#11215163)
Correct, but very unintuitive
Re:Curl vs. Wget by lmz on Thursday March 03, 2016 @12:54AM (#11215266)
Also doesn't work with the command line utility's retry option as it gets the size only once and does not update it when retrying.
Re:Curl vs. Wget by dsmithatx on Wednesday March 02, 2016 @06:16PM (#11213846)
The article is not coming up for me. Perhaps it's ycombinated. Anyway I agree curl is more of a developer tool although using it to download files is not the first thing that comes to mind. I use it daily to identify caching issues and redirect issues. The -SLIXGET flags in particular are very useful for this.
Re:Curl vs. Wget by Taniwha on Wednesday March 02, 2016 @08:26PM (#11214430)
Yeah I've been doing some libcurl development today .... Online docs have been unavailable all day ..
Re:Curl vs. Wget by awkward on Wednesday March 02, 2016 @09:27PM (#11214649)
Huh. I've always thought of them as the opposite - wget is the full featured spidering tool, curl is the easy to run one when I need a command line thing or to bang web stuff into a janky copy and paste workflow.
Re:Curl vs. Wget by Legogris on Thursday March 03, 2016 @04:05AM (#11215764)
This is interesting. Do you want to give your reasons or is it just an arbitrary habit?
Re:Curl vs. Wget by awkward on Thursday March 03, 2016 @09:06PM (#11221576)
wget leaves files everywhere, and isn't useful without picking a few flags to use, carefully. I use it as a tool to check builds for broken links

curl is a unix-way type program that interacts with standard out in a pretty predictable way.

Re:Curl vs. Wget by eru on Wednesday March 02, 2016 @05:58PM (#11213749)
Looks like a frontend to curl that has wget's defaults would be useful.
Re:Curl vs. Wget by theoh on Wednesday March 02, 2016 @06:19PM (#11213866)
Merge the code, hard link the filenames to the same executable and get it to change behavior based on the name it's invoked under.
Re:Curl vs. Wget by eru on Wednesday March 02, 2016 @07:10PM (#11214127)
No need to merge code. Take curl as is, and set different defaults based on executable name.
Re:Curl vs. Wget by lucb1e on Thursday March 03, 2016 @01:21AM (#11215338)
In that case I'd love for it to include resuming downloads though, but given the differences I don't think merging is going to be more useful than just adding the feature in curl.

Edit: nevermind, I just learned that wget can download a page's resources or even inline them (-k apparently), that's a bit of a way off from curl's purpose. Better keep them separate tools, although wget might benefit from using libcurl so they don't have to implement lots of stuff like https or http2.

Re:Curl vs. Wget by kcbanner on Wednesday March 02, 2016 @06:40PM (#11213994)
Sounds like a lot of complexity for no benefit.
Re:Curl vs. Wget by theoh on Wednesday March 02, 2016 @06:46PM (#11214016)
Either tool can provide the functionality of the other with a little work, is what I meant to imply...
Re:Curl vs. Wget by baudehlo on Wednesday March 02, 2016 @07:47PM (#11214289)
It's what vi does.
Re:Curl vs. Wget by baldfat on Thursday March 03, 2016 @08:28AM (#11216670)
I totally agree except over the years I have been using aria2 more and more instead of wget. aria2 supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink with the same sane wget syntax and defaults.

example

$ aria2c http://yourlink.com/file.*

-x2 allows using 2 connections per host.

https://aria2.github.io/

Re:Curl vs. Wget by voltagex_ on Thursday March 03, 2016 @10:39AM (#11217464)
I'm having to use aria2 more and more to maintain decent download speeds, occasionally even from Akamai and CloudFront. S3 seems to be the exception.
Re:Curl vs. Wget by brobinson on Wednesday March 02, 2016 @10:08PM (#11214778)
curl is the one where I have to remember whether to use -o or -O when trying to download a file with the original filename and just use wget instead because it's faster than reading the curl man page.
Re:Curl vs. Wget by paulmd on Thursday March 03, 2016 @10:00AM (#11217181)
I have the same problem. Curl is "unix-y" in the sense that the default options are optimized for a shell script and make no sense for interactive usage.

bropages is where it's at for that kind of stuff. curl is actually their usage example :)

http://bropages.org/

Re:Curl vs. Wget by jandrese on Friday March 04, 2016 @11:03AM (#11224597)
Their example will just dump the webpage to stdout! Almost certainly not the desired behavior given the comment. Then they include second example of a use case that almost nobody has, instead of giving the option that everyone is actually looking for.
Re:Curl vs. Wget by foota on Wednesday March 02, 2016 @11:24PM (#11215024)
wow what the heck libcurl supports a ridiculous number of protocols (https://curl.haxx.se/libcurl/)
Re:Curl vs. Wget by ucho on Thursday March 03, 2016 @04:12AM (#11215782)
> Wget's [..] defaults are more sane

Does it finally use filename from "Content-Disposition" without need for any switches?

Re:Curl vs. Wget by heinrich5991 on Thursday March 03, 2016 @05:33AM (#11216034)
That sounds like a security issue, allowing the attacker to name a file like they want.
Re:Curl vs. Wget by Kurtz79 on Thursday March 03, 2016 @04:52AM (#11215906)
"This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy."

Agreed, and even "emacs vs. vim" or "tabs vs. spaces" do not really need to be "emacs vs. vim" or "tabs vs. spaces"-type dichotomies.