News for hackers. Stuff that matters.
Re:Call for testing: OpenSSH 8.0 by segfaultbuserr on Friday March 29, 2019 @05:00AM (#19520100)
It includes an interesting comment on the scp vulnerability.

> This release contains mitigation for a weakness in the scp(1) tool and protocol (CVE-2019-6111) [...] The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.

I think it's the time to "alias scp=sftp". If the developers officially believe that scp should be retired, let's do the switch. Both are parts of OpenSSH and the commandline argument is almost identical.

Also, it has

> ssh(1), sshd(8): Add experimental quantum-computing resistant key exchange method, based on a combination of Streamlined NTRU Prime 4591^761 and X25519.

This is big. Together with XMSS signature, it means we already have a complete suite of post-quantum cryptography (experimentally) deployed in OpenSSH! It may be the first mass deployment of post-quantum cryptography in a major protocol.

One month ago, I commented that the introduction of XMSS post-quantum signature as "useless" (https://news.ycombinator.com/item?id=19160739), as the decryption of key exchange is much more vulnerable than spoofing the signature. But now NTRU+X25519 is deployed, great progress here!

Re:Call for testing: OpenSSH 8.0 by MayeulC on Friday March 29, 2019 @06:31AM (#19520422)
Frankly, I had not realized scp and sftp were two distinct protocols, despite my heavily using them. I thought sftp was an interactive client, while scp was meant for scripting, and never really bothered to dig further. I now will, thank you.

However, I expect plenty of people to be in the same case.

Re:Call for testing: OpenSSH 8.0 by chasil on Friday March 29, 2019 @12:56PM (#19523390)
I have actually been able to separate the external sftp-server from sshd, and connect it instead to stunnel. The sftp client binary can also be instructed to use something other than the ssh client to establish connectivity.

A simpler invocation could run it through (unencrypted) inetd/socket activation.

SELINUX really throws fits of hysteria when running this, however.

  $ cat sftp-tls.sh
  #!/bin/sh
  exec nc --ssl server 52345


  $ sftp -S ./sftp-tls.sh bogus
  Connected to bogus.
  sftp> cd /nobody
  sftp> get sftp-tls
  Fetching /nobody/sftp-tls to sftp-tls
  /nobody/sftp-tls       100%   15     8.9KB/s   00:00    
  sftp> quit


  # cat /etc/stunnel/sftp-ssl.conf 
  #GLOBAL####
 
  ;sslVersion =     TLSv1.2
  TIMEOUTidle =     6000
  renegotiation     =     no
           FIPS     =     no
        options     =     NO_SSLv2
        options     =     NO_SSLv3
        options     =     SINGLE_DH_USE
        options     =     SINGLE_ECDH_USE
        options     =     CIPHER_SERVER_PREFERENCE
         syslog     =     yes
          debug     =     debug
        setuid      =     nobody
        setgid      =     nobody
  #     chroot      =     /var/empty/stunnel
 
        libwrap     =     no
        service     =     sftp-ssl
  ; cd /var/empty; mkdir -p stunnel/etc; cd stunnel/etc;
  ; echo 'sftp-ssl: ALL EXCEPT localhost' >> hosts.deny;
  ; chcon -t stunnel_etc_t hosts.deny
 
  ; https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
 
 ciphers=ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:RSA+AESGCM:!aNULL:!MD5:!DSS

          curve     =     secp521r1
 
  #CREDENTIALS####
 
  #      verify     =     4
           cert     =     /etc/stunnel/sftp-ssl.pem
 
  #ROLE####
 
           exec     =     /usr/libexec/openssh/sftp-server
  #     execargs=   sftp-server -d /home/nobody
  #     execargs=   sftp-server -l DEBUG3
Re:Call for testing: OpenSSH 8.0 by kbenson on Friday March 29, 2019 @11:16AM (#19522394)
Don't forget some of your other options, such as piping across ssh commands, which is specifically very useful when piping tar output[1]. Piping tar commands traditionally was useful because it dealt better with some file types than scp did (devices, sockets, named pipes, tar is better overall at dealing with them, or was in the past). The other benefit is that after you realize you're just putting a sequence of commands into place to be run remotely, you start realizing you can do other things prior to the tar command if you need...

Rsync's ssh support is generally good, so might be about equivalent, but it's not always handy.

1: https://www.cyberciti.biz/faq/howto-use-tar-command-through-...

Re:Call for testing: OpenSSH 8.0 by pushpop on Friday March 29, 2019 @10:40AM (#19522063)
I’ve been using Linux and UNIX systems professionally for ~20 years and this is the first time I realised they were distinct protocols too.
Re:Call for testing: OpenSSH 8.0 by simias on Friday March 29, 2019 @06:59AM (#19520544)
I was definitely in the same boat. I guess I'll just progressively phase out my use of scp in favor of rsync.
Re:Call for testing: OpenSSH 8.0 by pletnes on Friday March 29, 2019 @08:56AM (#19521181)
Some servers, and some clients, only support one of them. I’ve run into servers without sftp and subsequently stopped using it, as scp always works.

Also rsync does not exist on windows, for instance git bash has only scp AFAIK.

Re:Call for testing: OpenSSH 8.0 by 0xfeba on Friday March 29, 2019 @09:47AM (#19521618)
A while ago I tried making windows' remote differential compression COM library work in a CLI utility like rsync.

https://docs.microsoft.com/en-us/previous-versions/windows/d...

Re:Call for testing: OpenSSH 8.0 by LinuxBender on Friday March 29, 2019 @10:17AM (#19521874)
You can install cygwin on windows. This will provide sftp, rsync and lftp which is awesome for doing rsync-like transfers on sftp-only chroot setups.
Re:Call for testing: OpenSSH 8.0 by djmdjm on Friday March 29, 2019 @07:10AM (#19520592)
actually, sftp is a bit more scriptable than scp. You can pass sftp a "batchfile" of commands and control per-command whether their success should terminate the batch.

scp wins on brevity of commandline syntax

Re:Call for testing: OpenSSH 8.0 by throwaway2048 on Friday March 29, 2019 @07:29AM (#19520678)
The script-ability of sftp is pretty awkward, it doesen't conform to how virtually every other tool works in terms of "command + args" in a shell scripting context, instead you need to pass it commands through a pipeline, don't love it.

Would be nice to be able to just type sftp user@domain:/file and have it do the right thing.

Re:Call for testing: OpenSSH 8.0 by MayeulC on Friday March 29, 2019 @10:11AM (#19521830)
Oh, that means you have to open a pipe to the process if you want to control it from another programming language? That would be a bit harder to leverage for automation (though I guess it's just a matter of writing a few helper functions, or you could always use bash and redirections if the language is ill-suited to pipes).
Re:Call for testing: OpenSSH 8.0 by LinuxBender on Friday March 29, 2019 @10:21AM (#19521916)
Add lftp on top of sftp and you get extensive scriptability. Working public anonymous demo: [1]

[1] - https://tinyvpn.org/sftp/#lftp

Re:Call for testing: OpenSSH 8.0 by bArray on Friday March 29, 2019 @05:54AM (#19520286)
Checking out the mitigation for CVE-2019-6111 [1], how does `sftp` help?

     * scp(1): Relating to the above changes to scp(1); the scp protocol
       relies on the remote shell for wildcard expansion, so there is no
       infallible way for the client's wildcard matching to perfectly
       reflect the server's. If there is a difference between client and
       server wildcard expansion, the client may refuse files from the
       server. For this reason, we have provided a new "-T" flag to scp
       that disables these client-side checks at the risk of
       reintroducing the attack described above.
You could just do a remote `ls` and then `sftp` all the files listed to your client - nothing stopping a malicious return from `ls`? Surely it's the risk of running any kind of wild card copy? If the server is compromised then there's no telling what will be returned from such a command?

    The scp protocol is outdated, inflexible and not readily fixed. We
    recommend the use of more modern protocols like sftp and rsync for
    file transfer instead.
The protocol itself might be outdated, but surely one of the best aspects of `scp` comes from its simplicity [2]?

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6111

[2] https://github.com/openssh/openssh-portable/blob/master/scp....

Re:Call for testing: OpenSSH 8.0 by boomlinde on Friday March 29, 2019 @07:37AM (#19520712)
IMO sftp is underspecified. The 14 IETF drafts that exist for it have all expired and never reached RFC status.

In practice, this means that the OpenSSH implementation is the de facto standard, and that interoperating a non-OpenSSH client with a non-OpenSSH server is a gamble.

It's interesting that they push for sftp. Personally, if scp is vulnerable to certain types of attacks, I'd rather see an entirely new standard and an effort to avoid having it in draft state for decades.

Re:Call for testing: OpenSSH 8.0 by throwaway2048 on Friday March 29, 2019 @08:06PM (#19526658)
sftp died in IETF draft because they kept tacking on outdated and misguided concepts from the original FTP like ASCII/binary transfer mode and record file types(really guys, when was the last time an operating system that used this was popular, the 1980s?), as well as tons of other completely pointless cruft to an otherwise very clean protocol, so OpenSSH decided to stay at draft version 3, and being as they are the dominant implementation by far, the effort died.

I'm not sure why we need to see an entirely new standard, sftp as implemented by openssh is well conceived, simple, and covers its use case very well.

Re:Call for testing: OpenSSH 8.0 by boomlinde on Monday April 01, 2019 @09:14AM (#19542884)
> I'm not sure why we need to see an entirely new standard, sftp as implemented by openssh is well conceived, simple, and covers its use case very well.

The key to my point here is "as implemented by openssh".

A relevant anecdote: I had to connect from my program to a third party (version 3) SFTP service, IIRC backed by some Oracle software (but don't quote me on that). I had a version 3 client library. The OpenSSH client worked just fine with the remote, and the client library worked just fine with an OpenSSH server. When connecting to the third party, the client gave up.

After some debugging I learned that the third party server didn't include the language tag in its response statuses (as SFTP version 3 software should). OpenSSH was just fine with this and ignored the problem, and the third party software was probably only ever tested with OpenSSH based clients.

It was definitely a bug in the third party service as far as I was concerned, but I think that this is the natural result of relying on behavior "as implemented by ssh" rather than a stable, well defined (and non-expired) standard.

Re:Call for testing: OpenSSH 8.0 by axaxs on Friday March 29, 2019 @11:57AM (#19522759)
I'm not sure SCP is specified at all, though. I recently wrote a true SCP client for Go (not SFTP, which oddly enough all the other "SCP" clients for Go use). The only documentation I could find at all was a single blog post from a guy who worked at Oracle.
Re:Call for testing: OpenSSH 8.0 by djmdjm on Friday March 29, 2019 @06:00AM (#19520312)
scp only appears to have simplicity because it outsources large parts of its functionality (e.g. glob expansion) to the destination host's shell. scp/rcp was a great protocol for 1981 (yes, that's when it was introduced) but not for 2019
Re:Call for testing: OpenSSH 8.0 by tgragnato on Friday March 29, 2019 @06:56AM (#19520530)
And this leads to the question.. OpenBSD is famous for removing unwanted or outdated code. What makes scp a special case? Is it too used to be deprecated and removed?
Re:Call for testing: OpenSSH 8.0 by djmdjm on Friday March 29, 2019 @07:04AM (#19520570)
yeah, pretty much. If someone implemented scp's command-line with sftp underneath then we could start the (slow) deprecation process.
Re:Call for testing: OpenSSH 8.0 by throwaway2048 on Friday March 29, 2019 @07:25AM (#19520661)
its part of the ssh protocol, getting rid of cruft from network protocols is almost completely impossible unfortunately, because its going to destroy many people's workflows, in a way that they can not fix, guaranteed.

Yes, we all know about xkcd, no need to link guys.

Re:Call for testing: OpenSSH 8.0 by djmdjm on Friday March 29, 2019 @08:24AM (#19520987)
scp isn't part of the ssh protocol. It's a command that runs over it.
Re:Call for testing: OpenSSH 8.0 by Operyl on Friday March 29, 2019 @10:22AM (#19521918)
Out of curiosity, since I wasn't sure which one in particular throwaway2048 was talking about, here are my closest guesses:

927 - Standards - https://xkcd.com/927/

1127 - Workflow - https://xkcd.com/1172/

1323 - Protocol - https://xkcd.com/1323/

Re:Call for testing: OpenSSH 8.0 by adrian_b on Friday March 29, 2019 @07:35AM (#19520703)
The only good file transfer solution has always been to use rsync over ssh instead of scp or sftp.

I have stopped using scp and sftp many years ago. Besides the fact that sftp can be too slow, both scp and sftp fail to make identical copies of the transferred files because sometimes they can lose some metadata, e.g. extended attributes or high resolution time stamps.

Re:Call for testing: OpenSSH 8.0 by adrian_b on Friday March 29, 2019 @07:40AM (#19520722)
What I am normally using is:

alias scp='rsync --archive --xattrs --acls --progress --rsh="ssh"'

Re:Call for testing: OpenSSH 8.0 by danudey on Friday March 29, 2019 @04:04PM (#19525121)
I've become a huge fan lately of using "--info=progress --no-inc-recursive" for rsync. It can result in more memory usage, since it builds the file list ahead of time rather than dynamically, but it provides you with an overall progress report of the entire transfer, rather than per-file progress reports which may be much less useful.
Re:Call for testing: OpenSSH 8.0 by seqizz on Monday April 01, 2019 @08:06AM (#19542475)
Aaaah I was so happy to switch until I see "The source and destination cannot both be remote."
Re:Call for testing: OpenSSH 8.0 by chrisweekly on Friday March 29, 2019 @08:36AM (#19521058)
This gets my vote for nicest alias in the thread.
Re:Call for testing: OpenSSH 8.0 by tyingq on Friday March 29, 2019 @08:01AM (#19520836)
Using ssh with tar on both ends is my go to for a new box that doesn't have rsync. Long command lines, but it retains metadata. Like:

  ssh remote 'cd /src;tar -cf - .' | (cd /dst;tar -xf -)
Doesn't call you out so well on typos though.
Re:Call for testing: OpenSSH 8.0 by justinclift on Friday March 29, 2019 @08:22AM (#19520977)
lftp can be handy too, as that can automatically parallelize transfers.

eg, instead of one connection (eg rsync), lftp can transfer each file using n parallel streams.

Very useful with large files, when there's some kind of bandwidth limit per TCP connection.

Re:Call for testing: OpenSSH 8.0 by pnutjam on Friday March 29, 2019 @12:48PM (#19523296)
Yeah, sftp can't see extended attributes. It's flags are also more cumbersome then scp. For example:

sftp -o IdentityFile=keyfile or scp -i keyfile

Re:Call for testing: OpenSSH 8.0 by pletnes on Friday March 29, 2019 @08:58AM (#19521188)
Too bad there’s no rsync for windows
Re:Call for testing: OpenSSH 8.0 by e12e on Saturday March 30, 2019 @04:33PM (#19531666)
Re:Call for testing: OpenSSH 8.0 by LinuxBender on Friday March 29, 2019 @10:19AM (#19521894)
Cygwin provides rsync, sftp and lftp for windows.
Re:Call for testing: OpenSSH 8.0 by user9182031 on Friday March 29, 2019 @11:11AM (#19522348)
Cygwin just feels "heavy" when all I want is rsync. Without any package management (that I'm aware of) I can't just install rsync and whatever other dependencies it has, I have to put hunderds? more binaries on my system.
Re:Call for testing: OpenSSH 8.0 by LinuxBender on Friday March 29, 2019 @11:14AM (#19522363)
There is also a linux subsystem for windows 10 which I have never used. Perhaps someone can say if that has rsync.

[ edit ] if that and cygwin are too heavy, you could request the samba team create a posix windows build of rsync. They had one long ago, but stopped supporting it. That would be a single binary. I suspect they will push back.

Re:Call for testing: OpenSSH 8.0 by user9182031 on Friday March 29, 2019 @12:36PM (#19523171)
This is even heavier. It's a full blown Ubuntu install.
Re:Call for testing: OpenSSH 8.0 by user9182031 on Friday March 29, 2019 @01:49PM (#19523931)
After a bit of Googling, it looks a company has complied and packged rsync for Windows at a cost of $19 per license @ https://www.itefix.net/cwrsync

I found a Github repo with an older (2017) version of cwRsync that may be from before it went behind a paywall @ https://github.com/billyc/cwrsync-installer

Re:Call for testing: OpenSSH 8.0 by pletnes on Friday March 29, 2019 @12:10PM (#19522894)
It does.
Re:Call for testing: OpenSSH 8.0 by pnutjam on Friday March 29, 2019 @12:51PM (#19523327)
Check out MobaXterm. It's free as in beer, and seems to wrap up cygwin and x in one console. It's my go to putty replacement.

You can run a local console and rsync directly from your window drives (/drives)

Re:Call for testing: OpenSSH 8.0 by ptman on Friday March 29, 2019 @09:49AM (#19521636)
WSL
Re:Call for testing: OpenSSH 8.0 by pnutjam on Friday March 29, 2019 @12:49PM (#19523307)
hmm... works fine in my mobaxterm window.
Re:Call for testing: OpenSSH 8.0 by djmdjm on Friday March 29, 2019 @05:17AM (#19520155)
Unfortunately, you can't alias scp=sftp as the commandline syntax is different. I'd love someone to write a scp replacement that attempted the sftp protocol first and optionally fell back to scp if that didn't work.
Re:Call for testing: OpenSSH 8.0 by low_key on Friday March 29, 2019 @08:47AM (#19521122)
This. I've been using rsync for a long time just for the resume feature even without knowing about some of the other benefits. I tend to use the flags "-avzpP"
Re:Call for testing: OpenSSH 8.0 by pnutjam on Friday March 29, 2019 @12:52PM (#19523341)
Not all ssh servers enable the sftp subsystem by default.
Re:Call for testing: OpenSSH 8.0 by jolmg on Friday March 29, 2019 @11:57AM (#19522765)
> I think it's the time to "alias scp=sftp". If the developers officially believe that scp should be retired, let's do the switch. Both are parts of OpenSSH and the commandline argument is almost identical.

I've never used sftp before. Does it offer equivalents to:

  scp server:'*.pdf' .
or

  scp server:'*(oc[1])' .
for getting the last created file with zsh extended globs, or

  scp server:'$(custom_remote_program)' .
where custom_remote_program is shell code that outputs a list of files using server-side state?

EDIT: Reading:

> * scp(1): Relating to the above changes to scp(1); the scp protocol relies on the remote shell for wildcard expansion, so there is no infallible way for the client's wildcard matching to perfectly reflect the server's. If there is a difference between client and server wildcard expansion, the client may refuse files from the server. For this reason, we have provided a new "-T" flag to scp that disables these client-side checks at the risk of reintroducing the attack described above.

I guess, I'll actually want:

  alias scp='scp -T'
or just use `-T` on a need by basis. I admit I don't need that feature every week, but it is a relief that they didn't remove the functionality outright. I think the closest equivalent is something like:

  ssh server 'tar c $(custom_remote_program)' | tar xC destination
which I'd rather not be writing.
Re:Call for testing: OpenSSH 8.0 by xorcist on Friday March 29, 2019 @07:15AM (#19520614)
> I think it's the time to "alias scp=sftp"

May I suggest using rsync instead? The syntax is much closer to scp than sftp is, to the point of being compatible for the most trivial use cases. But rsync is actively developed and has much saner defaults, can continue broken transfers etc.

Re:Call for testing: OpenSSH 8.0 by fanf2 on Friday March 29, 2019 @02:10PM (#19524089)
PuTTY's pscp has a command-line UI like scp, but (by default) it is implemented using the sftp protocol. It's a pity OpenSSH scp doesn't work the same way.
Re:Call for testing: OpenSSH 8.0 by 0XAFFE on Friday March 29, 2019 @06:30AM (#19520421)
This works pretty well:

alias scp="rsync --partial --progress --rsh=ssh"

You should keep in mind that it is rsync in the background, because parameter are different.

Re:Call for testing: OpenSSH 8.0 by Arkanosis on Friday March 29, 2019 @11:08AM (#19522334)
I'd recommend to use `--partial-dir=.rsync-partial` instead of `--partial`, as the latter does not prevent an incomplete transfer from looking like a complete one.