It'll break a lot of valid functionality that relies on it, and the only case I can see presented as to why is for situations when servers choose to offer scp when they should have chosen to offer sftp.
Complaining that some people hand out shell access and this creates a "problem" that they gave people access to shell commands verges on ridiculous.
That sounds like it will probably be accepted after a few more minor fixes.
What about this part of the article:
Finally, while the danger is remote, it is worth noting that a local file name containing `backticks` (a file named `touch you-lose`, for example) will be handled the same way on the other end; if a user can be convinced to perform a recursive copy of a directory tree containing a file with a malicious name, bad things can happen.
I much prefer the status quo. Don't agree that this change is necessary.
Is that not correct?
And you don't deprecate a 25-year old utility with just a replacement "idea".
https://ajaxnwnk.blogspot.com/2020/10/on-abandoning-x-server... https://news.ycombinator.com/item?id=24920183 https://www.phoronix.com/scan.php?page=news_item&px=XServer-... https://news.ycombinator.com/item?id=24884988
First it didn't support the authentication method I wanted to use, and then it didn't have any port forwarding ability. I'm betting it doesn't have sftp support either.
ssh cat /foo/bar > /bar/baz
ssh tar c /foo | tar x -C /bar
mostly because variants such as vagrant ssh or docker-machine ssh, cross-product with the remote target being any weird platform like solaris or a limited busybox or has sftp disabledIn a previous team where we dealt with shipping around lots of ZFS snapshots (where both the performance of reading from disk can vary based on the ARC and filesystem metadata) and occasionally using VPN tunnels on public internet links, pv to get even larger pipe buffers than standard was often a huge performance equalizer. Readers read as fast as they can, writers write as fast as they can, and the buffer massages any variability in the middle.
(Those extra mechanisms are usually beneficial in WAN scenarios as well, except when you have high speed links, in which case blasting the full files may be faster than doing the IO to figure out comparisons and skip transfer.)
Edited: adjusted confused punctuation.
dd if=filename | ssh hostname dd of=remote_filename
Quite a bitSomehow this worked:
dd if=foo | adb shell dd of=/storage/.../fooA example the C standard could stand to learn from, then.
cat file | ssh user@host 'cat >file'
ssh user@host cat file | cat >file
?PS.
I use mc to copy: F5, Enter.
Maybe it’s all the bottom up thinking we do.
> You: understand the article better!
(But not deleted for now)
A new command with similar cli but different name is in process but it will not (and can't) support all features of scp so it can't be a drop in replacement and as such you can't name it scp as this would brake existing systems.
Eh.
Seems like a half-assed solution for a non-problem, that will simply break stuff, remove functionality, and increase attack space for no benefit. If you need sftp, use sftp.
0: that is specific to scp rather than ssh in general, and where the client (rather than some third party) is the attacker
Unfortunately, it took only one more time for it to be forgotten, then another few times to be re-learned, then forgotten again, then half-remembered the wrong way, then....
I pretty much have it down now, though. The secret to my success: remembering that the differing behavior applies to the source side only. Trying to work out the matrix of possibilities between source, source/, dest, and dest/ is where I usually got lost, until I finally got it through my head that the destination syntax is irrelevant.
Now it makes sense: "foo/bar/" is the directory named "bar"; "foo/bar" is the directory entry within foo/ named "bar". Copying a directory puts its contents on the destination side. Copying a named thing, whether the name refers to a file or directory, puts that name on the destination side.
rather than consult the manpage yet again. When I want those semantics, I tack on the self-referential dot. `rsync foo/bar/. baz`
What is faster depends a lot on what you are trying to do. Rsync can be several times slower at times.
Sometimes this optimization isn't the exact optimization you want. :-)
I know that you could have an SSH implementation without sftp - but I'm arguing about the regular case. AFAIK sftp is usually there.
Why would and admin enable full shell access but not sftp? What does sftp allow that shell access doesn't?
Pure shell access clearly is a superset of all functionality - for example, given shell access, a poor man's scp would be
ssh user@remote "cat /path/to/target/file" < local file
and if you need to do more files at once, add `tar` to the mix.How do you know it's less popular? It seems like every non embedded distro enables it. OpenSSH linked sftp-server into sshd because SFTP only configurations were popular.
So you can use ansible to install a common and secure and standardized sshd config across, say, legacy linux and new freebsd servers.
With the tiny little problem that filesystem paths will be different for linux and freebsd.
On modern freebsd, from memory, this sshd config line works:
Subsystem sftp /usr/libexec/sftp-server
I forget the path on legacy linux.
Anyway the simplest and most secure way is just to disable sftp across all systems and just use ssh and scp.
The solution to this problem is likely not to be shipping multiple inevitably slightly incompatible sshd configs, but I'll probably end up shipping exactly and precisely one sshd_config with the path on all operating systems being /usr/local/libexec/sftp-server or something like that, and then ansible rules to symlink freebsd and linux to the "standard" path.
Or I'll go all in on freebsd and get rid of the last legacy linux servers.
Or I'll reduce security by using ansible with multiple sshd_config files.
Or recode everything that used scp to use rsync or other alternatives (LOL as if thats happening)
At any rate as usual with unix there's multiple ways to handle things, which is good. Its not like windows or systemd based operating systems.
I've never done so, and it's worked fine on every server I've ever built. It's at least enabled by default, I guess.
The point is moot though because disabling the sftp subsystem yet granting people shell access is backwards.
Also no, you don't need an scp on the remote side.
I use tinyssh in a few contexts where I don't want file transfers of any kind.
___
TinySSH doesn’t have SCP?
No, ‘rsync -e ssh’ makes same job. If you really need scp, use scp for example from OpenSSH. TinySSH doesn’t have problem with scp protocol, only doesn’t have scp program.
Can I use sftp using TinySSH?
Yes. TinySSH doesn’t have sftp program, but can run e.g. OpenSSH /usr/libexec/openssh/sftp-server. Sftp support can be enabled using switch ‘-x’
... tinysshd -x sftp=/usr/libexec/openssh/sftp-server /etc/tinyssh/sshkeydir
https://tinyssh.org/faq.htmlIf you do "scp foo user@host:/bar", then scp ssh's into user@host and executes scp(1) there with a flag that tells it to go into sink mode. When you swap the arguments, it uses a flag that tells it to send the files you want in source mode.
I do not use sftp because it is a bad protocol, which cannot reach high speeds. This has been explained by other poster.
I frequently transfer tens or hundreds of gigabytes of files over ssh and with sftp that would be unacceptably slow. Also, sftp does not handle correctly all file metadata, in certain cases making incomplete file copies, with only partial metadata.
I use rsync over ssh, which does not have any of the problems that plague scp and sftp.
I deactivated sshd multiple times for some clients (needlessly I should add since the VM only network access was through VNC, and sometimes nfs wo) to avoid data leaks. When some data (often code) had to be sent to the VM, I don't see one case when reactivating ssh and not sftp is better than the reverse. Activating sftp only allowed us to keep track of the file put on the server, effectively copying everything passing through the chroot directory.
Do you have any examples?
To be clear, the problem is not that sftp is unlikely to be available, but that it's possible that it will be the case; hence it does not replace scp in that scenario without complete loss of functionality. It is an objectively inferior option.
(Yes, seriously.) A creaky old system if I've ever seen one. Of course, I'd enjoy hearing anyone else who has experience with worse.
Another revelation for me from the article - ifconfig is deprecated? I did not get that memo.
Of course the problem with this approach is that you won't receive security updates - but if the developers already consider the tool insecure, it might not matter much, especially if you use it only between hosts you control.
For example, in any decent Linux distro you’d want to replace the package, not a single executable. So at a minimum it would be openssh-clients, which is a pretty important package to potentially get wrong.
That being said, scp-the-protocol is actually very simple. There is no spec for it, but a number of interoperable implementations and the protocol is really damn simple (it's basically goes "file <length of name> <name> <size> write <length> <data> write <length> <data>" and so on). It achieves good throughput (for large files) over SSH, but because every file involves a few ping pongs, it is RTT-bound for small files.
SFTP is much, much more complicated. And the spec situation is much worse, because there are like a dozen drafts and half a dozen different versions of the protocol. SFTP also pulls in half of the POSIX file semantics. SFTP naively is RTT bound for throughput; read size is limited to 64K in OpenSSH, so with 20 ms RTT you're only going to get at most ~3 MB/s with a naive client.
SFTP is essentially NFS, but over single SSH channel (and different). You get to ask for file handles, and then you can do requests on those handles. You get to opendir() remotely and get a directory handle and so on.
Like NFS, SFTP supports having multiple requests in flight (how many: implementation defined / no way to find out), so you can request multiple reads and wait for them to get around the 64K limitation. Problem: maximum read size is implementation-defined / no way to find out, which makes this really quite complex, since you have to account for reads coming back out of order and for reads being shorted than you requested without having reached EOF. Say you want to transfer a 500K file in 256K chunks, you schedule two reads of 256K and 500K-256K = 244K. Let's call them r0 and r1. Now r1 comes back, but it only read 64K (or 8K or 16K or whatever the implementation felt like). Now you need to figure out that (1) you should hold this data back, because the data before the offset of r1 has not been read yet (2) you need to issue another read to get the contents from 320-500K where (3) you may figure out that the implementation probably only does 64K reads (note: SFTP read request length field is 32 bit... expectations and all), so you get smart and schedule a few more reads: r2 for 320-384, r3 for 384-448 and r4 for 448-500K. Now you wait for the responses and get, e.g. r3, r4, r1, r2. You need to hold all this data and shuffle it around correctly, then write it in-order to the file (assuming you want to write the file sequentially, which is very reasonable if you want to have any chance at all of resuming the transfer).
This is on top of SSH already having throughput issues in the basic protocol over long fat networks.
Madness.
What scenarios are you talking about where chunks are important and you have to be concerned about ordering? Is this strictly for applications that perform large sync'ing jobs where "to-the-limit" performance is important?
It doesn't seem like a huge deal to deprecate scp and start using a short stanza of sftp for simple file transfers.
Is that why I've sometimes observed slower-than-expected transfers when using rsync over ssh to do a mass migration of server data from one data center to another? Can you recommend an alternative (besides writing the data to external media and physically shipping it)?
Also, make sure TCP window scaling is working. I was making transfers through a F5 Big-IP which was running a profile that disabled it.
If SCP/SFTP is the problem due to small-ish files, use a tarpipe instead. Nothing beats tarpipes for small files.
A replacement with a very similar interface is in process of being made. But it will not have all features scp had.
You can't fix the protocol because people rely on it's exact behavior including the parts which have security problems. E.g. people rely on backtick expansion to run a command in the ssh session before the copy. But this also can lead to injection vulnerabilities in management scripts.
Can you explain what you mean by this? I use rsync with the 'same syntax'; instead of 'scp src user@remote:/dst' I can do 'rsync src user@remote:/dst'
scp (and cp, for that matter) don't take this difference into account. That leads to gotchas with recursive (-r) copying. Most importantly, scp isn't idemopotent:
scp -r fromdir todir
If todir doesn't exist, scp will copy the contents of directory fromdir to a new directory named todir.
Execute the same command again (now that todir exists), and scp will copy fromdir to todir/fromdir .
On the other hand:
rsync -a fromdir/ todir
will always copy the contents of fromdir into a directory named todir (effectively, a directory rename operation), whether todir exists or not.
rsync -a fromdir todir
will always copy the directory fromdir into the directory todir, whether todir exists or not.
These rsync operations are idempotent, which it important because rsync is designed to incrementally re-sync directories. It is expected that it will commonly be run more than once, which is why it needed to address this IMHO fundamental bug/limitation in cp and scp.
Scp and cp have the same semantics, so cp is vulberable to this gotcha as well. Rsync doesn't have to be used with remote sources or targets (nor does scp).
This is in fact why use rsync for local copies as well, it's much more consistent.
For example, Windows symlinks and Cygwin symlinks are quite different. I don't think Cygwin rsync can replicate a directory containing Windows symlinks properly, so that the replica behaves the same as seen by Windows programs.
I'm not sure about the other Windows attributes such as ACLs but I would be surprised if Cygwin rsync can replicate those.
That's different from MacOS, where there's a Mac-native version of rsync to transfer attributes that don't exist on $another_os.
apple decided to include those tools by default possibly in order to attract users. then they decided to stop updating and now macos is no better than those other unix systems used to be.
i used a mac for a few years and i don't miss the experience. gnu/linux just works
It also has a much better protocol design.
For your daily use it is almost syntax compatible too.
I my opinion, the default behavior of any file copy program must be to make exact copies of the sources.
I find it very annoying that all UNIX copying programs do not have this behavior and by default they will lose information.
Therefore I always use aliases for all copying commands (cp, scp, rsync etc.), so that by default they will make exact copies.
For example, to make exact copies rsync needs "--archive --xattrs --acls", and cp needs "--no-dereference --recursive --preserve=all". cp also needs to be compiled with enabled extended attributes, which many Linux distributions disable, otherwise you lose the extended attributes without any warning or error.
Another trap on Linux, which may prevent making exact file copies, is when tmpfs is used for /tmp and some file is copied through /tmp, e.g. for passing it to another user. A copy through tmpfs may lose extended attributes and it also may truncate the timestamps of some file systems.
scp file.bin server:
rsync file.bin server:
The obvious difference is that rsync defaults to a more unixy no-news-is-good-news output, it will only output errors. To show interactive progress use rsync -vP file.bin server:
Use "-e" to set ssh options, just like with scp. Use "-r" to transfer whole directories, just like with scp. But you're likely to use "-a" with rsync instead which is recursive preserving all attributes and timestamps.Another difference is that rsync will resume transfers, which scp can't. It can do sparse transfers reasonably well. It also has more sane security defaults And, as implied by the name, update a remote directory including deletions.
Check out "man rsync" for the full story. It's not long after your shell and editor in daily usefulness.
On my rsync -e sets the executable, and my scp doesn't have -e but -o. Even then it's a bit of a stretch (or I'm missing something); e.g. for the ssh option I use mst of the time, a non-default port:
scp -P 4222
is equivalent to scp -o"Port=4222"
is equivalent to rsync -e "ssh -p 4222"
Again, this is just what I use after having to look it up at one point so I might be missing something, but it's not like the arguments translate one to one?except the 'almost' can blow away filesystems if not used correctly (trailing slash vs no trailing slash being context dependent on presence/absence of remote files).. no idea why rync used differnt path semantics - cpdup for example uses identical semantics for a similar (but less performant) utility
[1] - https://serverfault.com/questions/836103/replace-scp-with-sf...
And while I use scp rather often, mostly for single file transfers, I still dislike that cp -a is scp -p instead. But my finger memory learned that a long time ago.
As soon as there's more to transfer, I switch to rsync, not the least because of the possibility to optionally resume partial transfers.
ssh uses '-p' to use a non-default port but scp uses '-P' (-p preserves mtime, as you pointed out). scp requires ipv6 to be in brackets (in order to parse the : correctly) but ssh requires it to be without brackets. Makes converting one command line to an other more annoying than it should be.
-p for --port does seem like it's needed more often than -p for --preserve-mtime, but running ssh on non-standard ports was less of a thing when scp was designed.
ssh -p 2222 root@fd95:74a4:b46c:1000::4219
scp -P 2222 root@[fd95:74a4:b46c:1000::4219]:/tmp/foo .
It generates a lot of editing when converting an ssh command into an scp one, or vice-versa. It's minor, but I deal with raw IPv6 hosts a lot for various reasons and it's a daily annoyance.Anyway, it's pure bikeshedding at this point, it's just that it's such a common annoyance for me that I can't help but nitpick.
Yeah. It's one of the 13 (of 52) unused one-letter options, but such a common one you'd think they could waste a second letter on it.
Computer security evolves over time, and we have to cope with APIs and workflows which change to keep pace. Nothing supports SSLv1 anymore. Telnet is (largely) a thing of the past. Yes these migrations can be painful, and place some burden on users to change their practices. I get that there are a large number of scripts out there which may need to be adapted to work in this new world - but how is that different to any API change? Major breaking changes like this are rare, because the cost to users is high, but even so I don't recall seeing this insistence that things must _never_ change in, for example, glibc.
This is very similar to GUI revamps. Users always hate them, even if they're super well-executed.
People just don't like change, and asking them not to grumble works about as well as asking them not to breathe.
It was not my intention to appear to be making demands, I was genuinely asking why deprecate the tool without a replacement. It seems there was a misunderstanding on my part, it is in fact the protocol being deprecated, contrary to how the majority of the article makes the point appear.
As a software engineer, I too have users, and deprecating a simple, functional and easy to use tool is not something I would do without expecting a response, or for them to ask questions about how reasonable that is, whether they pay me or not. Expecting them to use a more inconvenient solution is something we should avoid as software engineers, and instead aim to make their life easier, whether they pay us or not.
I have no issues with switching tools. I have embraced systemd, iproute2, ss and others. Deprecating scp as a tool/syntax, to me, seems more fundamental. I'd expect no less of a response than if we were to be told that "cp" itself was being deprecated in favour of something which may or may not offer a similar syntax, may require more thought to use and may or may not be available under certain conditions.
Also musescore: Gets the guy who roasted them on board as Head of UX.
Musescore, but written in Gtk: "That composer just doesn't understand how things are done properly in Musescore. Also, he should shut up and be grateful that it's free."
rsync is not a replacement, sftp is not a replacement. If I can't use cp like syntax, it's not a replacement.
I use scp almost daily to copy things to/from/between remote machines, the syntax is quick and easy.
If the issue is with the protocol, why is the protocol not being fixed or updated while continuing to support the syntax?