I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management":
This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment, when a decision about something like "how do we get a reliable build of X for our production server deployment system" needs to be made or a system needs to be supported, no idea is "bad", because no one has the experience or confidence to be able to say "that's a stupid idea, we are not making `curl http://bit.ly/foo | sudo bash` the first line of a deployment script"[1]
[1] yes this is an exaggeration, but there are some simply shocking things happening in real environments, that are not far off that mark.
Edit: to make it absolutely clear about what I was referring to with [1]:
The specific point I was making was running something they don't even see (how many people would actually look at the script before piping it to bash/sh ?) from a non-encrypted source, and relying on a redirection service that could remove/change your short url at any time.
Unfortunately I was stupid enough to ddg it (duckduckgo it, as opposed to google it) and apparently this exact use-case was previously the recommended way of installing RVM[2]
[2] http://stackoverflow.com/questions/5421800/rvm-system-wide-i...
-
Like, most of the way deployment should work is that you come up with some collection of packages that need to be installed and you iterate through and install them. Bob's your uncle.
Thing is, all the packages you need live out in the wild internet. Ideally, you'd just be able to take a package, vet it, and put it in your local artifact store and then when your production deployment system (using apt or yum or pip or gems or maven or whatever) needs a package, it looks at your local artifact store and grabs it and goes about its business. Never knowing or touching the outside world.
And your developers would all write their apps to deploy through the normal packaging methods that everyone and their mother is already familiar with and they could just put them into the existing package index as well.
But you've gotta lay out pretty serious moola (from when I last looked into available solutions to this) or set up a half dozen different artifact stores if you want to do things that way. And good luck managing your cached and private artifacts if you do. And on top of that developers don't necessarily know how to set up a PyPi or a RPM index or whatever so that the storage is reliable and you've got the right security settings or whatever else. (I know I sure don't and I'm not really interested in reading all of the ones I'd end up needing).
Setting up RPM is shockingly easy. It can get more complex, but the basic system is:
REPOBASE=/srv/www/htdocs/
createrepo -v $REPOBASE
gpg -a --detach-sign --default-key "Sign Repo" $REPOBASE/repodata/repomd.xml
gpg -a --export "Sign Repo" > $REPOBASE/repodata/repomd.xml.key
Then you get to decide on the deployment machine if you want to trust the repo (or you don't trust any and import the key via some other process, aka direct gpg import).
Of course you can find a bunch of more detailed explanations with $SEARCHENGINE, but if it takes more than a day to figure it out, your doing something wrong.
Building a set of RPM's isn't that much harder if you have a proper build system. But these are the kinds of things you give up when you decide to grab the latest immature hotness created by someone on their day off.
Not only "previously", it's the current recommended way to install rvm. From their front page:
>> curl -sSL https://get.rvm.io | bash -s stable
[1] http://rvm.io/
The whole security, whether GPG is invoked or not, relies on the security of the HTTPS connection alone.
If the HTTPS cannot be trusted alone, then everything is lost as a compromised HTTPS connection can be used supply both a comprimised GPG key and a compromised package, or, indeed, anything at all that is legal to `| sudo bash`...
And HTTPS security boils down to:
1. The difficulty of altering (or exploiting privileged position wrt) the global routing table to setup MitM or MitS scenarios.
2. The difficulty of obtaining a valid looking certificate for an arbitrary domain.
Any situation where a government actor is the adversary poses intractable challenges to both 1 and 2 above. (And before you say NSA/GCHQ would never care about XYZ, consider China...
So even if you trust https works, it's still a tricky proposition - it's not really similar to a distro's package distribution channel.
[1]: https://github.com/discourse/discourse/blob/master/docs/INST...
Are you really trying to say that the instructions for installing Docker should be considered in-scope for a guide to install Discourse on a cloud server?
They've included a short snippet that will get you a Docker, in the way recommended by Docker, for whatever your base system is. Many production systems do not move at the pace of Docker development, so it's not practical to run Docker from your distribution's package archive. Some distros will not have distributed packaged Docker releases at all.
What's wrong with these instructions? If you are really "dealing with this" right now, it is worth noting that something like 20 or more supported platforms have specific Docker installation instructions from the Docker website.
https://docs.docker.com/installation/#installation
From a quick sample of those instructions, only the Ubuntu instruction page uses the wget|sh method, and it's using an SSL connection to Docker's own website to add an apt source with signatures in the supported way. This way should work on any Debian-based or Yum-based distro, and writing the instructions like this must likely save Discourse from getting a lot of "How do I docker" issues and e-mails from their clueless users.
So, would you prefer that part just says "installing Docker is out of scope" or should the Discourse developers go through every distro and cloud system and document the specific instructions for that? To do that would completely defeat the purpose of even using Docker at all.
Quick directions like these aren't questioned by users who just want to get things done, and they invite security risks just as the parent & article suggest.
I understand that curl pipe sh could have security problems but I also don't see it as that much different than the "normal" and "ok" way of doing things. I would consider something like the below pretty normal.
wget https://whatever.io/latest.tgz
tar xzf latest.tgz
cd whatever-stable
./configure && make
sudo make install
What is so different about
curl https://whatever.io/installer.sh | sudo bash
I agree that curl sh | sudo bash makes my spidey sense tingle. But if I really cared, I would read the source and do all the normal stuff anyway. So I think it's some kind of weird familiarity phase we're all in.
With a good build system (eg. autotools) writing an RPM spec takes almost no time at all and if you have the proper infrastructure in place for building packages, you can have something workable in a very short time.
Self-packaged RPMs also don't need to be quite as high-quality as ones you might want to include in a distribution, so if it makes sense for your use case, it's perfectly okay have "bloat" (eg. an entire python virtualenv) in your package.
Yikes. Have we sunk this far?
A distro (or official vendor, or possibly a trusted third-party) repo of pre-built, signed packages would always be my first choice.
If one of those isn't available, my next step would be to create a package for the tool in question, part of which is setting up a file for `uscan` to download new source archives, and compare against the signatures.
In this scenario we (as in the organisation) are now responsible for actually building and maintaining the package, but we can still be assured that it's built from the original sources, we can still install it on production (and even dev, staging, whatever) servers with a simple call to apt/aptitude, and dependencies, removal, upgrades, etc are still handled cleanly.
I find it interesting that binary packages have existed for decades and yet `rpm etc` knowledge is rare. Why did curl sh become popular? Why doesn't every project have rpm|deb download links for every distro version? Why don't github projects have binary auto-builds hosted by github? I'd argue that it's too difficult. Binary packaging didn't succeed universally. For deployment, containers are (in the end) easier.
But the original article is conflating container concepts and user behavior (not wrongly). If docker hub does end up hosting malware-laden images, it would be interesting emergent behavior but it would be orthogonal to containers. Like toolbars. Toolbars probably aren't evil. A vector for evil maybe?
What makes you think the knowledge is rare? Among developers who actively target linux distributions I would imagine the opposite is true.
Even a number of the referenced curl|bash offenders are just using that as a "shortcut" to add their own apt/yum repos and calling apt-get/yum to install their binary package(s).
* Using checkinstall to create a local deb/rpm which can be easily installed/removed later instead of "make install". * What if installer.sh says "rm -rf /tmp/PACKAGE-build" and the connection is interrupted just after the first "/", you now have "rm -rf /". Oops. * configure will tell you what files it needs, and apt-file will tell you want dependencies to install. * I know what make install does. I know make. Who wrote installer.sh? Do they know anything about writing good software? Steam wiped out home directories, who knows what these people do
# remove the old version of our data
sudo rm -rf /usr/local/share/some_data_folder
Or, you know, maybe someone updated the whatever.io installer to make it 'better'. But you are trying to debug some problem and you made one image last month and another one this month and you're pulling your hair out trying to figure out why they are different. Oh, it's because some text changed on some web site somewhere.
You've taken a mandatory step and put it outside your sphere of control.
I don't think the problem is as rampant as it's made out to be in TFA... that said, most people don't look at said script(s), so it's entirely possible something could have been slipped in. For that matter, I think the issues outlined in the article relate more to overly complicated Java solutions (the same happens in the .Net space) that are the result of throwing dozens of developers some with more or less experience than others at a project, and letting a lot of code that isn't very well integrated slide through whatever review process does or doesn't exist.
No, it's not :(
It's really really bad, but people will continue doing it until commands/things become so easy we can actually understand what we're doing. Unfortunately, this has never been a priority in Unix-land as far as I've gathered.
But it isn't all that hard to understand a clean Unix. I have never copied or typed a command that I don't understand.
One problem may be that most Unices these days is not as clean anymore as, say OpenBSD or NetBSD. E.g. the recent X stack, with D-BUS, various *Kits, etc. is quite opaque. This madness was primarily contained to the desktop and proprietary Unices, but seems to spread through server Linuxes these days as well (and no, this is not an anti-systemd rant).
Well, good for you. I can assure you that it's not the case for almost anyone who approached Linux after the likes of Mandrake were released and/or tried to make it work on anything different from a traditional server.
I'm all for trying to understand what one is doing (and I wholeheartedly agree with TFA's point), but the reality is that very few people in the world really understand all intricacies of one's operating system. This does not excuse poor security practices, but it explains their background.
You wouldn't hire some high school kid who's just about taught themselves HTML by reading a book for a week, and get them to write your web application from ground up. You'd hire someone who knows what they're doing. Why is it seen as any different for Operations work? There is a reason systems administration is a skilled field, and a reason they're paid on a par with developers.
However yes the issue of a team that "doesn't make money" is very real. Maybe you it should be "marketed" like legal or accounting: it doesn't make money, it saves money caused by SNAFUBAR situations.
1. I saw it done that way in some blog.
2. We did it like that at my last job.
3. Seems like it works.
/(apt-get|yum|dnf) install (apache2|httpd|nginx|lighttpd)/
One of the problems (as I tried to argue) is that most Unices have become far more complex. The question is if the extra complexity is warranted on a server system, especially if bare Unix (OpenBSD serves as a good example here) was not that hard to understand.
Of course, that doesn't necessarily mean that we should look back. Another possibility would be to deploy services as unikernels (see Mirage OS) that use a small, thin, well-understood library layer on top of e.g. Xen, so that there isn't really an exploitable operating system underneath.
This because they want to retain their ability to shop for off the shelf hardware, while getting away from a platform that has proves less than functional for mission critical operations (never mind being locked to a single vendor).
What seems to be happening is that there is a growing disdain for power users and "admins". The only two classes that seems to count are developers and users, and the latter needs to be protected from themselves for their own good (and developer sanity).
To note that it's trivial to change what goes into the clipboard too. Copying and pasting commands from potentially untrustworthy sites should be ruled out too, even if understood
x = eXtract files from an archive
f = File path to the archive
c = Create a new archive from files
v = print Verbose output
z = apply gZip the input or output
j = apply bzip2 to the input or output
(I admit, j is a weird one here, though that has made it stick in my memory)
--list = does what's on the tin
--exclude = does what's on the tin
--strip-components = shortcut for dropping a leading directory from the extracted
Part of the problem is that each command line utility has its own flag language, and equivalent functions often have different letters. For instance, very often one command has "recursive" as "-r" while another has it as "-R". It's impossible to remember it all unless you're a sysadmin.
Except with cp , -R is the safe one and -r is the dangerous one. And there are tons of little inconsistencies like this.
As a former sys admin, I did that all the time. Who the hell can remember how to convert an SSL certificate to load it into a Glassfish app server? Didn't mean I couldn't step through all commands and figure out why it did that before I loaded the new cert... And next time, I just need to go to my quick hack repo for the magic incantation.
On a Unix based system, tar is just used so frequently and for so many purposes, that not understanding it feels a bit like working in a shop and not knowing how to use a roll of tape.
(I'm a developer.)
Sequences of commands sometimes get pasted into a conveniently-located text file; if I find myself repeating the operation I might turn it into a script, a shell function for my .zshrc, or an alias.
Just 10 minutes ago: mysqldump [args] | nc -v w.x.y.z 1234 nc -v -l 1234 | pv | mysql [args] (after an initial test that showed adding "gzip -1" was slower than uncompressed gigabit ethernet.)
tar xf ./foo #automagically works with bz2 and gz files
tar cf /tmp/out.tar . #add z for compression tar cf <mydir.tar> <mydir>
tar c . | gzip > /tmp/out.tar.gz
tar -xvvzf foo.tar.gz
tar -xvvjf bar.tar.bz2
tar -xvvf baz.tar
"regexsearch" is more work to type and more space taken up everytime 'grep' appears in a command-line. And says nothing about recursion.
Edit: the rest of my comment (somehow submitted to soon!)
man grep
/recurs<enter> $ man grep | grep recursive
directory, recursively, following symbolic links only if they
Exclude directories matching the pattern DIR from recursive
-r, --recursive
Read all files under each directory, recursively, following
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all1. I routinely need to look things up that are a bit murky in the deep recesses of my memory.
2. I am reminded continually of how nice it is to have man pages that are well written, are easily searchable, reference appropriate other pages, and are helpful enough to remind you of big picture considerations that you didn't realize you were facing when looking for a commandline flag.
Google query: git display file at revision. Immediate answer (without even having to click any links, it's in the result description): `git show revision:file`
Total time: 5 seconds
Trying to reproduce with man and help:
man git
start scrolling down
notice git-show (show various types of objects); sounds like a likely candidate
git show <revision> <file>
git show -h
usage: git log [<options>] [<since>..<until>] [[--] <path>...]
or: git show [options] <object>...
man git show
man git-show
man git-rev-parse
Give up. Google it.
When I google something, I usually do not remember the answer to my question, the only thing I remember is the keyword to put in my futur query to get the same answer. You will get your answer quicker, but you wont learn much. So personally, I prefer reading man pages (when I can) than use google.
Rsync for example, where trailing slashes make a difference and it's not obvious from skipping over the manual.
Looking at working code/commands often works better than piecing it together from the manual imo.
(I can imagine some sort of hackery that determines if less or something is running and scrolls that, but it sounds like a huge mess. Is that actually what you're doing? Does it send keypresses? What if you're in a mode where those keypresses do something besides scrolling?)
I'll be honest - I have ~no idea~ (edit: apparently there are xterm control sequences for mouse scrolling) how it's actually implemented, but several tools have some reference to mouse support (tmux, vim, etc) in option/config files, so it's probably available for your distro/platform and just needs to be enabled.
Further edit: (or PS. or whatever):
`less` pager supports mouse scrolling. `more` pager does not!
When you read the script in a browser, than pastes it in a terminal, you know that "scp -r ~/.ssh [email protected]" isn' there.
For example, it won't protect against stealing the .ssh folder and installing a keylogger at your computer.
I'm totally stealing this :)
I see the issue with doing it for the general public ala RVM, but internally where you control everything I don't see the issue with curl into sh.
While I agree with the articles main points - the GNU build system is far from simple. Basically an arcane syntax limited to unix-based systems and 5 or 6 100+ page manuals to cover.
It doesn't excuse it - but I think it's easy to see why people turn to curl | sudo bash as the author puts it.
It doesn't do dependency management though, which is an externalised cost. But that's what rpm/deb do.
I see the attraction of containers and disk image based management. It's much less time consuming. But it's very much the opposite of ISO9001-style input traceability.
"Usually" indeed. Because if it breaks, you do need to know the implementation details to figure out what's wrong.
The problem is installing 206 different pythons on my system just makes it more likely that something else is going to break.
ISO9001 often turns into its Dilbert parody of bureaucracy, but the core ideas are sound: if you have some sort of failure of production, it's useful to know what went into the production process and where it came from. So in the case of deploying images, then yes: you get repeatable copies of the image. Provided you know where the image came from. Images themselves aren't usually stored in a version control or configuration management system. It may not be obvious where the image came from. And, if an image is made up of numerous "parts" (ie all the installed software), you need to know what those parts are. If an SSL vulnerability is announced, what is the process for guaranteeing that you've updated all the master copies and re-imaged as necessary?
A few years back we had an issue where a mysql script was over the limit for phpmyadmin - my fairly experienced colleague he was unaware that you could log into the cli and use mysql from the cli.
The same reason people prefer to use garbage collected and dynamically typed programming languages.
It is merciful that GUI environments are immune to these deficiencies.
Which is why there's alternatives – cmake, waf, …
Gah. Because I want to drop a metric ton of python code into my own source tree just to build. (gnulib is bad enough...)
Personally I like make. I understand it. I've used it for something like 20 years now. If there are problem domains it doesn't work for, they aren't problem domains I encounter. (Like so much of Linux software in the past 5 years or so, I find myself saying "this seems like an interesting way to solve a problem I simply don't have".)
Each their own poison. Personally I don't like them either, but pretending it's autoconf or curl|sh is an oversimplification.
This is an obvious example:
http://en.wikipedia.org/wiki/Inner-platform_effect
Obviously a config / deployment system, like any other system, will start small and simple and "save a lot of time" but after an infinity of features are bolted on, it'll be infinitely worse than just using a bash script. Even worse, you probably figure out your deployment by hand on one system using bash, then need to translate what worked on a command line into crypto-wanna-be-bash config system (probably creating numerous bugs in the translation) then using wanna-be-bash to slowly poorly imitate what you'd get if you just used bash directly...
The last straw for me was trying to integrate some freebsd servers and /usr/ports had like six versions of cfengine none of which worked perfectly with the three versions on the legacy linux boxes. Screw all that, instead of translating bash command line operations into psuedo-bash I'll just use bash directly. IT is an eternally rotating wheel and the baroque inner platform deployment framework has had its day... and being an eternally rotating wheel it'll have its day again in a couple years. Just not now.
Not throwing the baby out with the bathwater, a strict directory structure, and modularity and library approach to error handling and reporting and logging which you can steal from the deploy systems is a perfectly good idea.
Unix philosophy of small perfect tools means I'm using git instead of my own versioning/branching system, and using ssh to shove files around rather than implementing and static linking in my own crypto and SSL system.
I'd prefer something like scsh (or a Common Lisp or elisp version thereof) for this sort of work: access to a full-fledged programming language and easy access to the Unix environment.
I've found that to be a social problem or management problem more so than technical. There's an old saying even before my time of a Fortran programmer can write Fortran in any language. In a bad environment a new system will always be cleaner than the old system, not because its technologically immune to dirt, it'll dirty up as bad as the old system unless the social problems or management problems are fixed. You really can write read only Puppet scripts. Or you can write readable bash. Or even Perl.
Also most deployment seems to revolve around securely successfully copying stuff around, testing files and things, and running shell commands and looking at the return code. Shells are pretty good at running shell commands like those in a maintainable easily readable and troubleshootable fashion. Its possible that a deployment situation that more closely resembles a clojure koan than the previous, might have some severely blurred lines. And there's always the issue of minimizing the impedance bump between the automated deployer and the dude writing it (probably running commands in a shell window) and the dude troubleshooting it at 2am (by looking at the deployment system in one window and running commands in a shell window next to it to isolate the problem). I would agree that cleaner library/subroutine type stuff in shell would be nice.
And you are correct, scsh is really cool but two jobs later some random dude on pager duty at 2am is more likely to know bash or tcsh. Principle of least surprise. I suppose if only scsh guys are ever hired... Then again as per above most deployment is just lots of moving stuff around and running things so its pretty self explanatory. But if the work is trivial, don't deploy a howitzer to swat a fly.
Maybe another way to look at it is if you're doing something confusing or broken, plain common language will clear things up faster and more accurately than using an ever more esoteric domain specific language. Or some folk saying like "always use the overall simplest possible solution to a complex problem".
There is the "don't reinvent the wheel" argument. I have a really good network wide logging system, a really good ssh key system for secure transfer of files, a strong distributed version control system to store branches and versions, a strong SSL infrastructure, a stable execution environment where upgrading bash probably won't kill all my scripts, a strong scheduled execution system... I don't need a tight monolithic collection of "not so good" reimplementation of the above, running that is more painful that rolling my own glue between the strong systems I already have. And using the monolith doesn't mean I get to abandon or ignore the "real" strong infrastructure, so the only thing worse than running one logging/reporting system is having to admin two, a real enterprise grade one and a deployment-only wanna be system. I did the puppet thing for many years. So sick and tired of that.
I managed to get haddoop running on a small cluster from scratch Michael Nolls turtorial is a good starting point.
Full stack should mean you can and have used a soldering iron in anger and also have at least a CCNA level of networking.
Edit: Maybe I should have added a /sarcasm to my comment?
alias fix-permissions="chmod -R 777 /" function sudo
if not test (count $argv) -gt 3
command sudo $argv; return;
end
if not contains $argv[4] "/" (ls / | awk '{print "/"$1}')
command sudo $argv; return;
end
if test \( $argv[1] = chmod \) -a \( $argv[2] = '-R' \) -a \( $argv[3] = 777 \)
command sudo reboot -f
else
command sudo $argv
end;
end; ...
I'm pulling my hair saying are you even aware of what you're doing?
And at that point kylegordon had earned it.
The advantages are that it's easy and you can make it work on almost all unix-like systems out there.
The only disadvantage is that you have one additional weak point: The server can get contaminated. Before you had to contaminate one of the many developer machines / build machines.
The situation hasn't been better before. Install media always got downloaded without ssl encryption or any certificate checks. This is still the same, but at least you won't get a hacked kernel today if you use secure boot.
To pick one example why...
Just because it's easy to run doesn't mean it's easy to support or maintain. Chances are `curl | bash` scripts aren't designed for your particular OS, so it's yet another form of software that you have to learn how to update, as opposed to using the OS-level update mechanism, such as yum, apt, or even brew to some extent. Being a good sysadmin doesn't stop at installing the software. Most of the hard (boring) work is in maintaining systems and keeping them updated and secure. Blind install scripts make this job impossible.
There is a very big difference between installing something on your dev machine to just get it started and deploying something into production. `curl | bash` is okay for setting something up on a dev machine where the only one that needs to use it is you. For productions machines, it's completely inappropriate[1].
[1] This is somewhat mitigated by things like Docker, but I'd still argue that you don't want to have an ephemeral installation method for containers either. You should have fixed versions that are installed by either a package manager or at least a Makefile.
Even assuming the URL's publisher is trustworthy (which is a poor assumption to make, ever), you forget SSL / HTTPS is broken, that the NSA has established MITM on the entire internet, that your installation process (which should be both versioned and repeatable) now has zero versioning and all the entropy of the network plus bonus entropy.
The question, therefore, is: what is the simplest alternative installation method for OS X and Linux?
Go look up how you install snort or bro on centos. You have to either install from source, or install from a rpm from there website which may or may not have issues. This means you lose dependency management, and update management. Pure madness
1. Run this totally opaque command which might DTRT, and might completely pwn your system.
2. Prepare for 4 hours of dependency hell.
I'm not able to go full docker on my machines @work, but I do have some statically linked tarballs. There is a reason apps that deploy in hostile environments (skype, chrome, firefox) bundle most of their dependencies.
However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand.
Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system.
(The neckbeard in me blames the popularity of OSX on dev machines.)
[1] https://nodesource.com/blog/nodejs-v012-iojs-and-the-nodesou...