Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.
And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.
Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.
I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.
It's a complete scripting language, with tons of features, from a comprehensive standard library with something for most needs, to a packaging system, isolated environments, async io, and more. And lots of expressivity in structuring your program (including optional types).
So, not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not.
>Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
If you're doing dev-ops with such scripts, you already need to handle the environment in general.
That scripting in Python is pretty bad. It makes thing harder and at the same time you still have to think a lot about what can go wrong, because there's no enforced error handling.
> "Scripting" in a compiled language is not practical.
The premise is that scripts grow, and you should switch to Python. Which might be okay for Google, but seems to be only a very small step up from Bash, if any at all. (I've outlined the reasons, why I think it's not worth it.)
I do a lot of scripting, in Bash, and when that stars to grow on me, it's time to think about the problem for real. And usually a blind rewrite in Python is not the way to go.
> For some tasks that's perfectly sensible.
Those are the one-off scripts, no? Anything that should run with some regularity, anything that should be reproducible, that handles valuable data, is not something that should be cobbled together in bash/python and thrown into /etc/cron.d . Or apparently that's how Google rolls.
Yeah, well, maybe not everyone is an SRE, so the use case here seems to be build scripts and other not-so-critical scripts, that still have to be maintained, where knowledge of Bash/POSIX-sh is not really assumed, it should just work for anyone who checks out the repo.
>Python lacks efficient, quick and dirty process control.
Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash.
Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.
I don't have more faith in Python's except than in Bash's trap.
If you use 3rd party code or a subprocess, then both are pretty weak compared to something kernel enforced.
> makes it massively easier to handle errors and give decent diagnostics to the user.
I don't really find that. You have to fight for input/output for subprocesses in Python.
Sure, logging is easier in Python/perl/PHP/whatever, than in Bash, but error handling is about the same. You have to do everything manually, and there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Trap is not like an exception, it's more like "atexit" or a signal handler. You can't really nest them, you can't really recover from them. So I definitely disagree that error handling is about the same, python's a lot better.
>there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Well that's the whole static vs. dynamic debate but that's a different story. Use C# or Rust or whatever you feel like if you really can't stand python or scheme. Maybe there's a niche to be filled for statically typed scripting languages, there are no popular ones that I'm aware of.
Still, that's besides the point, while I personally prefer static typing like you there's really no contest between dynamic typing and "everything is a string and you can only concatenate them" typing.
You are aware that error handling is done via exceptions python?
I'm talking about how weak guarantees Python gives compared to Bash about the correctness of the program. It's easy to get None-s, it's too easy to forget to handle file not found or lack of permission.
I've almost never wanted to have my program do anything but crash upon file not found or permission error.
Same goes for permissions issues and a whole slew of similar scenarios. For many use cases, the lack of forced error checking is a feature.
Concrete example: build scripts. As long as the cause of the build failure is clear from the uncaught exception, there's no reason to handle it. I don't need a friendly UI to avoid scaring off non technical users, I just need something that handles the happy path well and crashes with a meaningful stack trace when something unexpected happens.
So basically, for them, Python requires no fiddling and works as is.
Same for the shell. They say "use [[]]" because they know that compat is not a problem: it's the same bash everywhere.
Google has millions of servers, billions of lines of code and surely of lot of experience with deployment, managing complexity and handling teams.
I'm kinda trusting them on this choice.
Very useful when debugging.
What it begs is for a new tool. A single file, statically linked “super shell” that lets you do structured scripting and has builtins for a bunch of common things (reading in addresses, daemonizing processes, reading the process table, etc. and that tool needs to be build able on nearly everything. Now I like rust and the attention it is getting and the interest in safety and correctness. I can’t think of many places where I’d reach for bash but be happier writing rust, a lot of it is 15minute disposable programming jobs. At a glance, I think I’d trade 5minutes of set -e to find some error I skipped for 45 minutes of trying to rejigger things to properly borrow some shared value.
A modern take on Tcl is what we need, in my opinion.
Surely we should be able to build something nicer 30 years after the first release of Tcl.
Tcl's string representation for objects is just a serialization/deserialization mechanism, which seems to be pretty popular in other languages as well.
Additionally all of the cool things in Tcl such as coroutines, threads (as an additional, but included, package), great documentation delivered as man pages, virtual filesystem access, virtual time, safe interpreters for running untrusted code, a stable ABI (stubs) so that compiled extensions can be used for decades, a small memory as well as disk footprint, extremely cross-platform, easy to embed into larger programs as an extension language, easy to compile a large static program with Tcl as the "main" entrypoint, native BigNum support, many thousands of great packages, ....
What would a modern take on Tcl improve on Tcl that Tcl couldn't just build easier ?
Has something changed suddenly in the last 5 years? Yes, the docs had a decent API reference but everything slightly higher up than "what parameters this command has" was on that wiki with comments from 2004 marked as out of date by out of date comments from 2007...
Interested! Available?
[1] https://chiselapp.com/user/rkeene/repository/tuapi/doc/trunk...
But now, to think of it, just using Node + TS (+ yarn) sounds better than Python.
The major disadvantage of any "comfortable" language aside from Python is that they aren't installed by default on nearly every distribution of every OS.
This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage.
If you already have a good mechanism for packaging and distributing software to all your machines (which I think is a reasonable expectation for Google) then go ahead and use whatever language you're well equipped for. But know you'll be sacrificing the easy and wide distribution that using Bash or Python brings you.
Yes, Ansible transfers everything. It's ugly and slow :(
I pretty much gave up on [vanilla] OpenStack because their idea of DevOps is custom python script generated Ansible plays.
But probably for deploy scripts, go with a static binary (hosted on, let's say your GitLab instance - using the GitLab HTTP API with a token to wget/curl it) is nigh unbeatable in end-to-end development and deployment time.
Here's an example that builds a React Native app, while updating version numbers and Git tags:
https://github.com/laurent22/joplin/blob/master/Tools/releas...
I love Node but I use it for backend where the event-loop model is useful.
It's obviously not perfect, but it has replaced lots of Bash scripts for me.
Oh, and of course in doing so, I always try to minimize the amount of calling out to other processes. Why call awk when you could use `while read` instead?
You only get one array to work with in portable Bourne shell, $@. You can parse lines using read, and strip prefixes and suffixes using parameter expansion (`${parameter%pattern}`). It's a restrictive little language, with a lot of quirks and edge cases, but it always makes a nice little challenge to figure out how you can use it safely.
One of the constraints I impose is that spaces in pathnames should always be handled correctly; the biggest form of trouble you can easily get in is to forget to quote something or to parse something on space delimiters when it can include spaces. I've seen someone write a script that accidentally turned into an `rm -rf /path/to/important/data` because of poor handling of spaces.
I generally do not attempt to handle newlines in pathnames. While it's possible to do so in some cases with null delimiters, I have never seen a case of newlines in pathnames in the wild. Of course, this means that I have to make sure that these scripts are never exposed to anything that could contain malicious filenames.
Also, I move away from portable shell if it will really make the code too convoluted or fragile. For instance, if I need to parse long options (--foo), I use GNU getopt rather than the built-in getopts (with appropriate test to make sure the getopt I'm using is GNU, so it won't break silently on a system without GNU getopt), or move to a real language.
Anyhow, while I know it's counterproductive, the puzzle-solving part of me really likes trying to solve the problem of implementing something in portable Bourne shell.
I tend to do the exact opposite. If a filename has spaces in it, i like my script to fail spectacularly while insulting the user (who is typically me....)
Filenames are variable names. It makes no sense to allow spaces in them.
#! /usr/bin/env rdmd
import std.stdio;
void main() { writeln(2); }
#!/usr/bin/tcc -run
#include <stdio.h>
int main() {
printf("Hello, tcc\n");
return 0;
}I think it's nicer.
#!/bin/sh
crun() {
local file="$1"
shift
local exepath="$(mktemp)"
if [[ "$file" =~ \.c$ ]]; then
gcc -g -Wall "$file" -o "$exepath" || return $?
else
echo "no filetype detected"
return 126
fi
"$exepath" "$@" & fg
}
#!/usr/bin/env zsh
function crun {
zparseopts -E -D -- -gcc::=use_gcc \
c:=custom_compiler \
o+:=copts \
Wl+:=lopts \
-dump::=dump_asm \
v::=verbose \
h::=usage \
g::=debug
if [[ -n $usage ]]; then
cat <<- EOF
usage: crun [options]... <filename>
--clang (default) use clang for C & C++ files
--gcc use GCC for C & C++ files
--dump dump assembly of program
-o supply an option (e.g -o -Wall)
-v verbose
-g debug
Compiles and runs a C, C++ or x86 Assembly file.
EOF
return 126
fi
# select unique entries of `copts` and then slice copts[2..] (copts[1]
# contains the flag, e.g "-o")
local file=${@[-1]}
local options=${${(u)copts}[2,-1]}
local exepath="$(mktemp)"
if [[ $file =~ \.(cc|cpp|cxx)$ ]]; then
local compiler="clang++"
$compiler -std=c++1z -g -Wall -Weffc++ ${=options} $file -o $exepath
elif [[ $file =~ \.c$ ]]; then
local compiler="clang"
[[ -n $use_gcc ]] && ccompiler="gcc"
$compiler -g -Wall ${=options} $file -o $exepath
elif [[ $file =~ \.(s|asm)$ ]]; then
local objpath="$(mktemp)"
nasm -felf64 $file -o $objpath && ld $objpath -o $exepath
else
echo "no filetype detected"
return 126
fi || return $?
if [[ -n $dump_asm ]]; then
objdump -S -M intel -d $exepath
else
[[ -n $verbose ]] && echo "exepath: $exepath"
if [[ -n $debug ]]; then
gdb --args "$exepath" "$@"
else
"$exepath" "$@" & fg
fi
fi
}I've told developers that if I catch them writing shell scripts in python they must commit a shell script doing the same set of operations. 2x the work usually dissuades from this type of nonsense.
(If you told me I had to rewrite a working Python script in shell, I'd probably think you were joking. Talk about nonsense.)
My route is to make a good rule about writing python with more than 'n' os., subprocess. and *.Popen calls automatically requiring shell script counterparts in case we find your need to pythonize unsafe, unreadable and slow|broken|buggy.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.