If you want to switch your existing distro scripts to fish, you need to rewrite a lot of stuff. If you want to start a new distro, "it's written in fish!" isn't a terribly compelling selling point, since I don't think anyone is actively picking distros based on their tooling language.
For the general devops case, there's a lot of existing example code out there for bash scripts, and far less for fish. "How do I do X in bash?" is probably going to get you a decent example on Stack Overflow. Devops (I feel) cares a bit more about the installed-everywhere thing, and fish isn't a standard part of most distros.
Oil having a goal of being completely sh/bash compatible gives it much greater odds of something like Debian switching to it, since it wouldn't carry huge technical debt along with it.
Any new shell that hopes to compete in the systems space will have to be POSIX compliant. At least until there is a new widely adopted shell standard.
I would use the crap out of a Linux distribution built in python.
https://github.com/dcantrell/pycoreutils https://github.com/davidfischer/pycoreutils
http://u-root.tk/ does that for Go.
actually now you mentioned it, barring possible performance issues I'd also like a Python distro
The thing that is working for me is to explore VB.net . It is really an interesting language with many historical features. I've noticed that all the languages I use (Python, C#, Java, Go, etc.) distracted me from the essence of programming. When I started programming I immidiately thought about patterns, architecture, unit-tests, integration-test etc.
In VB.net I feel like I'm slowly reexperiencing the joy I've felt in my earlier days of programming.
Ruby was the first language that I experienced that got out of the way, so I felt like I could think about what I was trying to achieve, rather being distracted by house-keeping. Of the other languages that I've tried, I've only gotten close to the same feeling with JavaScript, Go and Python.
I suspect that a lot of it is to do with your own mind-set at the time: I first learned Python as an applications language, with all of the add-on tools and baggage, and didn't like it much, but recently I wrote a one-file, no-dependency thing to do exactly one job in Python, using just the standard library, and it was a joy.
The difference was, I think, that I did not feel the internal pressure to meet best practices, and could just write the code. For some languages, the stuff that you are supposed to do and the tools that you are supposed to use are a heavy burden.
the illusion is almost real, because sometimes i also run linux in vmware/docker/... and when i am not paying attention i forget easily which os i am on.
the unix subsystem for win is quite boring as you can't mix windows applications with linux ones, for example running visual studio build from bash.
powershell would be probably worth learning, if i'd have to, but i do not. it may be godsend to win sysadmins though, as classic cmd is a 40 year old joke.
that was not possible when i tried it, thanks for pointing it out
I recommend Emacs. Maybe not as a daily driver (it's a matter of preference), but for the experience. I recommend at least a month with it, make sure to write some original Lisp code for your customisations. (You WILL end up customizing it, the defaults are crap.)
> or a window manager
I recommend Awesome. The core is in C, but that's basically the low-level stuff, the actual WM is written in Lua. If you'd start with an empty init file, you'd have a moderately sized side-project on your hands to put something usable together. But if you'd start from the stock rc, there's a world of endless tweaking and customisation waiting for you...
Watch out, both Emacs and Awesome are rabbit holes.
As someone who has recently joined the 1k LOC Emacs configuration file club, I have to disagree. The default settings are a well thought out starting point that needs minimal tweaking to get to exactly where you want. I try out a lot of packages on ELPA, and many that purport to provide alternatives for defaults end up being inferior to using the defaults with some small tweaks. The convenient thing is that you can customize the third-party packages in the same way and just use the parts you like from them.
The other good thing about Emacs is that you don't need a separate window manager[1], and you don't need to use a separate command shell (Emacs comes with eshell), or to use a terminal emulator for SSHing into remote machines (eshell over TRAMP is absolutely amazing), or a file manager (Emacs comes with dired, which works over TRAMP which is amazing), or a...
org-mode is a good one, too, but I feel like I've barely scratched the surface there.
When it comes to magit and TRAMP, definitely the best part is with-editor[1]. This is one of those ingenious "why didn't I think of this?" hacks to let you use your local Emacs as EDITOR on remote machines without having to do SSH port forwarding for emacsclient.
I even started to suspect that an easy customization of a tool can be a bad sign. It is almost like developers delegates all usability issues to the end user making the default case really bad.
I've used -- for longer periods of time, of course -- Awesome, Xmonad, i3, and dwm; while some of them are better at some things than StumpWM, StumpWM is the only one that provides an Emacs, live-hackable experience. So these days, I strongly prefer that.
Now, I quite seriously believe that a 1000 line shell script only exists out of error. I still occasionally end up doing 2-300 line dense shell scripts, but not without feeling very dirty along the way. Either split into small, simple shell scripts (which is fine), or a different language.
In the cross platform build pipeline at work, I keep a strong discipline when it comes to scripts: They must be short (<=100 lines), and if their complexity exceeds a certain threshold (parsing more than a "grep | cut" here and there, total program states exceeding some low number), then a shell script is no longer acceptable regardless of length. And, well, it's not safe to assume the presence of anything more than a shell.
If you are writing and dealing with 1000+ lines of shell scripts, then experience tells me that you are shooting yourself in the foot. With a gatling gun.
(I used fish, btw. The interactive experience was nice, but the syntax just felt different without much gain, which was frustrating to someone who often writes inline oneliners. Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control. I can't force the entire company to install fish on all our lab servers, nor is it okay to install a shell on another persons' dev machine just because we need to coorperate.)
My rough heuristic is, "no more than ten lines, nor more than two variables." Yes, that's short almost to the point of absurdity. The only good thing that one can say about Bash as a scripting language is that it's better than csh. Bash is taken seriously because of its longevity and ubiquity, but it's fundamentally limited in what it can express, and it is quite trivial to exceed those limitations.
It's definitely easier/safer to write longer programs in other languages, but with proper discipline it's also possible to write really big, robust programs in Bash/shells--people have done this, and some of that code is probably still running today.
A lot of that code is still being written today, which is one of the things that OP is trying to change.
It's funny though, after decades of shying away from them I've now gone full circle and embraced Makefiles once more. Perhaps it's the cleanness of the bmake/pmake extensions as compared to those of GNU Make, but the determinism, zero dependencies (no CMake or meson and its python baggage), automatic parallelization, and strict error handling call to me each time I have to write mission critical "glue code" that must. just. work.
That's a fair point. Nothing beats shells for ubiquity and supportedness everywhere . . . in theory.
In practice, even people who write "portable shell scripts" (almost) never really write portable shell scripts.
If you learn the POSIX sh standard like the back of your hand, and stick only to the features in it, never using bashisms or equivalent, most shellscripts still rely on external programs (even if only grep/sed/etc) to do their heavy lifting.
And that's where you get into trouble. Because compared to the variability in behavior of even "standard" ultra-common programs, the variability in behaviors between shell syntaxes/POSIX-vs-non-POSIX shells is tiny. The instant your code invokes an external program, no matter how ubiquitous that program is, you have to worry about:
- People screwing with PATH and changing the program you get.
- People screwing with variables that affect the external program's behavior: LD_LIBRARY_PATH for dynamic executables, or language-specific globals for programs' runtime behavior (CLASSPATH, PERL5LIB, PYTHONPATH, etc.).
- External program "editions" (e.g. non-GNU sed vs GNU sed). Good luck using PCRE with a non-GNU "grep"! Oh, and if you're sticking to POSIX-only shell semantics (no bashisms), you don't even get full BREs in most places you need them in the shell; you're stuck with limited BRE or globbing, which makes editing 100-line PCRE regexes feel like a dream.
- External program per-version differences.
- External program configuration.
- Etc.
Dealing with those issues is where "self-test"/"commandline program feature probe" things like autotools really shine. Raw shellscripts, though, very seldom live up to the "ubiquity" promise.
After learning and using many other "modern" Makefile replacements of various complexities (ninja, cmake, scons, tup, meson, bazel, and others) I realized if you're not using exactly what the tool was designed for, you end up recreating a Makefile (e.g.: meson is awesome for cross-platform C++ builds, but if you try to use it to build an environment composed of output from various processes that aren't C++ compilers, writing a Makefile is easier). CMake, apart from also being too C/C++-specific, would be nice if it didn't require a million different files and didn't have such a god awful language.
The only one I liked as a general purpose build system was ninja but it is too restrictive to code in (requires duplication of code by design, not meant to be written by hand though I still do from time to time) and tup, but tup is built around fuse (instead of kqueue/inotify/FindFirstChangeNotification) and so is a no-go for anything serious.
Once I embraced Makefiles, it turned out that most things traditionally built with shell scripts should actually be Makefiles for determinism. For example, I just used bmake to take a clean FreeBSD AMI and turn it into the environment I need to perform some task every n intervals (the task itself was turned to a Makefile rule), in place of where Puppet and other tools would normally have been needed, but would have been overkill for my needs.
The only drawback to Makefiles that I haven't found a clean solution to is when you need to maintain state within a rule (without poisoning the global environment). The only solutions I can see are either a) using a file to store state instead of a variable, which is just stupid, b) calling a separate shell script (with `set -e; set -x` to try and mimic Make behavior) which sort of defeats the point of Make, and c) multiline rules with ;\ everywhere, which is hideous and error-prone but works (though it makes debugging a nightmare as the rule executes as one command, again defeating some of the benefits of Make).
Perhaps the best use case for Oil is to provide a debugging environment where you can figure out what your legacy shell scripts are doing, and rewrite them in another language.
> Unlearning bash-isms is not a liberty I can afford, as I need to be proficient when I SSH into a machine I do not own or control.
This is a slippery slope. I have heard things like, "don't make your own custom aliases / shell functions, because they won't be available when you SSH to another machine." Forcing yourself to always use the lowest common denominator of software is not a fun path.
(Although note that bash actually has a debugger called bashdb, which I didn't know about until recently, and I've never heard of anyone using it.)
One intermediate step I would like to take is to provide a hook to dump the interpreter state on an error (set -e). Sort of like a "stack trace on steroids".
If anyone is running shell cron jobs in the cloud and would like this, please contact me at [email protected].
I want to give people a reason to use OSH -- right now there is really no reason to use it, since it is doing things bash already does. But I think a stack trace + interpreter dump in the cloud (like Sentry and all those services) would be a compelling feature. I'm looking for feedback on that.
Shell scripts have a very small amount of in-process state, so it should be easy to dump every variable in the program.
Also some kind of logging hook might be interesting for cloud use cases.
Second this. Been there, and back, and there again, and recently back again.
Customize the hell out of your shell, make it your place, make it nice. You're spending your day there, every day. Treat it like you'd treat your work desk.
If it's a stranger's machine, well, OK, suffer through the 10 minutes of troubleshooting, an occasional session with busybox also helps keep "pure POSIX" skillset fresh. If you're becoming a regular there, it's time to think how to make your dotfiles portable. google.com/search?q=dotfiles
OTOH, I’m also against, for example, installing zsh fleetwide in production fleets to accommodate choosy folks. So I’m on both ends of the problem, and know it.
- So you've found yourself SSH'ing into a particular box, because that's just the best way to work on a problem. Either it's a pet box (in which case it makes total sense to drop your dotfiles there), or it's a cattle box and you don't care about the trash you left behind, because autoscaling will clean it up for you.
My setup is customized, but the core experience is kept relatively untouched. If someone experiences a problem, I need to be able to debug it in place efficiently. I write drivers for the company's hardware for a living—troubleshooting takes more than 10 minutes.
My approach to that problem is to just instrument scripts to write out the command line every time a tool is called, and infer the logic behind it. When I started here, build/package/test on Windows were controlled by 6000 lines of Perl... A printing version of system() and a few days later, it's 40 lines of batch and 150 lines of Python.
> Forcing yourself to always use the lowest common denominator of software is not a fun path.
Never said it was fun. Also, I need to be able to use the lowest common denominator, I don't necessarily have to be the lowest common denominator. I use a configured zsh and nvim instance myself, but I take care to ensure that if I'm stuck with vi and (ba|da|k|c)sh, I'm still productive. The core behavior of my VIM instance stays close to stock, but it has a bunch of extras like fuzzy search, linting, a nice theme, etc.
And, if I can make a choice without caring, my needs for a shell and a programming language are in direct opposition. I want my shell to put all of its energy into a useful interactive experience.
If you're logging into a machine to the point you're editing files or require your customizations in this day and age you're doing it wrong IMHO.
echo "${myArray[2]}"
Bash is weird.
Subprocess is a bit verbose to use, though... A wrapper would be nice.
... unfortunately, I'm stuck with python2.4 or 2.6 on most machines my Python code has to run on. :/
Not even 2.7?! 2.6 hasn't had a security update since 2013[0], I dread to think how old 2.4 is.
You can install a newer version, but our CentOS 5 machines are meant for compatibility tests, so modifying their setup is unacceptable.
I believe they are available for CentOS 5.x?
Well, the question is: what do you use as a replacement? For my scripts I tend to pull in some version of PHP as soon as needed. Perl is something that's universally available in anything based on Debian or Ubuntu, but it's a maintenance nightmare, and PHP doesn't care whether I use spaces, tabs or a mixture, or if there are due to any circumstances mixed line endings in a file whereas Python may or may not barf on encountering any of said things...
https://github.com/roswell/roswell
(As a side-note: Perl 5 was really the ideal language for this, by design; we might be in a different world today if it hadn't been derailed by Perl Forever^W 6).
Anyway, I usually avoid perl. While definitely a true programming language, I really don't like it. I find that it lacks clarity, and quickly collapses into bash-like hackery.
I personally use Python when it's a notch above shell, but not enough to pull out the big guns.
(I use to write a lot of perl when I was a kid, even made an accounting system in it, so my dislike of perl is from experience. We also had a 6000 line perl build script at my current job which was the last drop. Rewrote it to 40 lines of batch and 150 lines of Python. Likewise, I also wrote a lot of PHP, and can no longer consider it a proper programming language.)
Are you claiming that its size was because Perl forced it to be long winded?
I feel like some programming languages, no matter the structure of the application, feels unstructured and write-only in nature. To me, Perl is one of such languages. Python has a similar effect, but I feel like it is more controllable as long as the project is kept within a sensible length.
So while that abomination was primarily the fault of the designer, I do think that Perl invites this behavior one way or another.
(I gave up entirely on reading that disaster of a build script, and instead instrumented system() so that I could trace the calls to various tools and infer the logic myself.)
http://www.oilshell.org/blog/2018/01/28.html#toc_7
I didn't write those scripts, and you rely on them, whether you know it or not.
Do you use Unix? Do you use the cloud? You rely on them. See:
https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_...
Also, shell has functions.
However I do think it is a disaster that people can't break up their shell code into more modular components, and there is no excuse for it.
And I especially do not wish to discourage you from trying to reinvent shell, because someone has to do it, but at minimum I don't see myself moving away from my shell for the better part of a decade. Hopefully at that point Oil's community has matured to the point where I feel comfortable trusting it to be stable and free of any critical privilege escalation bugs.
Good luck!
People also implement new SQL injection bugs every day—just because a lot of people do it doesn't mean we should just let it slide.
> 1. People who use shell to type a few commands here and there.
> 2. People who write scripts, which may get into the hundreds or even thousands of lines.
> Oil is aimed at group 2. If you're in group 1, there's admittedly no reason to use it right now.
From my perspective, fish is basically the opposite of oil in that it mainly targets group 1 (admittedly I'm biased in that I rarely write shell scripts at all). IME fish scripts as a direct replacement for bash scripts are pretty rare, but that doesn't mean it's not a success if it's aimed at interactive use.
Where's "my shell is my desktop environment"? :/
Everyone's desktop environment is a shell.
But any shell you like is suitable for that use case since it doesn't have to be compatible with anything other than your fingers.
Second, I was recently chided by another fish user for writing fish scripts: "you're using it wrong" was the sentiment. The argument was it's a shell intended to be interactive, but extendable. I don't know how pervasive this line of thinking is, but who's going to learn all about their shell if they're not supposed to write scripts in its language?
There's sh which is the most portable. Why write scripts in anything else but an actual programming language when things get more complex?
Do you really need 1000+ lines of shell scripts? I almost have 20 years with shell scripting and I still have the rule of not writing longer than 1-2 page shell scripts and use something else for longer projects or break it down to much smaller chunks and invoke the parts with a main.sh.
Fish is a for making software engineering and system administration easier by having functionality like command completion (without tab) and it works like a charm. I would never ever would nother to rewrite my bash scripts in fish simply because bash is on all the servers and fish is on none by default.
I am not sure what you are trying to solve here but good luck.
1. Keep up the great work (both oil and the blog!) Both seem fun :)
2. Reading your blog really shows that despite the great ideas, shell languages come from an age where computing was still in its infancy. We’re now approaching our teen years, but we’re still ahead of infants. So many awkward design decisions and organic evolutions. A shame that replacing POSIX at this point is a gargantuan undertaking.
I talked with the authors of Ion on github almost a year ago. Ion was influenced by Oil, in particular this post:
Not sure whether I will actually get to it though. Parsing sounds like a lot of headache. Your work is very impressive, but I’m not sure I will want to go through that... :)
Maybe bind is easier than I think; maybe it's just a few readline calls. I haven't looked into it.
But also, I noticed almost no shell scripts use extended globs like @(foo|bar), whereas the bash-completions package uses them EVERYWHERE. Completion scripts are almost like a different dialect than other shell scripts.
If you want to source /etc/bashrc on most distros, you're going to end up with thousands and thousands of lines of plugins from the bash-completions project.
The parsing is a big headache, but I think it forced me to learn how people use shell, which will benefit Oil.
It's been a full year since our alternative shells thread... might be time to revive it and see what's changed! Congrats on the Elvish releases!
I'm not sure how much work one needs to do to implement things like extended globbing, but my gut feeling is that once you can parse them it's more than half done. The functionalities are quite simple, as my impression goes.
I was mainly thinking about environment variables, aliases and wrapper functions when I said sourcing bashrc; that's probably what most people's bashrc files are mostly made of.
We totally should revive our alternative shells thread! I'll try to summarize my progress in the last year. Congrats on your progress with Oil too! ^_^
The two shells target different use cases. Hopefully they are both successful in attracting a good audience.
However I don't see it being used AT ALL for the cloud/linux use case? Those are the cases where you tend to get 1000+ lines of shell scripts.
For example, I mention Kubernetes/Docker/Chef, and I've never seen fish in that space.
I also don't know of any Linux distro that uses fish as their foundation -- they all appear to use a POSIX shell like bash/dash/busybox ash. fish is "on top".
See Success with Aboriginal, Alpine, and Debian http://www.oilshell.org/blog/2018/01/15.html -- these distros are built with thousands of lines of shell scripts.
Either 1) I don't know about such usage, 2) people don't know that fish can be used this way, or 3) there is some problem with using fish this way.
I link to this post in the FAQ, which I think is a lot closer to Oil:
https://ilya-sher.org/2017/07/07/why-next-generation-shell/
It's basically the "devops" use case. (And as I mention the main difference between Oil and NGS is that Oil is compatible / has an upgrade path from bash.)