Re:Oil 0.9.4 – User Feedback
by chubot on Saturday November 20, 2021 @07:14PM
(#29292994)
(author here) If you've been following the project and are interested in getting involved, now is a good time as I'm helping new contributors get up to speed. This post describes recent efforts, which I want to expand on.
If you know Python and shell, or C/C++ and shell, you can almost certainly contribute something!
I should add that if you've gone through Crafting Interpreters (https://craftinginterpreters.com/) like I've seen so many people do, Oil is nearly identical architecturally to the first interpreter in Java, even though it's written in (statically typed) Python. The ideas apply directly, but this is a bit more "real world", with more detail, and with the addition of about 20 syscalls to implement a shell!
Re:Oil 0.9.4 – User Feedback
by encryptluks2 on Saturday November 20, 2021 @05:29PM
(#29292419)
It is refreshing to see an open source project that uses other open source components for their community, like Zulip. It is frustrating when you go to an open source project page and their entire community is designed around SaaS and proprietary solutions, like Slack, Discord, Twitter, etc.
Re:Oil 0.9.4 – User Feedback
by yeetaccount2 on Saturday November 20, 2021 @10:18PM
(#29293900)
I got out of the software world a long time ago but when I dip my toe in and see FOSS projects with proprietary software/SaaS used for dev coordination I lose my shit. Email, forums, and IRC will take you a very long way. Can’t say I’ve ever needed trademarked services to coordinate. You only lock yourself into the limitations somebody else sets for you.
Re:Oil 0.9.4 – User Feedback
by DarkCrusader2 on Sunday November 21, 2021 @07:33AM
(#29295889)
From the readme:
> It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python.
Why Python developers will do absolutely anything to work around the limitations of python but not use a different language.
Re:Oil 0.9.4 – User Feedback
by godDLL on Sunday November 21, 2021 @11:48PM
(#29302864)
Python is a very sound environment, with fantastic libraries and documentation. The tooling is superb. The resulting code is poetry, readable and expressive.
I personally have moved on to a Nim + Lua combo (depending on use case), because I can see your point. But I will always love Python.
Re:Oil 0.9.4 – User Feedback
by DarkCrusader2 on Monday November 22, 2021 @02:48AM
(#29303875)
Agree with you there, except the tooling part. I still don't love the dependency management having gone through pip/venv, conda, pipenv, poetry, ?? over the years. Everything else is fantastic with excellent community.
I too am looking into nim more and more recently because I don't want to write code without static typing anymore and it has very nice syntax and native compilation.
Re:Oil 0.9.4 – User Feedback
by godDLL on Wednesday November 24, 2021 @04:47PM
(#29335561)
The depend mgmt part is like that because people have different wants. There are DevOps wants, system packager wants, user wants and coder wants. And they are somewhat orthogonal, but not entirely.
Each of the tools you mention have their good part wrt/ one of those four categories, and the rest is an afterthought.
They did not end up doing one thing well, but one thing well and the other three too. Not as well.
The way, or one way to change that would be to borrow the overall idea and semantics from each "good" implementation, but to limit it to just that; and build four separate projects under one community, so that there is pollination of knowledge and efforts, but no cross-contamination of goals and semantics.
I'm not sure I am the person to start something like that.
Re:Oil 0.9.4 – User Feedback
by nawgz on Sunday November 21, 2021 @08:02AM
(#29296011)
> why python developers
Isn’t it a perfectly understandable reason? We regularly pan C++ on here for its smorgasbord of implementation options for any given thing, and the complexity, customization, and fragility of a C++ code base always seems to be high especially for non experts in the domain.
So, accordingly, people who know how to maintain code in a simpler and more flexible setup and can still convert to C++ for performance are taking a probable performance hit relative to “native” C++ (whatever that is in 2021), in order to maintain their tool with more ease.
It’s very silly you move to slandering python developers and not contemplating the trade offs meaningfully
Re:Oil 0.9.4 – User Feedback
by DarkCrusader2 on Sunday November 21, 2021 @08:13AM
(#29296046)
I am not sure how this is a slander. My point is, if python does not meet your performance/size/distribution requirements, choose something else instead of using "custom tools". Unless, compilation to other languages is supported as a first class feature by the language itself (for eg. the way Scala supports javascript and native binary), these work around are duct tapes to make Python do something it is not meant to do.
Furthermore, there is no shortage of statically typed compiled languages to choose from. Go, Rust, Java(with GraalVM) or my recent favourite, Nim comes to mind.
Re:Oil 0.9.4 – User Feedback
by nawgz on Sunday November 21, 2021 @08:20AM
(#29296070)
It is slander because you asked a question as if it was rhetorical when that actually had the easy answer that “the devs find Python to be the nicest C++ front end”, and again as I suggested it more or less seems to me that C++ front ends are all custom.
Re:Oil 0.9.4 – User Feedback
by EdwardDiego on Saturday November 20, 2021 @10:28PM
(#29293954)
Question for Oil users who've also used Fish, how would you compare the two?
I'm greatly enjoying Fish, I really appreciate `history merge` and fish_add_path, and only hit the occasional muscle memory issue with things like $(expr) should be written as (expr), but Oil looks far more like Bash++, which I'm in favour of also.
Oil has the architecture and possibility to be something like fish (a great interactive shell). It has a nascent headless shell mode, e.g. for being driven by GUIs, which no other shell I know of has. As far as I recall, the Warp terminal that appeared here has to parse the stdout of a shell session in a weird way to deliver some nice UI features. The headless shell provides a clean API for that (or at least it should; people need to test it).
And Oil uses a principled and accurate parser is for autocompletion. No other POSIX shell does this.
But personally I'm putting that as a low priority in favor of the "batch" use case. I think the "killer use case" will probably be to have a better language for continuous builds and "cloud automation", which on current platforms like Github, Gitlab, and sourcehut use a lot of YAML and shell.
I use a lot of this automation myself, and I expect it to dramatically increase in the software world in the next decade. And I expect new platforms to appear.
There a lots of people complaining about playing whack-a-mole and with YAML and shell errors, waiting for a "git push" every time, etc. Also, these systems are profoundly inefficient. A better language could be the foundation for much better systems. Over the last year, the blog has hinted at this kind of thing pretty frequently. Get in touch if you're interested :)
Re:Oil 0.9.4 – User Feedback
by jwdunne on Sunday November 21, 2021 @04:55AM
(#29295220)
> I think the "killer use case" will probably be to have a better language for continuous builds and "cloud automation", which on current platforms like Github, Gitlab, and sourcehut use a lot of YAML and shell.
I feel the pain of this.
In fact, it often goes like this:
1. Start in Yaml since things are simple enough
2. Use a multi-line Yaml string when things get worse
3. Extract to a shell script when things get even worse than that
Except things are still painful after step 3: I’m now writing bash. Shellcheck helps but still hit gotchas. Other team members avoid it completely.
The problem with “just use Python” is that it’s not as straight forward as doing that. We have to set up venvs and make sure deps in requirements.txt are installed. And we’re not using it on the project we’re building (the languages we are using simply are not as good for scripts).
What is needed is a better shell, where we don’t feel as apprehensive about using shell scripts for project automation and those shell scripts are maintainable by everyone on the team.
I could internalise enough bash warts to write and maintain them, but I can’t expect to burden my team mates with crap that’s far more painful than it should be to maintain.
So yeah, I’m now at a point where:
1. I’ll try a new shell and try for pastures greener
2. I’m sick to death of debugging half-baked Lisp-as-Yaml for automation and if the guys who solve 1 also solve this, I’d owe a debt I could never repay.
Re:Oil 0.9.4 – User Feedback
by chubot on Sunday November 21, 2021 @11:47AM
(#29297634)
I could internalise enough bash warts to write and maintain them, but I can’t expect to burden my team mates with crap that’s far more painful than it should be to maintain.
Yup, this reminds me of how the Oil project started.
I was getting a lot of work done in shell -- invoking tools I wrote in other languages. And sometimes my coworkers were interested in that, although sometimes they avoided it.
But either way, I couldn't really recommend shell to them "with a straight face". It's just too weird and unfamiliar. These are largely Python, C++, and occasionally R users. I saw people write "shell scripts" in C++ simply because that's the language they knew.
So basically Oil is evolving shell into a language that will be more familiar to Python and JavaScript users (and yes even C++ or Rust users), which is the tagline on the home page.
Re:Oil 0.9.4 – User Feedback
by JonChesterfield on Sunday November 21, 2021 @05:35AM
(#29295340)
You might like lua as a compromise between shell and python for this stuff. Embed any extra libraries you want in the lua executable and put that one file on $PATH.
Re:Oil 0.9.4 – User Feedback
by chubot on Sunday November 21, 2021 @10:16AM
(#29296717)
I'd be interested in a report from someone who tries, but I doubt it will be pleasant.
By default, typos in variable names are worse than in Python and worse than shell with set -u: they result in nil rather than an immediate exception.
The 1-based indexing is something people complain about when coming from almost any other language (Python, JS, C, C++, Go, Rust, etc.). The key issue with shell is that most people write it without studying the language.
I'm also unsure of Lua's stability guarantees. It's an embedded language that can change on every version. I know it has broken compatibility at least 4 times, because they're on Lua 5.x. You're expected to change your programs when you upgrade version of Lua, because it's an embedded language by design. It's very different than a shell.
I have heard of people using Lua for "private" shell-like tasks, but I've never seen it in a real codebase, even though Lua is very popular and well documented.
Re:Oil 0.9.4 – User Feedback
by 41b696ef1113 on Sunday November 21, 2021 @04:52PM
(#29300111)
>I'm also unsure of Lua's stability guarantees.
LuaJIT/Lua 5.1 are not going anywhere. Development is done. NeoVIM has made LuaJIT it's default scripting language and has a pretty great FAQ[0] on the topic.
Re:Oil 0.9.4 – User Feedback
by pydry on Sunday November 21, 2021 @03:54AM
(#29295034)
I've tried fish / zsh and I've found them both a bit underwhelming. They were prettier and a bit more convenient, but IIRC in both cases the thing that made me pack it in after a trial was trying to run bash scripts written by other people. They would interpret them differently (once in a subtle, really annoying way) and give weird error messages.
I am curious about oil but I found the "why use oil shell" explanation also slightly underwhelming. I'm also a bit concerned that its sales pitch seems to be almost entirely about "fixing" bash so that it is a better/stricter language. This is setting off alarm bells in my head coz it sounds like the exact same merry go round as last time.
Like the author I heartily agree bash is a shit language and I'll never use it to script anything over 25 SLOC, but the shell is so embedded at this point that I'm loath to switch unless there are some really big wins and/or the transition is seamless. I'm unconvinced of either in this case.
I do get the sense that there's some depth to this shell that isn't readily apparent from a surface glance and there is a lot that has been very carefully thought out, but whether that translates into some notion of "abstract language purity" or something that will actually benefit me on a day to day basis isn't clear.
I also don't quite get this:
>Why not use only Python? Because I'd rather write 100 lines of shell + 100 lines of Python than 500 lines of Python. This is basically the Unix Philosophy, and I didn't understand it until after I'd been programming for many years.
I don't think I've ever written a python program that could be decomposed into 100 lines of shell + 100 lines of python. I'm curious as to what kind of example the author had in their head when they wrote this.
In general I find python programs to be no longer than an equivalent shell, safer and often shorter, largely thanks to the library ecosystem.
Re:Oil 0.9.4 – User Feedback
by bronzecarnage on Sunday November 21, 2021 @05:44AM
(#29295364)
Regarding your first point, why not just run scripts with `bash script.sh` or even better, use shebangs and `./script.sh`?
The standard shell experience is decades old, and makes it cumbersome to keep up with the pace of today. The appeal of zsh/fish is the user experience and extensibility. I have a shell config using zinit [^1] that installs my dev environment for me (fzf, tmux, nvim, and my zsh-plugins) as long as zsh exists on the new machine. While one may argue that they prefer strictly standard bash so that everything is POSIX compliant, you just have to ensure that your scripts are run with bash or whatever shell the script was made for.
Re:Oil 0.9.4 – User Feedback
by jcelerier on Sunday November 21, 2021 @07:16AM
(#29295820)
It makes as much sense to try to run a bash script with zsh than a python or lua script imho. Those are different programming languages, why would you expect things to work the same ?
Re:Oil 0.9.4 – User Feedback
by chubot on Sunday November 21, 2021 @11:41AM
(#29297578)
The project has two sides: a lot of it is about running existing programs and improving the language (OSH), but there is also an upgrade path into a new language Oil:
The OSH part is mature, but the Oil part is less mature, which is why there is more written about OSH. (Yes it's a bit confusing that the whole project is called Oil, and it contains a part called the Oil language.)
----
I need to write a blog post about it, but basically ALL my Python programs are driven by shell scripts, and it saves a ton of code.
Factoring into processes is an important design skill. It lets you write a lot less code and thus create more reliable and stable systems. It's basically policy vs. mechanism, or control plane vs. data plane. These are very important but seems to be out of fashion for some reason. So yeah I have been behind in writing about them, but there are some under #shell-the-good-parts
Re:Oil 0.9.4 – User Feedback
by EdwardDiego on Saturday November 20, 2021 @10:46PM
(#29294045)
* Breakwater (coastal protection against erosion) != water break (sports protection against dehydration)
* love-in (peaceful public gathering focused on meditation, love, music, sex and/or use of psychedelic drugs) != in love
* takeout (North American word for food intended to be consumed away from a restaurant's premises) != outtake, a scene or song not included in the final film/album, although sometimes funny ones appear in the credits.
Re:Oil 0.9.4 – User Feedback
by jwandborg on Sunday November 21, 2021 @10:01AM
(#29296589)
You'll have the same issue with fish (even fish shell), Go, Rust (the game, the language), Airflow, and Dataflow.
You can't reach perfect optimization for searchability without picking a name nobody would pick before now or in the future, and anything less than perfect is arguably subjective, and/or circumstantial, in that it depends on how you formulate your questions, and what the rest of the world has been doing up until you pressed "search".
Search query disambiguation is a good skill to learn, e.g. "fish shell documentation", "go lang machine learning", "Airflow DAG best practices", "GCP Dataflow optimizations".
If you know Python and shell, or C/C++ and shell, you can almost certainly contribute something!
I should add that if you've gone through Crafting Interpreters (https://craftinginterpreters.com/) like I've seen so many people do, Oil is nearly identical architecturally to the first interpreter in Java, even though it's written in (statically typed) Python. The ideas apply directly, but this is a bit more "real world", with more detail, and with the addition of about 20 syscalls to implement a shell!
> It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python.
Why Python developers will do absolutely anything to work around the limitations of python but not use a different language.
I personally have moved on to a Nim + Lua combo (depending on use case), because I can see your point. But I will always love Python.
I too am looking into nim more and more recently because I don't want to write code without static typing anymore and it has very nice syntax and native compilation.
Each of the tools you mention have their good part wrt/ one of those four categories, and the rest is an afterthought.
They did not end up doing one thing well, but one thing well and the other three too. Not as well.
The way, or one way to change that would be to borrow the overall idea and semantics from each "good" implementation, but to limit it to just that; and build four separate projects under one community, so that there is pollination of knowledge and efforts, but no cross-contamination of goals and semantics.
I'm not sure I am the person to start something like that.
Isn’t it a perfectly understandable reason? We regularly pan C++ on here for its smorgasbord of implementation options for any given thing, and the complexity, customization, and fragility of a C++ code base always seems to be high especially for non experts in the domain.
So, accordingly, people who know how to maintain code in a simpler and more flexible setup and can still convert to C++ for performance are taking a probable performance hit relative to “native” C++ (whatever that is in 2021), in order to maintain their tool with more ease.
It’s very silly you move to slandering python developers and not contemplating the trade offs meaningfully
Furthermore, there is no shortage of statically typed compiled languages to choose from. Go, Rust, Java(with GraalVM) or my recent favourite, Nim comes to mind.
I'm greatly enjoying Fish, I really appreciate `history merge` and fish_add_path, and only hit the occasional muscle memory issue with things like $(expr) should be written as (expr), but Oil looks far more like Bash++, which I'm in favour of also.
I talked about that here: https://news.ycombinator.com/item?id=28547185 (also linked at the end of this blog post)
Oil has the architecture and possibility to be something like fish (a great interactive shell). It has a nascent headless shell mode, e.g. for being driven by GUIs, which no other shell I know of has. As far as I recall, the Warp terminal that appeared here has to parse the stdout of a shell session in a weird way to deliver some nice UI features. The headless shell provides a clean API for that (or at least it should; people need to test it).
And Oil uses a principled and accurate parser is for autocompletion. No other POSIX shell does this.
But personally I'm putting that as a low priority in favor of the "batch" use case. I think the "killer use case" will probably be to have a better language for continuous builds and "cloud automation", which on current platforms like Github, Gitlab, and sourcehut use a lot of YAML and shell.
I use a lot of this automation myself, and I expect it to dramatically increase in the software world in the next decade. And I expect new platforms to appear.
There a lots of people complaining about playing whack-a-mole and with YAML and shell errors, waiting for a "git push" every time, etc. Also, these systems are profoundly inefficient. A better language could be the foundation for much better systems. Over the last year, the blog has hinted at this kind of thing pretty frequently. Get in touch if you're interested :)
I feel the pain of this.
In fact, it often goes like this:
1. Start in Yaml since things are simple enough
2. Use a multi-line Yaml string when things get worse
3. Extract to a shell script when things get even worse than that
Except things are still painful after step 3: I’m now writing bash. Shellcheck helps but still hit gotchas. Other team members avoid it completely.
The problem with “just use Python” is that it’s not as straight forward as doing that. We have to set up venvs and make sure deps in requirements.txt are installed. And we’re not using it on the project we’re building (the languages we are using simply are not as good for scripts).
What is needed is a better shell, where we don’t feel as apprehensive about using shell scripts for project automation and those shell scripts are maintainable by everyone on the team.
I could internalise enough bash warts to write and maintain them, but I can’t expect to burden my team mates with crap that’s far more painful than it should be to maintain.
So yeah, I’m now at a point where:
1. I’ll try a new shell and try for pastures greener
2. I’m sick to death of debugging half-baked Lisp-as-Yaml for automation and if the guys who solve 1 also solve this, I’d owe a debt I could never repay.
Yup, this reminds me of how the Oil project started.
I was getting a lot of work done in shell -- invoking tools I wrote in other languages. And sometimes my coworkers were interested in that, although sometimes they avoided it.
But either way, I couldn't really recommend shell to them "with a straight face". It's just too weird and unfamiliar. These are largely Python, C++, and occasionally R users. I saw people write "shell scripts" in C++ simply because that's the language they knew.
So basically Oil is evolving shell into a language that will be more familiar to Python and JavaScript users (and yes even C++ or Rust users), which is the tagline on the home page.
By default, typos in variable names are worse than in Python and worse than shell with set -u: they result in nil rather than an immediate exception.
The 1-based indexing is something people complain about when coming from almost any other language (Python, JS, C, C++, Go, Rust, etc.). The key issue with shell is that most people write it without studying the language.
I'm also unsure of Lua's stability guarantees. It's an embedded language that can change on every version. I know it has broken compatibility at least 4 times, because they're on Lua 5.x. You're expected to change your programs when you upgrade version of Lua, because it's an embedded language by design. It's very different than a shell.
I have heard of people using Lua for "private" shell-like tasks, but I've never seen it in a real codebase, even though Lua is very popular and well documented.
LuaJIT/Lua 5.1 are not going anywhere. Development is done. NeoVIM has made LuaJIT it's default scripting language and has a pretty great FAQ[0] on the topic.
[0]https://github.com/neovim/neovim/wiki/FAQ#why-embed-lua-inst...
I am curious about oil but I found the "why use oil shell" explanation also slightly underwhelming. I'm also a bit concerned that its sales pitch seems to be almost entirely about "fixing" bash so that it is a better/stricter language. This is setting off alarm bells in my head coz it sounds like the exact same merry go round as last time.
Like the author I heartily agree bash is a shit language and I'll never use it to script anything over 25 SLOC, but the shell is so embedded at this point that I'm loath to switch unless there are some really big wins and/or the transition is seamless. I'm unconvinced of either in this case.
I do get the sense that there's some depth to this shell that isn't readily apparent from a surface glance and there is a lot that has been very carefully thought out, but whether that translates into some notion of "abstract language purity" or something that will actually benefit me on a day to day basis isn't clear.
I also don't quite get this:
>Why not use only Python? Because I'd rather write 100 lines of shell + 100 lines of Python than 500 lines of Python. This is basically the Unix Philosophy, and I didn't understand it until after I'd been programming for many years.
I don't think I've ever written a python program that could be decomposed into 100 lines of shell + 100 lines of python. I'm curious as to what kind of example the author had in their head when they wrote this.
In general I find python programs to be no longer than an equivalent shell, safer and often shorter, largely thanks to the library ecosystem.
The standard shell experience is decades old, and makes it cumbersome to keep up with the pace of today. The appeal of zsh/fish is the user experience and extensibility. I have a shell config using zinit [^1] that installs my dev environment for me (fzf, tmux, nvim, and my zsh-plugins) as long as zsh exists on the new machine. While one may argue that they prefer strictly standard bash so that everything is POSIX compliant, you just have to ensure that your scripts are run with bash or whatever shell the script was made for.
[^1]: https://github.com/zdharma-continuum/zinit
Because I forget. Because some scripts are run from others. Because I don't want to check the shebang of every script I run.
I would probably deal with the annoyance if I could see a really compelling reason to switch.
>The standard shell experience is decades old, and makes it cumbersome to keep up with the pace of today.
Supposedly but the features I'm missing out on seem to either be cosmetic or something I've been doing in bash for years.
>I have a shell config using zinit [^1] that installs my dev environment for me
I have something similar in bash.
http://www.oilshell.org/release/latest/doc/oil-language-tour...
The OSH part is mature, but the Oil part is less mature, which is why there is more written about OSH. (Yes it's a bit confusing that the whole project is called Oil, and it contains a part called the Oil language.)
----
I need to write a blog post about it, but basically ALL my Python programs are driven by shell scripts, and it saves a ton of code.
The oilshell.org website (which is bigger than it seems) is written this way: http://www.oilshell.org/site.html
All the tests and benchmarks are written this way, links here: https://www.oilshell.org/release/0.9.4/
There are a lot of examples in the benchmark/, test/, build/, etc. dirs: https://github.com/oilshell/oil/tree/master/benchmarks
Factoring into processes is an important design skill. It lets you write a lot less code and thus create more reliable and stable systems. It's basically policy vs. mechanism, or control plane vs. data plane. These are very important but seems to be out of fashion for some reason. So yeah I have been behind in writing about them, but there are some under #shell-the-good-parts
https://www.oilshell.org/blog/tags.html?tag=shell-the-good-p...
I kinda want to make a funny web page that insists that
Oil Shell != Shell Oil
just like
Breakfast (meal) != Fast Break (basketball play)
Blackjack (game) != Jack Black (actor)
Bookcase != Casebook (type of textbook for law students)
Backdrop (a kind of picture) != Drop back (football play apparently)
upturn (improvement) != turn up (to be found)
More examples appreciated :)
* love-in (peaceful public gathering focused on meditation, love, music, sex and/or use of psychedelic drugs) != in love
* takeout (North American word for food intended to be consumed away from a restaurant's premises) != outtake, a scene or song not included in the final film/album, although sometimes funny ones appear in the credits.
Great takeout: https://m.facebook.com/ninsbin
Great outtake: https://m.youtube.com/watch?v=bOpePo5Lx_I
Also thought of overturn (reverse a decision) / turnover (accounting or employees)
You got a typo there, compliment != complement
社会 → "shakai" → society, the world, public
... and probably thousands of others from Japanese.
call in != incall
insight != sight-in
in turn != turn in
instill != still in
upend != end up
You can't reach perfect optimization for searchability without picking a name nobody would pick before now or in the future, and anything less than perfect is arguably subjective, and/or circumstantial, in that it depends on how you formulate your questions, and what the rest of the world has been doing up until you pressed "search".
Search query disambiguation is a good skill to learn, e.g. "fish shell documentation", "go lang machine learning", "Airflow DAG best practices", "GCP Dataflow optimizations".