News for hackers. Stuff that matters.
Re:Shell Scripting with Elixir by DeathArrow on Tuesday February 13, 2024 @03:09AM (#39355609)
As I am in the process of learning F# I am considering using it for scripting.
Re:Shell Scripting with Elixir by thibaut_barrere on Tuesday February 13, 2024 @05:37AM (#39356395)
I have read good things about that!
Re:Shell Scripting with Elixir by miloandmilk on Tuesday February 13, 2024 @05:59AM (#39356560)
Me too!
Re:Shell Scripting with Elixir by ryanjshaw on Tuesday February 13, 2024 @06:54AM (#39356831)
For personal stuff I use F# or C# (less and less, these days) for everything unless it's chaining together other command line tools in a way they're explicitly designed for. I've tried PowerShell and can't stand it.

I have 20+ years of .NET, VS IDE and debugger experience I can leverage - it doesn't make sense for me to use anything else.

Re:Shell Scripting with Elixir by hresvelgr on Tuesday February 13, 2024 @03:43AM (#39355787)
Shell scripting in Elixir is like leaf-blowing your driveway with a helicopter.
Re:Shell Scripting with Elixir by lionkor on Tuesday February 13, 2024 @04:24AM (#39355993)
You should, if you can?
Re:Shell Scripting with Elixir by adityaathalye on Tuesday February 13, 2024 @04:31AM (#39356021)
Absolutely. What is life without any fun?
Re:Shell Scripting with Elixir by diggan on Tuesday February 13, 2024 @03:51AM (#39355823)
Awesome, but too expensive to be within reach of the ordinary person and more harmful to the environment compared to alternative methods?
Re:Shell Scripting with Elixir by bryanrasmussen on Tuesday February 13, 2024 @04:31AM (#39356022)
elevator leaf blowing might be harmful to the environment but whats the argument about elixir in this analogy?
Re:Shell Scripting with Elixir by diggan on Tuesday February 13, 2024 @06:01AM (#39356569)
I have no idea what parent is trying to say with their analogy, I'm trying to tease out some meaning from what they said. If anything, Elixir would be more energy efficient than bash as far as I can tell.
Re:Shell Scripting with Elixir by fredrikholm on Tuesday February 13, 2024 @06:50AM (#39356806)
> Elixir would be more energy efficient than bash as far as I can tell

For long running scripts perhaps. BEAM idles at ~50Mb, and startup is often more than 500ms.

Eg:

  $ time ./hello
  hello world

  real    0m0,009s
  user    0m0,003s
  sys     0m0,007s
  
  $ time elixir hello.exs
  hello world
  
  real    0m0,755s
  user    0m0,992s
  sys     0m0,304s
Where the first example is a BASH script.
Re:Shell Scripting with Elixir by throwawaymaths on Tuesday February 13, 2024 @09:52AM (#39358114)
Yes because when you load elixir you're loading the entire gOddamn Erlang VM. Which you might want for things like reliable parallel downloading, but you don't for hello world.
Re:Shell Scripting with Elixir by toast0 on Tuesday February 13, 2024 @11:19AM (#39359185)
I don't know that you'll get it down to 10 ms, but you can get beam startup a lot faster if you adjust parameters.

The default startup does a lot of things you simply don't need in most shell scripts. Starting a bunch of threads, setting up dist, I can't remember what else off the top of my head.

Re:Shell Scripting with Elixir by fredrikholm on Tuesday February 13, 2024 @01:20PM (#39360842)
Even without the shell and with epmd turned off we're still in the ball bark of 200-300ms:

  $ time erl -eval 'io:format("~s~n", [os:cmd("echo hello world")]), halt(0, [{flush, false}]).' -noshell -start_epmd false
hello world

  real    0m0,332s
  user    0m0,461s
  sys     0m0,092s
I spent some time last year trying to get those numbers down but this is as far as I made it. Do you know of any more flags that might help?
Re:Shell Scripting with Elixir by toast0 on Tuesday February 13, 2024 @03:53PM (#39362673)
Try -boot no_dot_erlang

A couple years ago that got me to 100 ms

https://news.ycombinator.com/item?id=25243246

Today's measurements with FreeBSD 13.2 on the same Pentium G3470

time /bin/sh -c "echo hi"

   hi

   real    0m0.004s
   user    0m0.001s
   sys     0m0.002s
time /usr/local/bin/bash -c "echo hi" hi

   hi

   real    0m0.006s
   user    0m0.007s
   sys     0m0.000s
time /usr/local/lib/erlang26/bin/erl -eval 'io:format("hi~n"), halt(0, [{flush, false}])'

   Erlang/OTP 26 [erts-14.2.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns] [dtrace] [sharing-preserving]

   hi

   real    0m0.197s
   user    0m0.178s
   sys     0m0.018s

time /usr/local/lib/erlang26/bin/erl -eval 'io:format("hi~n"), halt(0, [{flush, false}])' -noshell -start_epmd false -boot no_dot_erlang

   hi

   real    0m0.183s
   user    0m0.181s
   sys     0m0.011s
This doesn't look very good, actually! FreeBSD has OTP 21-26 available through pkg, I ran the same thing on each of them several times to warm up, and get (real time):

   21:  81ms
   22:  81ms
   23:  74ms
   24: 129ms
   25: 135ms
   26: 177ms
So maybe there's some new stuff that needs to be avoided now.
Re:Shell Scripting with Elixir by fredrikholm on Wednesday February 14, 2024 @02:29AM (#39367421)
It was your original post that set me on course for experimenting with startup times, cheers!

Unfortunately, that final flag doesn't change much on 26 for me. IIRC I dropped a bunch of flags after concluding that no shell and no epmd were the only flags that I noticed improvements with.

Re:Shell Scripting with Elixir by toast0 on Wednesday February 14, 2024 @06:18PM (#39376973)
Yeah, it looks like it got a lot worse. I don't think it's JIT, because I ran a test with a 32-bit version of OTP 26 (JIT is only available for 64-bit x86 and arm). But I'm not sure what it is.

Send me an email (in profile), and I'll try to find some time to look closer.

Re:Shell Scripting with Elixir by thibaut_barrere on Tuesday February 13, 2024 @05:35AM (#39356380)
If your script needs to do things like reliable HTTP querying and JSON parsing, I would say using Elixir (once you get used to it) won’t be harmful in any way, quite the contrary.

Interestingly, what drove me to Ruby initially is exactly writing scripting tasks in a C#/C++ culture where Ruby would have been considered harmful.

Now Elixir can be used for the same type of tasks, but only better (eg: trivial parallelized downloads!)

Re:Shell Scripting with Elixir by neonsunset on Tuesday February 13, 2024 @05:48AM (#39356470)
F# and C# are very different to C++ in this regard.

Parallel downloads in C# boil down to

    using var http = new HttpClient();

    var urls = new[]
    {
        "http://example.com/file1.jpg",
        "http://example.com/file2.jpg",
        /* ... */
    };
    var files = await Task.WhenAll(
        urls.Select(http.GetByteArrayAsync));
(you can replace '.GetByteArrayAsync' with .GetFromJsonAsync, .GetStreamAsync and more depending on your needs and e.g. forward the incoming data directly to files on the disk or another source)
Re:Shell Scripting with Elixir by thibaut_barrere on Tuesday February 13, 2024 @05:37AM (#39356389)
Ahaha, in a good way. Recently I have used Elixir scripts to mass-parallel-download numerous HTTP resources and do some stats on them, and it is exactly the feeling I had.
Re:Shell Scripting with Elixir by dtx1 on Tuesday February 13, 2024 @07:07AM (#39356884)
Re:Shell Scripting with Elixir by wturner on Tuesday February 13, 2024 @01:41PM (#39361121)
I envision the creation of a GenServer for each and every file on your system, using this to duplicate them by the thousands, then deleting them in a timed contraction over and over and over again. I call this, "the blowcopter".
Re:Shell Scripting with Elixir by Too on Tuesday February 13, 2024 @01:50PM (#39361239)
Still better than the inverse, that lots of scripters are doing; Building a helicopter out of a leaf-blower.
Re:Shell Scripting with Elixir by datadeft on Tuesday February 13, 2024 @04:07AM (#39355897)
One detail I really like about this is that you can install packages during the script run.

    Mix.install([
     {:req, "~> 0.2"},
     {:floki, "~> 0.30.0"}
   ])
Re:Shell Scripting with Elixir by lambdaba on Tuesday February 13, 2024 @04:35AM (#39356041)
Ruby can do this too:

  require 'bundler/inline'
  
  gemfile do
    source 'https://rubygems.org'
    gem 'json', require: false
    gem 'nap', require: 'rest'
    gem 'cocoapods', '~> 0.34.1'
  end
Re:Shell Scripting with Elixir by behnamoh on Tuesday February 13, 2024 @09:47AM (#39358058)
Can't Python do the same? Like !pip install blahblah
Re:Shell Scripting with Elixir by gkbrk on Tuesday February 13, 2024 @09:49AM (#39358077)
That's not Python syntax. It's a Jupyter extension.
Re:Shell Scripting with Elixir by behnamoh on Tuesday February 13, 2024 @09:53AM (#39358119)
Right, but Python can run command line code in subprocesses too, so it should be possible to run a pip install jupyter command inside Python.
Re:Shell Scripting with Elixir by networked on Wednesday February 14, 2024 @08:11AM (#39369386)
The problem is that installing dependencies for the user or globally will lead to version conflicts. You need a separate virtual environment or a custom installation directory for each script to avoid conflicts. There are script runners and copy-pastable code snippets for Python that handle this by creating the virtual environment and installing the dependencies in it. Check out https://github.com/stars/dbohdan/lists/python-scripts-with-d... (disclosure: my list, and it includes my repository).
Re:Shell Scripting with Elixir by behnamoh on Thursday February 15, 2024 @12:12AM (#39379351)
Thanks, I checked out the link, it's quite handy!
Re:Shell Scripting with Elixir by adityaathalye on Tuesday February 13, 2024 @04:29AM (#39356016)
Shell scripting with ideas from Elixir, chiefly, functional programming style would be profitable too. As an avowed (mostly) functional programmer, I do it FP style [1].

Of course, one can take a Shell script only so far. Eventually a Real Programming Language is the inescapable (pun intended) conclusion. [Insert "Always was" meme].

[1] I wrote about it here.

https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp...

Re:Shell Scripting with Elixir by bruh2 on Tuesday February 13, 2024 @04:51AM (#39356123)
That was so cool. Kind of an eye-opener. Bookmarked, thank you!
Re:Shell Scripting with Elixir by adityaathalye on Tuesday February 13, 2024 @12:20PM (#39360068)
Thank you for the kind remark. It so happened that I picked up shell scripting on the job at a workplace full of FP nerds. Pipelined code felt eerily familiar. Then at some point I read that Archmage McIlroy, a mathematician by training, invented the pipe. That was a lightbulb moment; a big reason why Functional style works as well as it does. Pipes are a mathematical idea.

That whole series of posts is basically a glorified bookmark for yours truly. Tactics and tricks accumulated over the years. I refer back to bits and bobs from time to time. Writing them was a lot of, ah, fn.

Re:Shell Scripting with Elixir by giovannibonetti on Tuesday February 13, 2024 @08:48AM (#39357552)
This Bash library [1] is very handy to make your scripts more functional.

[1] https://github.com/ssledz/bash-fun

Re:Shell Scripting with Elixir by adityaathalye on Tuesday February 13, 2024 @12:31PM (#39360196)
I feel like I've seen this one before. Thanks for bubbling it back up in my brain.

I think `map`, `filter`, and `reduce` (or `accumulate`) are automatic in pipelines; and have usually been enough for me. Still, it has a bunch of stuff one can pinch in a hurry. Starred!

Re:Shell Scripting with Elixir by irjoe on Tuesday February 13, 2024 @05:02AM (#39356170)
You can also use anonymous functions if you find the module syntax a little terse or clunky for shell scripting.

    double = fn a -> add.(a, a) end
    double.(4)
It starts to look a bit like a weird untyped OCaml / F# if you use pattern matching:

    f = fn
      x, y when x > 0 -> x + y
      x, y -> x * y
    end
Re:Shell Scripting with Elixir by losvedir on Tuesday February 13, 2024 @07:18AM (#39356951)
Ever since Elixir 1.12, which introduced the `Mix.install[..]` at the beginning of the script, Elixir has been awesome for this sort of thing. Prior to that, it just wasn't worth it.

I'm surprised the testing worked like that, without trying myself. When I last fiddled with it, I stole a page from how they recommend you do it in Livebook, and had `ExUnit.start(autorun: false)` at the top and `ExUnit.run()` at the bottom. I didn't realize you could just `ExUnit.start()` by itself up front.

For anyone else who's tried an approach like this, have you managed to get doctests to work? I really like them, and think they're especially appropriate where you probably want lots of examples like here, but haven't been able to figure out how to stick them in ad-hoc scripts like this. I get some issue about the code not being compiled and available.

Re:Shell Scripting with Elixir by vander_elst on Tuesday February 13, 2024 @08:10AM (#39357255)
The more I use bash the more I appreciate it. It has a lot of sharp edges and quirks, but it's coincise, it works basically on all Unix systems and everyone knows (at least a bit of) it.

A lot of attempts to fill this space have large gaps in this aspects. Not sure how this scores on those dimensions.

For up to ~50 lines of script, bash is the most effective thing I found so far. Tried zsh, fish, python, ruby, but for such glue code bash rules.

Re:Shell Scripting with Elixir by mhitza on Tuesday February 13, 2024 @09:06AM (#39357715)
> The more I use bash the more I appreciate it.

That is true for sure if you write bash scripts quasi daily. However if you don't it becomes very easy to forget, at least for me. That's why I'm the type of person that tries to look out for alternatives in different languages.

I've had and read from cover to cover, Bash, sed, and AWK books, yet when I have to get back to them I still only remeber the most fundamental features that I use every couple of days or weeks.

Re:Shell Scripting with Elixir by noisy_boy on Tuesday February 13, 2024 @09:45AM (#39358040)
My rule of thumb for moving from bash to Perl/Python etc is when I need to start implementing multiple arrays and associative arrays to get the job done.
Re:Shell Scripting with Elixir by vander_elst on Tuesday February 13, 2024 @10:55AM (#39358801)
Agree
Re:Shell Scripting with Elixir by SomeoneFromCA on Tuesday February 13, 2024 @10:42AM (#39358626)
I found that browsing through the bash source makes remembering things about how to use bash easier. It is simple enough to see the gears at work, comapared to say python interpreter sources.
Re:Shell Scripting with Elixir by behnamoh on Tuesday February 13, 2024 @09:49AM (#39358079)
don't those quirks and sharp edges leave security holes in the script tho?