News for hackers. Stuff that matters.
Re:Most programmers don't know what Expect can do for them by repairisntcheap on Sunday September 08, 2019 @01:28AM (#20908772)
One of my most fond programming experiences was using tcl/expect to create a custom web interface for an IBM Tivoli tape library. The stock web interface was too expensive for a college with a few thousand users, and didn't provide adequate access control to allow our users to restore ONLY their own files.

Yes, wrapping a telnet session is a hack... But it was a hell of a hack that did everything we needed!

Re:Most programmers don't know what Expect can do for them by foobar_ on Sunday September 08, 2019 @04:42AM (#20909311)
I'd still use expect instead of docker/ansible/salt ... if I could get away with it.
Re:Most programmers don't know what Expect can do for them by emmelaich on Monday September 09, 2019 @03:19AM (#20915918)
There is an expect module included in Ansible.

Backed by pexpect of course.

Re:Most programmers don't know what Expect can do for them by marmaduke on Sunday September 08, 2019 @05:00AM (#20909368)
Those aren't alternatives; you'd create a Docker image with expect installed, for example.
Re:Most programmers don't know what Expect can do for them by foobar_ on Sunday September 08, 2019 @06:15AM (#20909601)
Yeah ... but I don't want to touch Kubernetes. Kubernetes doesn't spark joy.
Re:Most programmers don't know what Expect can do for them by rmetzler on Sunday September 08, 2019 @11:20AM (#20911052)
Wrapping expect in an Ansible playbook is really useful though.
Re:Most programmers don't know what Expect can do for them by tarkin2 on Sunday September 08, 2019 @03:22AM (#20909088)
If, like me, you still weren't sure what Expect does then this is a great explanation https://www2.lib.uchicago.edu/keith/tcl-course/topics/expect...
Re:Most programmers don't know what Expect can do for them by doctor_eval on Sunday September 08, 2019 @04:56AM (#20909354)
I’ve written so much code in Tcl/Tk/Expect - but it was all pre-Java; literally before Java 1 was released. Tk was used in lots of Linux apps, most notably Zircon, an awesome IRC client.

I used to think that knowing Tcl was a kind of super power. It was incredibly productive. It was far better for quick scripting than either bash or awk - but large programs could get somewhat unmanageable.

It would be great to see a renaissance of TCL usage, though I’d be pretty much re-learning it today.

Re:Most programmers don't know what Expect can do for them by ghettoimp on Sunday September 08, 2019 @08:17AM (#20910085)
TCL is still in use everywhere in hardware development. Practically every EDA tool has a TCL interface.

I'm afraid I don't share your fondness for it. :)

Re:Most programmers don't know what Expect can do for them by tyingq on Sunday September 08, 2019 @08:35AM (#20910171)
It's also in other places, like F5 load balancers.
Re:Most programmers don't know what Expect can do for them by marktangotango on Sunday September 08, 2019 @09:26AM (#20910449)
Lots of talk about TCL in this thread. Everyone should be familiar with the TCL/Guile and GNU scripting story. Also Ousterhout’s dichotomy is a useful concept to understand:

https://en.m.wikipedia.org/wiki/Ousterhout%27s_dichotomy

Re:Most programmers don't know what Expect can do for them by tyingq on Sunday September 08, 2019 @12:01PM (#20911308)
Ousterhout also co-invented raft distributed consensus.
Re:Most programmers don't know what Expect can do for them by cobbzilla on Sunday September 08, 2019 @06:57AM (#20909748)
I used expect long ago, it was fun but a bit cumbersome.

Quite recently I was working on some python that used netmiko, which is similar but more ssh-oriented.

Having used both, I don’t know when I’d ever use expect. Maybe if there was no python available for some reason.

The developer ergonomics on netmiko are just better. It’s not anything about python vs tcl, it’s that netmiko “does more out of the box”, and expect is a bit lower-level.

Re:Most programmers don't know what Expect can do for them by computerfriend on Sunday September 08, 2019 @02:18AM (#20908916)
I ended up rewriting most of my Expect code in Pexpect and found it both nicer to write and much more robust.
Re:Most programmers don't know what Expect can do for them by based2 on Sunday September 08, 2019 @04:36AM (#20909291)
https://github.com/pexpect/pexpect Python module for controlling interactive programs in a pseudo-terminal
Re:Most programmers don't know what Expect can do for them by tyingq on Sunday September 08, 2019 @09:45AM (#20910530)
There's also Perl's Expect.pm. I know Perl isn't popular here, but it excels in this sort of use case. The compact syntax for regexes on responses makes it a good match for expect scripting. It can also be paired with IO:Pty for some advanced stuff.
Re:Most programmers don't know what Expect can do for them by DrJosiah on Sunday September 08, 2019 @11:35PM (#20915024)
You are fighting the good fight, I kept trying and failing. Thank you.

http://code.activestate.com/recipes/440554-module-to-allow-a... https://www.python.org/dev/peps/pep-3145/

Re:Most programmers don't know what Expect can do for them by ohazi on Sunday September 08, 2019 @04:12AM (#20909216)
Whenever I need to "screen scrape" an interactive console application, I usually end up running the process under tmux and interacting with it via Python after wrapping send-keys and capture-pane.
Re:Most programmers don't know what Expect can do for them by kragen on Sunday September 08, 2019 @04:09PM (#20913088)
Do you know there's an expect clone for Python?
Re:Most programmers don't know what Expect can do for them by ohazi on Wednesday September 11, 2019 @03:52PM (#20943983)
I do now. Thanks!
Re:Most programmers don't know what Expect can do for them by kragen on Wednesday September 11, 2019 @04:47PM (#20944502)
Happy to help!
Re:Most programmers don't know what Expect can do for them by kylek on Sunday September 08, 2019 @01:03AM (#20908717)
I winced at this as I've only ever used expect for dirty hacks like wrapping ssh (or telnet to strange devices without real management). I wouldn't discount it though, it's a decent tool when heavier machinery isn't viable.
Re:Most programmers don't know what Expect can do for them by stevekemp on Sunday September 08, 2019 @06:00AM (#20909549)
I used expect for wrapping router-setup, via telnet, back in the day. Hard to think what I'd use it for these days, thanks to puppet, ansible, docker, etc.
Re:Most programmers don't know what Expect can do for them by rubicks on Sunday September 08, 2019 @12:31PM (#20911526)
Expect is nothing short of a blessing for those who need to automate tools that actively resist automation. When some janky cli script decides to read the tty and not stdin, I reach for expect.
Re:Most programmers don't know what Expect can do for them by lcall on Sunday September 08, 2019 @06:51PM (#20913889)
For years I thought "I should learn to use expect...", and finally I used it for ~ integration tests for a project (http://onemodel.org - my AGPL, highly efficient desktop- and keyboard-oriented personal knowledge organizer that I use every day, slow progress lately but lots of plans...), and: expect is very useful, and has depth.
Re:Most programmers don't know what Expect can do for them by mzs on Sunday September 08, 2019 @02:01AM (#20908863)
I wish there was something like expect that was like, "ok I've done what I needed to init, now get out of the way." I've always had to create a loop that just reads and writes everything eventually. Maybe I could ldpreload a tcl interpreter and change some pointers but yikes.
Re:Most programmers don't know what Expect can do for them by _ph_ on Sunday September 08, 2019 @04:37AM (#20909297)
Isn't that what interact is meant to do?

https://wiki.tcl-lang.org/page/interact

Re:Most programmers don't know what Expect can do for them by jraph on Sunday September 08, 2019 @04:42AM (#20909312)
If I understood correctly, you may be able to do that using pexpect and the interact method.

    import struct, fcntl, termios, signal, sys

    def sigwinch_passthrough (sig, data):
        s = struct.pack("HHHH", 0, 0, 0, 0)
        a = struct.unpack('hhhh', fcntl.ioctl(sys.stdout.fileno(),
        termios.TIOCGWINSZ , s))
        if not child.closed:
            child.setwinsize(a[0],a[1])    
    
    child = pexpect.spawnu("sh", args, encoding='utf-8', logfile=sys.stdout)
    signal.signal(signal.SIGWINCH, sigwinch_passthrough)
    child.expect_exact("...")
    child.logfile = None # Otherwise, interact raises an Exception
    child.send("...\n")
    child.interact()
Re:Most programmers don't know what Expect can do for them by solresol on Sunday September 08, 2019 @04:28AM (#20909270)
What's wrong with the interact function?
Re:Most programmers don't know what Expect can do for them by bachmeier on Sunday September 08, 2019 @11:04AM (#20910960)
tcl brings out cries of "dead language" in a way that few other languages do. I'd not hesitate to write a tcl stript today, knowing that it'd still run correctly when I die.

Just in the past week, I've heard tcl, ruby, perl, and java all declared dead. Oh well. Internet comments are cheap and meaningless.

Re:Most programmers don't know what Expect can do for them by eesmith on Sunday September 08, 2019 @01:38PM (#20911987)
TRAC is dead. (Now I hope to hear from people using Mooers' TRAC in production code. Or CART or SAM76. I have talked to people who used TRAC back in the 1980s.)
Re:Most programmers don't know what Expect can do for them by skybrian on Sunday September 08, 2019 @12:12PM (#20911385)
I do know what it can do, but it seems like a specialized tool for screen-scraping command line apps that don't have a proper API. That seems like a rather unusual task these days?
Re:Most programmers don't know what Expect can do for them by rubicks on Sunday September 08, 2019 @12:57PM (#20911678)
I've wrapped expect scripts around prompt-happy config scripts supplied by third-party vendors. What they intend to be idiot-proof usually ends up as an unwieldy tool that takes no options and/or assumes bad defaults. The vendor didn't intend for it to be run more than once per site installation, but such intentions don't play nice with disposable environments like VMs and/or containers.
Re:Most programmers don't know what Expect can do for them by emmelaich on Sunday September 08, 2019 @09:50AM (#20910547)
The root of the problem is that too many programs require a tty.
Re:Most programmers don't know what Expect can do for them by rubicks on Sunday September 08, 2019 @01:08PM (#20911762)
This. Why do programs do this when they don't have to? I'm genuinely asking here; I'm trying to understand the thinking that motivates this kind of decision.
Re:Most programmers don't know what Expect can do for them by DictumMortuum on Sunday September 08, 2019 @07:45AM (#20909943)
Working in a quite popular b2b betting company. TcL is used for much of the backend and development is still active today. Expect is used , too.
Re:Most programmers don't know what Expect can do for them by based2 on Sunday September 08, 2019 @04:35AM (#20909289)
Re:Most programmers don't know what Expect can do for them by jolmg on Monday September 09, 2019 @01:26PM (#20920219)
I used expect once some years ago, but changed to using ruby's #expect method[1]. I think the reason I changed was because, in TCL, some functions seem to keep track of stuff by setting variables in the caller's scope. So, if I took some code I wrote and defined a new function with it, substituting the code with a call to the new function, stuff would break because calls to predefined expect functions wouldn't have the same caller's scope anymore.

I wish I had a concrete example to share so someone here could confirm, but like I said, it was some years ago.

[1] https://ruby-doc.org/stdlib-2.6.4/libdoc/pty/rdoc/IO.html#M0...

Re:Most programmers don't know what Expect can do for them by gnachman on Sunday September 08, 2019 @02:08AM (#20908887)
I created coprocesses in iTerm2 because I missed Expect but I also hated Expect because it had a very expect-y way of doing things that never felt comfortable (though I shan't say a nasty word about Tcl!) Also, what mzs said about "now get out of the way".
Re:Most programmers don't know what Expect can do for them by INTPenis on Sunday September 08, 2019 @08:54AM (#20910275)
I've used Expect a couple times in the last 8 years. First to make backups of Cisco configurations into version control. And second by Ansible (indirectly it's used to read the sudo prompt).

So more people use expect than know it.

The issue I feel is that it's flimsy to wait for some expected output. I mean, it's even in the name! Especially when working with unknown APIs like Cisco IOS CLI.

So whenever I've done manual expect coding I've always felt very much out of my element and out of control. I am a control freak btw.

It works while it works but as soon as something unexpected happens it all goes to sh*t.

Re:Most programmers don't know what Expect can do for them by denton-scratch on Sunday September 08, 2019 @07:14AM (#20909813)
I tried to use BATS (Bash Automated Testing System) with Expect and Tcl to test a series of interactive scripts. It turned out to be something of a rabbit-hole. I didn't spend long enough on the task to get really good at it, and the result was tests that were hard to read (i.e. not what you want). Perhaps there was a 'wrong tool for the job' factor at play.
Re:Most programmers don't know what Expect can do for them by noisy_boy on Sunday September 08, 2019 @02:38AM (#20908972)
I need to connect to various machines throughout the day and as a first step, need to sudo to a service account. Expect has been a godsend to automate the interaction/input my credential (I know that it is far from being the most secure thing but the convenience of typing an alias and being connected including sudo is significant).
Re:Most programmers don't know what Expect can do for them by falsedan on Sunday September 08, 2019 @04:02AM (#20909192)
The test suite for the grody old IRC bot a few jobs ago was written in Expect. One intern season we set a project to, when announcing a new PR was published, to format the primary reviewer’s handle slightly. It ended up taking 3 days to update the tests, understanding expect, then writing a filter to strip out IRC color codes.
Re:Most programmers don't know what Expect can do for them by elldoubleyew on Sunday September 08, 2019 @03:01AM (#20909031)
I use expect to write little "drivers" for serial devices all of the time. I don't have a lot of experience with hardware and often its the easiest way to set up an interface. I never really knew how much it could do until one of my coworkers recommended it to me for this purpose one day.
Re:Most programmers don't know what Expect can do for them by sus_007 on Sunday September 08, 2019 @02:16AM (#20908906)
TIL, Tcl is pronounced as Tickle, but not Tee-see-el.
Re:Most programmers don't know what Expect can do for them by bch on Sunday September 08, 2019 @02:23AM (#20908926)
Either is fine and used. Joke is “tickle” among the initiated/familiar, tee cee ell if you’re talking to management. However, never name a file test.tcl, regardless of how you pronounce it.
Re:Most programmers don't know what Expect can do for them by sus_007 on Sunday September 08, 2019 @02:28AM (#20908944)
I'm a total stranger to the Tcl universe, but I'm curious what does happen if a programmer names a file "test" ? Some kind of conflict, bug, error ?
Re:Most programmers don't know what Expect can do for them by daseiner1 on Sunday September 08, 2019 @02:32AM (#20908953)
“testicle”
Re:Most programmers don't know what Expect can do for them by taneq on Sunday September 08, 2019 @04:43AM (#20909318)
It’s nut really a problem.
[dead]
Re:Most programmers don't know what Expect can do for them by _ph_ on Sunday September 08, 2019 @04:23AM (#20909254)
Your post is so wrong it warrants a long reply.

First of all, tcl might be anything, but certainly is not dead. The name tcl stands for "tool control language", which means, it was developed as a language to script tools, that is applications. So it is less a language to develop applications from scratch (but it is often enough used for that too), but to script applications. And it is an excellent language to do that, as due to its syntax - which rightfully seems to be odd - but is excellent for this task. If you write a command extension "do_something" for you application in tcl, you can just call it as:

do_something foo bar baz 42

No quoting or syntax required, unless your parameters include spaces or make references to variables or should be evaluated in any other sense.

Ironically the "tools", tcl was designed to control are the eda desing tools which are used to design e.g. cpus. So this conversation wouldn't happen without tcl being involved.

Beyond that value, tcl comes with a bunch of excellent tooling, which makes it worth knowing. The here described expect is one of them. Another very fameous tool is Tk. Tcl/tk was the first true cross platform solution for producing GUIs. And till today is probably still the easiest solution for that task. Also, it still is the standard GUI toolkit shipping with e.g. python. And has bindings from many other languages, ranging from Perl to Common Lisp.

Never mind, that tcl had event based IO long before node.js made that concept popular. It is great for writing small servers. It had utf8-support for a long time, and even has a robust mechanism for multi-threading. And many more cool features.

So, not only tcl isn't dead, but it is really a tool I can only strongly recomment to anyone to learn about.

Re:Most programmers don't know what Expect can do for them by loulouxiv on Sunday September 08, 2019 @06:30AM (#20909650)
Tcl seems to be an interesting language to learn, but I am not sure what kind of toy project I could do with it. Is there any links to modern libraries, blogs or websites discussing its use you could share with us ?
Re:Most programmers don't know what Expect can do for them by _ph_ on Monday September 09, 2019 @05:30AM (#20916569)
Not sure what qualifies for a modern library. But you will find out that there are libraries for tcl for basically any purpose you might need. The most fameous one certainly is the tk library for creating GUIs. So doing a small gui might be a nice start, or any shell utility. The asynchroneus IO functions are great for creating small internet based servers and utilities. The most fundamental resource on tcl is probably the book by the creator, its worth a read: https://www.amazon.com/Tcl-Toolkit-2nd-John-Ousterhout/dp/03...

It not only covers the language and its usage but goes into depth of both extending tcl in C as well as embedding it into an application for scripting it.

Re:Most programmers don't know what Expect can do for them by blacksqr on Sunday September 08, 2019 @11:33AM (#20911125)
The Tcler's wiki is a frequently-updated site with over 20 years of accumulated information and code respecting Tcl/Tk.

https://wiki.tcl-lang.org/welcome

Re:Most programmers don't know what Expect can do for them by ptsneves on Sunday September 08, 2019 @07:50AM (#20909967)
There are plenty of resources but probably in book form. My first Linux book had an intro to Tcl/tk. I disagree with the op that tcl/tk is not dead. Yeah it still exists but it is rarely used and younger systems programmers/Admins never had to use it. It was a bit of a stop gap solution to create GUIs when there was rarely any. The expect functionality though it completely replaced by pexpect so it makes little sense to learn tcl/tk for such a use.
Re:Most programmers don't know what Expect can do for them by blacksqr on Sunday September 08, 2019 @11:18AM (#20911037)
Re:Most programmers don't know what Expect can do for them by rowanG077 on Sunday September 08, 2019 @09:11AM (#20910367)
Doing anything with expect is basically only coding for the happy path. It's fine for some throw away script but as soon as you need any kind of reliability it's useless.
Re:Most programmers don't know what Expect can do for them by GhettoMaestro on Sunday September 08, 2019 @12:03PM (#20911321)
Have you ever stopped and asked yourself: What is a conditional statement?
Re:Most programmers don't know what Expect can do for them by rowanG077 on Sunday September 08, 2019 @12:21PM (#20911446)
Have you ever asked yourself if an API can return an infinite amount of responses how many conditional you would need to handle that?

The output of a terminal is just simply too unstructured to work in any high reliability environment.

Re:Most programmers don't know what Expect can do for them by GhettoMaestro on Sunday September 08, 2019 @05:01PM (#20913389)
I would agree that using a pure CLI (Unix Shell) requires significant operator skill to not accidentally blow things up. But that is also why you write scripts, test them, etc.

Sitting at a root shell executing untested or uncertain things is pretty high on the list of don't do these things.