The trouble is that for the repl “test” to be reproducible, you don’t just need to store the relevant commands and result, but the full state/history of the repl. Since the point of a repl is to explore and experiment, it seems like your tests would end up with a lot of noise in them... but perhaps there are ways to get around this? I agree the concept of quickly converting repl sessions to tests is very compelling.

A common approach would be that the code you're testing / playing with is defined in a file somewhere, and reloading that file resets the REPL state sufficiently to iterate on your tests. e.g. if it does rely on external state, the reload will re-initialize that state. Morally similar to the setup instructions in a unit test.

yes, there's a lot to it!

I store the complete repl history, complete with results and session metadata, in a database.

Pruning a session's history down to only what's needed is a manual process, true, and laying the groundwork to provide that feature has been the main yak shave.

> ...manual process, true

Before I get to my real comment, I guess a fair criticism of it would be “Well, where in the [development process] hierarchy do you actually want to live?”, which I guess is a longer discussion, but as a developer, you can’t become battle-hardened without going to battle. So we can nod to Carl Sagan[0] and say “I made this from scratch”, and know there’s value in that. At least occasionally running your REPLs by hand and taking the actual trip, instead of just wanting the destination. It may not be tenable as a full-time production development strategy, but that doesn’t mean it’s worthless.

[0] https://youtu.be/7s664NsLeFM

For what it’s worth, here’s a very simplistic/comparatively incomplete attempt at something like that from Cognitect (the Clojure/Datomic folks) a while back:

https://github.com/cognitect-labs/transcriptor

What language is your REPL for? Can you inject a repl into a piece of production code, like.

    import IPython
    IPython.embed()

I have used similar techniques to inject unit tests into the same pass as the integration tests. Basically at this point in the program, pause and run a bunch of unit tests with the internal state setup by the integration test. Requires less mocking, can differentiate between an integration and unit test failure.

It's in LuaJIT, which lets me use first-class environments to separate the client-server relationship while running everything in a single Lua_state.

The (long) roadmap involves implementing the Jupyter protocol, adding a client library which can be injected into production code, and eventually using that common basis to extend it to the other languages supported by Jupyter, and any others which choose to do the groundwork.

Bringing in all of ZeroMQ is a lot of work I've been putting off, but I need it all over the place.

+1 for Jupyter protocol.

Like Elixir's Doctests?

https://elixir-lang.org/getting-started/mix-otp/docs-tests-a...

Python has the same thing. I like them.

https://docs.python.org/3/library/doctest.html

And Rust: https://doc.rust-lang.org/rustdoc/documentation-tests.html

That's exactly the sort of test you get with the Selenium browser plugin: click "Record", do some things, verify the end state, click "Stop". You can then replay what is effectively an acceptance test. The problem (and it's a doozy) is that these tests are about as brittle as it gets: change basically anything in the markup and the test will fail. A good unit test will only ever touch a tiny area of the system, and therefore has a correspondingly tiny chance of failing whenever the system changes. And when it does fail because of changes to the rest of the system, it's usually for a good reason, like a changed argument list, not a bad reason like moving a button.

In short, these sorts of tests are super easy to create but become a massive friction at every level of the application since they depend on so much.

Sounds interesting.

When I work with Ruby, I often set things up with a script that contacts a method that I can call to trigger reloading of everything and a "replay all known good steps" while storing some state in globals. This is fairly easy in Ruby as you can redefine classes at will. It takes a bit more effort if you want to replicate the state more perfectly, e.g. by deleting methods that have disappeared etc., but that's rarely needed.

I then use "pry", and fire things up while I keep my code in an editor. I can test things in the pry repl, and when I find something that makes sense I add it as an additional method to my code. I usually cut and paste and keep the editor separate, but "pry" does allow triggering your preferred editor directly from the repl and also supports its own code reloading -- I prefer my own mechanism because it gives me more direct control, though.

Regularly I'll exit and run from a fresh state just to make sure I've not accidentally messed up anything with the reloads, but really that's mostly paranoia.

It's not a replacement for tests, but it tends to produce relatively easily testable code because it encourages code that allows you to quickly and painlessly execute steps manually, and so you quickly will e.g. encapsulate mocking steps and setup in suitable helpers etc. because it gets tedious to write them manually.

I like that a whole lot better than TDD, because TDD tests to drive (over)thinking things first vs. experimentation, I feel, because you invest too much in the test cases upfront.

The test cases are important, but it is very easy to end up with a design that is optimized for the tests rather than for your real world use cases. It doesn't have to be that way, and some people manage to do TDD without getting caught up in that, but for me at least, I've found it's much easier to be disciplined about making the code easy to execute in general when I work with it in a REPL on a regular basis, and build tests around that.

The big advantage of TDD is: it guarantees you will write tests. Can't skip tests if you write the tests first.

The big disadvantage, which is why I don't use it, is that you pre-commit to the "correct" function signature and response to some number of parameters. Optimized for the test, as you say.

When I'm working with a repl, it's easy to mutate and experiment with those things until I'm satisfied with them.

But then you still have to translate that into tests, or you get regressions. I'm lazy; I want to skip that part.

Cool.

I actually want this for both code and test development. For me the REPL is the first place that sees an implementation- multiple implementations, weighing various design factors- which then gets copy pasted into a source file and then REPL state reloaded (the Clojure "reloaded" workflow), and then the capability offered by the updated implementation exercised with test-like code in the REPL- which then as you describe should magically get into the test suite source for reuse.

Good luck. What language are you doing this work in?

Thanks! Answered this in another reply.

This sounds extremely useful and touches on one of the big frustrations I have with "REPL driven development".

Say what you will about TDD, but it does (often? sometimes? depends widely on how the tests are being written) produce useful, informative and functional artifacts. If you need to know how to create some entity or how to run some process, chances are there are tests that you can refer to that will show you how things are intended to work. This shouldn't take the place of comprehensive documentation, but it's a welcome counterpart.

With REPL driven development, there's nothing of the sort. If the developer has not provided documentation, you'll probably have to try to reverse engineer a given workflow based on the state of the system, which can be very problematic.

Yep!

Now imagine that you can instantly load a REPL session with complete tests, already executed, hanging out in the session history.

Now you can change anything in the test and see what it does. That's the idea anyway.

> With REPL driven development, there's nothing of the sort.

Wouldn't there be production code that you can read? You're already working in the codebase after all.

I agree that a unit test might have a cleaner, distilled setup procedure though.

In the specific case I have in mind, there was a business process that involved multiple pieces of the application being used to accomplish a task (creating database entries, etc.). Prior to my inheriting the project, this had been done in an ad hoc manner using the REPL and was not documented. When I had to try to run this process myself, I had nothing to go on.

This is absolutely the result of poor practices and, documentation aside, if the developers had been using tests to drive out this behavior, I'd have had a sort of playbook that I could reference.

That sounds horrible! But this is an operational failure, not a development one. (Well, the one clear development gap is that nobody thought to write a script to replace all those REPL commands. There's no reason that couldn't have been done, that's not a REPL issue at all.)

It's not uncommon for REPL users to write tests for their code. And nobody worth their salt would think that the REPL is a license not to document and operationalize their work properly! Your root cause seems to be some mix of developer laziness or inexperience, and poor business practice -- not really a tooling issue.

This sounds like approval testing. http://approvaltests.sourceforge.net/

Yep, that looks like the IDE-oriented version of the same thing.