Tests are a form of documentation, but too much in the code, like too many comments, obscures. Higher level unit tests (acceptance tests) can be quite long, especially if there's a lot of setup - unlike their example code. Literate programming tried embedded documentation, but didn't catch on (even with Knuth's backing). Embedding tests makes them easier to keep in sync, but tests are already kept in sync by (hopefully) failing when not. However, their idea of automatically running tests is interesting, so there's no infrastructure to set-up, run etc (though you'd want to be able to disable them).

Nice for teaching.

Literate programming is not "embedding documentation". The main idea was to separate the order in which the code is read from the order in which the compiler sees it, and it embeds the code in the documentation, not vice versa. It was a very idiosyncratic thing, hard to imagine a team of programmers in a typical current commercial setting, developing a nice LaTeX essay around the actual code of the next social network web-app. As the program grows larger it also gets harder and harder to maintain the "story" around it.

    > Let us change our traditional attitude to the construction of programs: 
    > Instead of imagining that our main task is to instruct a computer what 
    > to do, let us concentrate rather on explaining to human beings what we 
    > want a computer to do.
    >                          — Knuth

Hopefully, it's clear how a program written in such a way could potentially be beneficial to a large team, trying to understand, modify, and improve it.

However, the `where` tests play into the type-inference story for the language!

Instead of writing tests in strings, the documentation tool should have been modified to render that code _in_ the documentation.

    def test_foo():
        assert False, "test code for foo"

    def foo(farb):
        pass

Code folding in a IDE goes a long way to make this bearable.

Code folding in IDEs is an indication that we are doing it wrong -- that is, we human beings are programming computers "wrong."

I'm not saying that code folding is a symptom. I'm saying that code folding shows how primitive our means of managing code is. It's as if the mesopotamians somehow invented computers, and because of tradition, all code has to be written as cuneiform on wet clay tablets, then fired in ovens before being read. At least text files in directories are digital, but they are as static and behavior-less as clay tablets, and all of the important relationships therein are expressed as implicit correspondences which programmers have to keep track of in their heads.

Code Bubbles is a beacon in the direction we should go.

https://www.youtube.com/watch?v=PsPX0nElJ0k

But we also had to make a conscious decision about IDEs, and we decided to be IDE agnostic -- IDEs should enhance the language but not be the only means of working with it, because programmers are really attached to their editors, and a language predicated on tearing people away from their editors will stuggle. Also, having been part of DrRacket from version 0.0, I know how long it takes to make one that's any good.

I wrote a significant fraction of a Smalltalk class browser.

IDEs should enhance the language but not be the only means of working with it, because programmers are really attached to their editors, and a language predicated on tearing people away from their editors will struggle.

So, basically you made the pragmatic decision. What you say is true, but I suspect this tendency is ultimately holding us back.

If I could get a reliable, efficient, cross-platform, extensible, quickly-ported-to-new-platforms environment that gave me rich editing, I'd grab it. I don't know of such a platform. So we made the "pragmatic" decision, yes. That's because this isn't the problem we're trying to solve.

We are actually trying to innovate in the programming environment space, and have been running with that experiment for the past semester internally at Brown. At some point, after it's been knocked around a bit more, we'll make it public. So we're not avoiding the environment space. But this is not a battle we're seeking to pick in that space.

I wish you luck with it, and hope we can build on the work of people like you.

I would say the more potent point is that it should not matter at all if the tests are in the function or in a different directory: the IDE should be able to fold them in either way, and in some sense, IDEs already do have partial support for this.

I said it was a "beacon in the direction we want to go," not a lighthouse at the destination!

> the point being that the underlying representation does not particularly matter. Having underlying text, however, means you have a good fallback in case your tools do fail.

Sure, do this, so long as the representation doesn't limit the objective capabilities of the environment and so long as it doesn't limit the ways tool makers think about code. Unfortunately, our current representations clearly do both.

I am in favor of giving links between data a more prominent place in our storage systems. I envision a system where the data is mostly fine-grained trees, like sexprs, where hard-links between trees are first-class entities. But at the end of the day it's just an abstraction over a bunch of bytes.

I know this is false. You can have a system where everything is an object and carries behavior around with it. We've had those for over 40 years.

> I am in favor of giving links between data a more prominent place in our storage systems.

A "more prominent place?" Isn't this a bit bass-ackwards? Aren't relationships in code where the primary value is?

Code folding in an IDE is an example of a way in which text files aren't necessarily static and behavior-less. I see no reason to move away from an underlying representation as text.

A big reason is that otherwise people won't move away from the paradigm of static text files. The best they'll do is text files with little gimmicks attached to them.

I would argue literate programming is experiencing a renaissance, thanks to docco:

https://github.com/jashkenas/docco

http://ashkenas.com/literate-coffeescript/

There also seems to be some traction for literate Haskell:

http://www.haskell.org/haskellwiki/Literate_programming

I experimented a bit with literate programming on an introductory course in programming using java. It's an interesting experience -- it's very easy to produce a very readable language that is still pretty poor java code, as it becomes so easy to split out fragments and "procedures", rather than follow the more common java class-oriented object orientated way of doing things.

http://qznc.github.io/d-tut/testing.html

And as other people mentioned, you can always use the check statement out of band, too.

Modern IDEs/code editors can support code folding as well as projections fairly easily. I don't see a problem.

Of course it's not equivalent, but Python doctest module at least allows you to keep tests close to the code. Here's Pyret example, ported to doctest:

  def sum(l):
      """
      >>> sum([])
      0
      >>> sum([1, 2, 3])
      6
      """
      return reduce(lambda x, y: x+y, l, 0)

  if __name__ == "__main__":
      import doctest; doctest.testmod()

It's been tried many times before and it has always failed. A few reasons:

- It clutters the code

- If you need simple tests, asserts suffice

- If you need more sophisticated tests, write functional tests, separately

The approach offered by Pyret (along with similar ones, such as design by contract) are compromises that are the worst of both worlds.

Nevertheless, perhaps you could provide some pointers to the "many" projects that have tried it before?

Assertions are not tests. This is a fundamental misunderstanding of the difference.

Pyret's `check` blocks exist to write complex tests, separate tests from definitions, or write tests that cross multiple functional units. So they offer the power of regular testing frameworks.

Thus, `where` is the bonus, not a "compromise".

Beyond that, I'd agree completely. Doctests, for instance, are quite terrible when you can't fit all of the logic into one line, but then I think it's usually a sign of a well-designed function when it's good a good enough interface to get a few pithy doctests in.

(Another virtue of test blocks: you can write little local helper functions. There's an example of this on the Pyret home page.)

    -- | Documentation goes here
    --
    -- > 2 + 2 == 4
    -- True
    --
    -- prop> \lst -> reverse (reverse lst) == lst

Hmm...Jane Street?

That said, I'm going to take the opportunity to ask a question: does Pyret's gradual typing cover parametric polymorphism and mutable data structures? Those usually are specially though to do in these fully gradual systems so I'm curious what you guys have done about that.

Re. your other question, yes, this is a wide-open problem, and we're trying to not get bogged down in research issues like this. We actually have one answer about what to do, presented in a paper we wrote some time ago (http://cs.brown.edu/~sk/Publications/Papers/Published/gmfk-r...), but we don't really want to do that. But because we're working on a (curious) type inference story, we may be able to carve up this problem in a very different way.

Sorry for the non-answer. You're right, and we're working on it.

In fact, we'd really love it if gradual typing folks would tell us what set of primitives they miss in a language. They aren't constrained to what Racket or Python or whatever; there may be a primitive that makes sense that Pyret can add to enable space-efficient, semantically-meaningful contracting or gradual typing in the presence of such operations.

Though I'm not enamored of the Typed Racket style of wrapping everything at the boundary (even as I appreciate why it's there). Not even hypothetical: we actually began developing Pyret in TR but, with great sadness, ended up having to move it out because of the enormous performance hits.

And yes, soundness + interoperation + structural types = heavy performance cost. You can give up on one of those easily, but the combination can't be free.

But the best way to make TR programs not pay that cost is to not cross the boundary all the time, which it sounds like is what you were doing. Why was that necessary?

http://en.wikipedia.org/wiki/Eiffel_(programming_language)#D...

Perhaps you think I was referring to something other than the quote in the comment I replied to.

That's why Pyret has both tests and contracts (for now, in the form of refinements). This is a distinction that has been explored extensively in Racket also, all fully aware of Eiffel DBC.

Also, maybe you should have called them something else since a lot of people will think about completely different when they hear this word... (http://www.ruby-doc.org/core-2.0.0/doc/syntax/refinements_rd...)

Rebol got there first though! - http://www.rebol.com/r3/docs/datatypes/refinement.html

Matz is known to like Rebol - https://twitter.com/matz_translated/status/25061436079433318...

There is also the whole dynamic typing thing. Eiffel is still fully statically typed.

In terms of my opinion of Racket/Pyret, though, there's been a number of Ruby projects to do similar stuff to this over the years, since it's trivial to add (just add a class method that redefines a method and wraps it in code to execute the contracts), and a lot of Ruby newbies try to find ways of adding back the restrictions they've lost when moving from statically typed languages.

Invariably they've ended up dying or not getting much interest, largely because a large part of the appeal of dynamic languages for a lot of people is that the code reads simple. In the Ruby world at least, aesthetics is a big deal. And putting the tests inline destroys that aesthetic and makes it harder to read the code.

Especially because tests tends to be trite and repetitive and contain a lot of stuff that is obvious to a human when you have the code right in front of you.

Especially when they go beyond documenting contracts that are intended to bind subclasses to an interface.

That code readability is important.

I have a prototype for my own Eiffel inspired language with DbC support, and my own experience with it was that as much as I love the idea, even Eiffel level contracts are bordering on being too verbose, for these reasons.

Pull them out into separate files, and they're a lot more acceptable, but then what is the difference from any other unit testing framework?

The same goes for making statements about programs.

Therefore, in Pyret, we recognize that there are lots of different levels at which we make statements about our programs: from tests through types to specifications, spanning both static and dynamic aspects. They not only aren't in conflict, they're even related. Most languages include only a subset of them or make them appear to be disjoint. In contrast, in Pyret, we're already working on linking these to one another, and are going to keep pushing in this direction.

Languages that leave out one or more of these end up forcing users to reinvent these wheels for themselves (because they're each good ideas that people inevitably want). But by forcing people to reinvent a wheel, they end up giving users ad hoc versions of these tools, and also introduce friction between these parts. (E.g., as Robby Findler found several years ago, just about every single bolted-on DBC system had significant limitations or errors.)

[1] http://dlang.org/unittest.html

(I think about tests a lot: http://akkartik.name/post/tracing-tests)

That said, I couldn't see what your post had to do with this concept; if I've missed something, please explain. (As an aside, I believe this -- http://cs.brown.edu/~sk/Publications/Papers/Published/mcskr-... -- generalizes the idea in your tracing post.)

One of the advantages of having a separate block for tests is we can do things in that block that won't necessarily pervade the whole language where it might not make sense (which we already do: eg, the keywords "is" and "satisfies"). We've all done some sort of "here's a list of tuples, now map the prefix of this as arguments and the suffix as the expected value over this function", so it is something on our mind.

But I've struggled a bit to find a good way of writing it. We've written a few things where it looked truly "naked" to have these tuples without the surrounding function call. So it's as much a matter of syntax design as anything else.

I must say that the code on that page does not look like something I want to copy; it looks like a good compromise, but with emphasis on both words. We don't need to compromise.

  check:
    insert(a-value, a-balanced-tree) satisfies is-balanced
  end

  check:
    tree-examples = [ ... ]
    values = [ ... ]
    for each2(t in tree-examples, v in values):
      insert(t, v) satisfies is-rbtree
    end
  end

At a glance, the Pyret default appears to be that tests run silently. How do I know the right tests were run?

  (with-test
      (defn my-function [x y]
      (+ x y))
    (is (= 4 (my-function 2 2)))
    (is (= 7 (my-function 3 4))))

What?! You can't write macros in Python? (right? have I missed something?)

That said, Pyret appears to be aimed at education so this may make more sense.

Put the tests and docs in with the code and you end up going into "scanning" mode. The actionable information density per character is lower.

Put the tests and docs outside the code and you (or that obnoxious guy who just changed > to >=) don't update the tests or the docs. In the worst case (cough functional web tests) people tend to disable the tests because it's too much work to get them to conform to the changes and everyone is on fire and babies are dying and this needs to be done in production now.

Does anyone have any experience with workflows like this? Are tests next to code a pragmatic technique?

Doctests: http://docs.python.org/2/library/doctest.html

https://github.com/sol/doctest-haskell#readme