One of the things I liked about Lisp when I first saw it was that, since it uses kebab-case and (usually) full words, I could just turn on my standard English spell-checker. After all, if "Programs must be written for people to read, and only incidentally for machines to execute", then it makes sense that I should be able to run a normal spell-checker on them. The only reason that was infeasible in languages like C is because they used (for both technical and cultural reasons) names like "memcpy".
Whatever field I'm working in already has its own vocabulary. I don't need to create a new <domain>-in-<proglang> vocabulary for a program about it. Or if I do invent an abbreviation that's so useful that I want to use it in my source code, I'll probably also want to use it in my documentation, webpage, emails, etc.
Most large programs that I've worked on, in other languages, have at least a couple misspellings in func/var names, which even Haskell's type system won't catch. "Referer" was clearly not written by someone with their spell-checker turned on for source code!
OTOH, when I'm doing more "building infrastructure" type work - implementing a data store, writing a compiler or interpreter, stuff like that - I start getting more interested in rigidity and formality. Static languages treat me well in these situations. The more rigid, the better - I'll prefer Scala to Java, for example, specifically because it gives me more tools in the type safety department.
I recently rolled my own (verg tiny and specific) build/deployment system for a project. Initial prototyping was a breeze, but once I had things more or less nailed down, I started adding types and, for example, was able to leverage the type system to make sure it can only be deployed listening on IP addresses in the private range. The next step is to rig up some sort of Zerotier integration to ensure I only make it available on my private networks.
The thing I've _consistently_ seen with every large Clojure project I've been a part of is that as the project grows, it becomes harder to make sense of the types used across the project. On the last couple of these projects I've been on, Schema was used to annotate many types, but I've never found this to be quite enough and I still remain firmly convinced that gradual typing just doesn't work well enough generally. I think that with a _very_ highly disciplined team of experienced developers it could probably work well. But we don't all get the opportunity to work on such teams. What I've personally seen is developers are all too willing to either add a partial type spec/annotation (with probably some s/Any's lazily thrown in where there should most definitely _not_ be), or just flat out not adding any at all ("Why would I need it here? This is very simple code and it's obvious what is happening here!" ... sure, maybe it is to you now... what about in 6 months? Or what about to one of the other developers on the team?).
When discussions about static vs dynamic typing come up (which is almost always a fruitless argument in my experience), I've noticed that the dynamic typing proponents tend to get overly focused on the idea of "correctness". This has always disappointed me. Correctness is certainly an important benefit (though I think it tends to get exaggerated a bit much), however, to me the idea of documentation in the form of a type definition is _much_ more valuable over the longer term, especially so for large projects. Developers are usually quite lazy, especially when it comes to documentation, and often when looking at a piece of code in a large code base, the only documentation about the values being passed to a function is going to be the type definitions (aside from reading the code itself of course).
Maintainable in what sense? And large in what sense?
I typically work on 1-10 million lines of Java code solutions split up into maybe 20-50 seperate components.
These 20-50 seperate components have untyped boundaries between them.
Setup correctly these untyped application boundaries rarely cause problems - of course you have to put a bit more thought into architecture, documentation, and verification but it’s not particularly onerous and we have been doing it successfully for decades now.
I’d say anecdotally errors caused by this lack of typing are well under 1% of total errors.
Are there languages you've seen that don't show an ugly side as projects grow to a large size?
Ex: Facebook and adding types with Hack & Flow. Dropbox's work to add types to python. Uber used to be a python and nodejs shop, and now new stuff is done in golang and java. Many nodejs shops go from pure js to typescript or flow.
Gradual typing is useful for companies that don't want to immediately migrate everything at once. If you have the right policy of 'type everything you touch', enforced with a linter, a codebase gets migrated relatively quickly.
Independent modules are easier to reason about, they're reusable, and they can be maintained on independent schedules. The biggest advantage though is that this approach allows you to split teams up into smaller groups responsible for the individual modules. One of the biggest problems in maintaining large projects is communication overhead, and typically it's very difficult for teams with over 5-6 people to be productive.
So, if your project outgrows limits of dynamic typing, chances are it should be split up instead of using static typing as a crutch.
Instead of that, it would be better to respond directly to the specific points the OP made, rather than move the argument to a bunch of new topics.
OP did not say anything about other languages, be it Java or otherwise. OP is just describing experience with closure.
In order to understand clojure better, it would be good to not shut out feedback.
In this sense Java is actually dynamically typed: there's no way of knowing at compile time whether an object is actually an instance of that object or is null. In languages like Kotlin, Haskell and Ocaml, which specify in the type system whether or not something can be none and force the user to check before using something that can be null, null pointer errors don't happen.
That is an extremely unusual definition of dynamic typing. If I have an Integer it might be null but it won’t be a BeanFactory (unless someone did a reflective call or wrote some bytecode or otherwise subverted the type checker).
I hope java gets native support for @nullable, but for now we have stuff like error prone and @nullable annotations.
Indeed one of the "joys" is that one can apply that verification a la carte, and possibly with different approaches [0][1]. Similarly, one may desire to dispatch on type [2], but might sometimes want arbitrary dispatch [3].
> After writing the 10,000th unit test that only exists to ensure that you spelled some function/method/hash-key correctly
This too is solvable with a la carte tooling [4].
Everything has trade-offs. Static type systems give a fixed set of benefits with a fixed set of costs. Some folks prefer to use their experience and judgement to choose when, where, and how to pay the associated costs (and that freedom of choice itself has a cost). It's okay to have different preferences.
[0] https://github.com/plumatic/schema
[1] https://clojure.org/guides/spec#_entity_maps
[2] https://clojure.org/reference/protocols
Compile time verification is nice if you know all your inputs at compile time. At some point, data from elsewhere comes in. Specs are an answer that preserve the composability goals of Clojure (who cares if I have a bit more information than I strictly need?) and allow you to validate them at runtime with useful error messages.
spec has incidentally made Clojure's compile term error messages better! But that's just because one program's compile time is the compiler's runtime, and the compiler is running spec verifications on your code (which is after all, just data!) during its runtime :-)
https://code.facebook.com/security/fighting-spam-with-haskel...
Lest you forget, both fight against Java: https://trends.google.com/trends/explore?date=today%205-y&ge...
Industry is not doing a 'mass migration' to clojure, haskell, or any other language not in the top 5.
So the OP was wrong to claim a 'mass migration' in the first place
It is an implementation of much larger messaging idea - message receiver only takes care about data it needs and nothing else.
so instead of
data Person = Person String String Int Float String String
you have :person/name ....
I’m not sure what these comments are arguing against, but it’s not compile-time analysis.
Seriously: you’re describing a feature that only exists in a small subset of languages and only a small subset of contexts at that. OCaml and Go do this but not for data, just for methods.
The language used in this thread so far has made it seem like runtime specs are the solution to this “only care about part of the bag” problem rather than what I consider a general feature of dynamic typing that certainly can be expressed statically as well.
Runtime specs absolutely also validate. And you can run them at test time. And you can put arbitrary code in them if you want. And sometimes you can statically validate them.
What I’m saying is you get ad hoc structural typing on everything, including data, which you’ve already said is a good thing, so I’m not sure what we’re arguing about or if we’re even arguing :)
EDIT: ah! You edited the paragraph. Simple question: where does this ideal static typing thing ship? Like, how do I do this on my machine right now? What package do I install?
People were talking about using runtime specs to annotate and validate the subset of information a function may care about. Which is certainly something I wish I could do in more languages, though ideally as much at compile time as possible.
Of course, runtime specs in Clojure let you make more assertions than just structure, so structural typing as seen in TypeScript, for example, certainly don’t go all the way. It just helps address what I thought was being discussed.
What is the ideal middleground? I think one possibility would be if Clojure was statically typed to begin with yet had the syntax of spec/schema.
Though I’m not sure we’re talking about the same thing anymore.
Starting with the fact that most dynamically typed languages today are progressively adding type checking (spec in Clojure, gradual typing in Groovy, etc...) while no statically typed languages are going the opposite way.
At the same time, industrially-popular static languages have added dynamic escape hatches. The trend is really toward optional static type checking, with dynamic languages moving to opt-in typechecking and static languages moving to opt-out.
You can use Apache Groovy for just the glue code and Gradle build scripts, and use a statically-typed language like Java, Kotlin, or Scala for the actual system you're building -- that's what most programmers do.
Of course, no-one actually thinks of C# as being a dynamically typed language despite its dynamic keyword, nor does anyone think of Apache Groovy as being a statically typed language despite its @CompileStatic annotation. In the everyday world, C# is used for building performant systems and Groovy is used for glue code and build scripts.
Languages tend to go both ways nowadays. Go first shipped with both static typing (though without generics except for builtin datatypes) and dynamic typing (though using the wordy interface{} marker).
Static languages added dynamic capabilities with reflection and run time code generation a long time ago.
Assuming, for the sake of argument, that that's true, it means that we've got to think about how you define "programmer". Does it mean anyone who uses a programming language, or does it mean something more like "software developer"?
TIOBE, for example, effectively measures popularity among the former group, not the latter. Which means that, in the case of Python popularity, it's going to be picking up on all sorts of things: analysts and BI people ditching Excel in favor of Pandas, ops people ditching piles of shell scripts in favor of Ansible, graduate students choosing Scipy over R or Matlab, 7th graders with Pi-tops, ...
My experience is that dynamic typing is problematic in imperative/OO languages. One problem is that the data is mutable, and you pass things around by reference. Even if you knew the shape of the data originally, there's no way to tell whether it's been changed elsewhere via side effects. The other problem is that OO encourages proliferation of types in your code. Keeping track of that quickly gets out of hand.
What I find to be of highest importance is the ability to reason about parts of the application in isolation, and types don't provide much help in that regard. When you have shared mutable state, it becomes impossible to track it in your head as application size grows. Knowing the types of the data does not reduce the complexity of understanding how different parts of the application affect its overall state.
Immutability plays a far bigger role than types in addressing this problem. Immutability as the default makes it natural to structure applications using independent components. This indirectly helps with the problem of tracking types in large applications as well. You don't need to track types across your entire application, and you're able to do local reasoning within the scope of each component. Meanwhile, you make bigger components by composing smaller ones together, and you only need to know the types at the level of composition which is the public API for the components.
[REPL driven development](http://blog.jayfields.com/2014/01/repl-driven-development.ht...) also plays a big role in the workflow. Any code I write, I evaluate in the REPL straight from the editor. The REPL has the full application state, so I have access to things like database connections, queues, etc. I can even connect to the REPL in production. So, say I'm writing a function to get some data from the database, I'll write the code, and run it to see exactly the shape of the data that I have. Then I might write a function to transform it, and so on. At each step I know exactly what my data is and what my code is doing.
Where I typically care about having a formalism is at component boundaries. Spec provides a much better way to do that than types. The main reason being that it focuses on ensuring semantic correctness. For example, consider a sort function. The types can tell me that I passed in a collection of a particular type and I got a collection of the same type back. However, what I really want to know is that the collection contains the same elements, and that they're in order. This is difficult to express using most type systems out there, while trivial to do using Spec.
It wasn't horrendous (and it's not like Scala is absolutely amazing for us), though maintenance was something of a pain point. We might use Clojure again, especially with some new blood who are Clojure enthusiasts.
These days I'm less and less certain of what proactive role a language plays in team success. That is to say I think the arrow of causation is usually pointed in the wrong direction. A well-functioning, excellent team will, given sufficient latitude, converge on a language that suits a team well. This language is often different for each team. Introducing a "good" language is unlikely to save a team.
This is a huge factor I'd like to emphasize and write about in a follow up post, having worked on many Clojure projects at different companies now... Another Year in Clojure :)
I've maintained some large Clojure codebases I didn't write, and the easy ones shared this quality. The more painful ones were reminiscent of large OOP codebases and honestly I would've preferred they'd been written in Java.
alternatively, I'm always receptive to reports about experiences.
I'm very skeptical about demands to produce "empirical evidence" for or against any sociological programming practice, because it tends to be a method to place the burden of proof on the opposing side, while reserving the default position for own own side - which just turns out to be a way to normalize one's own beliefs.
Furthermore, fixating on static typing as the one defining feature of a language amounts to concept space reduction. In practice, there are many complex factors that contribute to overall quality of the language that have to be considered holistically.
Programming is not my day job, so please correct me if I'm wrong - but every project I was involved in so far - having access to a production DB is a big no no.
Or does the project you refered to not contain any sensitive data, so having direct access to the production system is deemed 'safe'?
https://medium.com/wit-ai/open-sourcing-our-new-duckling-47f...
I'm curious how Clojure avoids this. In my large Clojure projects, there aren't "types" in the sense of static typing, but the shapes of data still exist, the types are there, but they are in my head or in other forms than that which a compiler can check. I still have to remember that a map is a certain "type" of data with certain keys/values that are also of certain types, and a large project might have many of these "types", even if they are not explicit. That is equally hard to maintain for me -- harder, really, since it's too easy to make mistakes.
I also find that you learn patterns for structuring code that make it easier to track the types. For example, I avoid renaming keys or restructuring data except at the edges of the application. Adding Spec or Schema at component boundaries is also helpful, as you can see what the shape of the data was at the API level and work your way through the helpers from there.
There's also a big benefit of working with data as opposed to objects. Each class is its own ad hoc DSL with custom behaviors. Knowing what one class does tells you nothing about what another does. This approach doesn't scale well for large applications where you have thousands of classes with each with its own behaviors.
All of a sudden you can stop worrying about "what am I getting in this function" and actually concentrate on the business logic.
(No, tests can't cover everything. Yes, we still have to write tests, but now they are all useful. Yes, the thing that types hinder you from doing changes is a lie, it's quite the opposite, much easier to change things since the compiler covers your ass)
I think what you're saying is something like "this function guarantees that the thing I get here conforms to this shape" and I do like that. Something I don't see a lot of people express is just the simple "I know what the shape of this thing is by looking at the signature". Whenever I find myself reading other people's Clojure code, it feels like I end up spending more time figuring out what is being passed into a function. I have to look around to find the origin of a value. I wanted to tear my hair out the first time I tried reading Ring's source code to figure out how it worked. With F#'s Suave, I can tell what each function gets and what can be done to it--it's self documenting. In Clojure, as the original author you can play around in the repl full steam ahead because the types are in your head at the moment. Those who come after you have to assemble the puzzle with less information.
utop # let avg a b = (a +. b) /. 2.0;;
val avg : float -> float -> float = <fun>
utop # avg 2 4;;
Error: This expression has type int but an expression was expected of type float
utop # avg 2.0 4.0;;
- : float = 3.
user=> (defn avg [a b] (/ (+ a b) 2))
#'user/avg
user=> (avg 2 4) 3
user=> (avg 2.0 4.0) 3.0
I think it is a matter of taste which one you prefer. With property based testing and specs you can go very far in term of pre-runtime checks and there was some data about the number of bugs per line published by Github where it turned out that Clojure did not do too bad in comparison to statically typed languages.
avg :: Num a => a -> a -> a
Where your "a" is a number (that is, a type that implements the Num typeclass)
This would work out of the box on floats, ints, etc
If you have operators that are typed like + and +. how can you make this work with Num?
λ> :t (+)
(+) :: Num a => a -> a -> a
λ> :t (/)
(/) :: Fractional a => a -> a -> a
import Data.List
average :: (Real a, Fractional b) => [a] -> b
average xs = realToFrac (sum xs) / genericLength xs
I have found too that when you are programming functionally as in Clojure you don't get bit by shared state things not being what you expected as everything you are writing should be data in, data out and easily reasoned upon. Nil handling in Ruby is a huge problem as people are chaining methods on mutable objects which becomes complicated, and this is the source of most bugs in Ruby. This is somewhat mitigated lately with safe navigation operator.
That is why TypeScript is appealing as you can use typing when you'd like to but still take in "unsafe" JavaScript or write in JS. Unlike if you were using Elm and had to write a wrapper to conform all JS code coming in.
Spec in Clojure is very nice as you can enforce and conform types where you need it, and also make generative testing for free. If there is some mass migration to OCaml, F# and Haskell I have yet to see it because very few businesses use them and the communities are much smaller.
For 10 years I've seen many developers who have only really used statically typed languages like C++ and Java, now finding the "joy" of dynamically typed languages and all the "freedom" they give you, only to realize within a year or two that, wait a second, verifying those types at compile-time was actually pretty useful. I've also seen mass migrations away from languages like Ruby, Python and Clojure towards OCaml, Haskell, and F#, because people really enjoyed the functional programming aspects but wanted their static typing back. After writing the 10,000th unit test that only exists to ensure that you spelled some function/method/hash-key correctly, it gets a little tiresome. Personally I've settled on TypeScript, which has nice IDE integration in VS Code so that it validates your types live while you code, while still giving you the flexibility of a dynamically typed language (it's all still JS at the end of the day).