Any bit of JSON either resolves to a valid object of some type in your data model, or is erroneous. "Simple and elegant traversal code" often makes assumptions about the data it operates over; when handed JSON that violates these assumptions, the result is undefined. If you use the "parse, don't validate" pattern (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...), by walking the complete data structure early on and ensuring via the type system that the assumptions hold, or failing, your JSON handling code is much clearer and more robust.

Static types are an unmitigated win, if you know how to use them.

Makes me think a bit of Python's "ask for forgiveness not permission".

I have a few problems with "validate then do something if its valid". On top of what that article mentions, there are concurrency issues (e.g. checking to see if a file exists, then if it does, open it... oops, race condition), and general ergonomics (if I always have to do something before I do something else, there should be a function call that effectively does both).

Wouldn't work if you were implementing a library like Pandas when the data can be almost any tabular structure that you need to clean and wrangle into shape before you can be sure what types you're dealing with.

Type systems give you a way to specify what you know at design time. If "has any tabular structure" is all you know then that is what you specify.

I don't think "but my data is dynamic" is a good argument for or against dynamic typing. It's an entirely separate issue.

> but my data is dynamic

In my experience, most dynamically types code makes assumptions about structure that may not hold later (better have tests!) and when it breaks some kind of validation is needed anyway to select between the correct version of the different dynamic options. In static languages this is simply typically done sooner, since the type system requires it to parse the data. (Or it’s in variants that are accessed without checking that the accesses are valid, sigh)

The type system depends on the kind of data you want to keep around in your columns, which could be a mix of types, and it could be all strings and NaNs, for example. This is common with DataFrames. There's a huge number of possible combinations of data you could be reading in that you need to clean up, reshape and then analyze. And you might be making decisions on how to do that in the middle of your wrangling. It's just like what you can put in Excel or a CSV.

I know. I've been doing this sort of thing for most of my adult life.

You have to be sure, like always. You can't write the code dealing with the data you have no clue about.

The type reflects invariants, constraints given for you data. The less you know, the more general type your data would have. Dynamic typing gives no advantages here safe for providing a super-general ANY type.

In case of statically typed language you would need to parse your data and construct an object representing that data just like in case of dynamically typed one. But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication.

> But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication.

The point of a library like Pandas is that you will be cleaning and reshaping your data until it's in a format that can be properly analyzed. And that data can be very dirty and very large.

But if you prefer to clean it up in Excel or with some bash scripts, or custom written functions in PL of choice, then go for it. You still have to deal with the cleaning and wrangling. The point of mentioning Pandas is that you don't always have a guarantee on properly structured data where you can predefine the types and read it in. And you don't always have control over how that data comes to you.

All data has SOME structural metainformation that can be captured in a type. If you are importing a CSV file, you know -- at a minimum -- that it's a CSV and can be parsed as a set of tuples of variants (or else there's something really wrong with the input and an exception continuation should be invoked to handle it). From there you can do analysis on the data to further refine its type.

But ALL data is typed, and encoding its type early in the code that processes it makes your assumptions clear and compiler-enforceable.

You're right that a typed approach to handling arbitrary input data is not right for a library like Pandas. It's a fundamentally different approach than the one Pandas takes, but it's no less valid and brings with it certain advantages.

For the library, you are right, it has to support any tabular structure — but that’s what even ricks are for, to encode that there are parts that you don’t know until use.

When using the library, though, there is some level of assumed structure (otherwise your code wouldn’t work since it would be accessing fields that don’t exist. Sure data is messy in real life, so you encode that messy-ness into your structure as variants or optional that you then check, and ideally are forced to check so that you can’t silently let errors slip in).

This works in statically typed languages right now using generics, templates, types that aren’t inferred until the library is used, or whatever the specific language has to deal with this.

I don't understand your JSON example.

If your Clojure code expects an API response's json.{id,username} to exist, and to not exist is a hard violation of your assumptions and thus your code, then how is that any different from formalizing it into:

    struct UserResponse {
        id: Int
        username: String
    }

?

(Damn, rest of the comment is pretty long but let me try to both sympathize and present a solution that I do like).

What I will say is that most statically-typed languages I've used do get annoying when there's no 1:1 mapping between your struct and the JSON body. I've dealt with some hairy JSON, like APIs that accidentally re-encode nested JSON strings so they become like triple escaped. Or values that can have like 5 different shapes so you really just want to try 5 different decoders until one is successful. (These cases are trivial in my upcoming code example)

For example, Rust's serde and Swift's Codeable are both pretty annoying if you want to apply arbitrary transformations between JSON response -> your canonical representation. Serde, for example, has you implementing the whole trait + visitor if you need to do anything the cute decorator DSL doesn't support. Or you implement a bunch of envelope structs even though you just want to pull out the value at "obj.a.b.c". Or, what happens when you can't simply use a codec from https://app.quicktype.io/?

JSON decoder combinators (e.g. in Elm) make JSON pleasant again while retaining static typing. https://package.elm-lang.org/packages/elm/json/latest/Json-D...

For example,

    data Role = Member | Admin

    data User {
        id: Int
        username: String
        role: Role
    }

    -- API example body: { id: "12", role: "a", info: { uname: "foo" } }
    decodeUser : D.Decoder<User>
    decodeUser =
        User
            -- Crappy API gives us either an int or a string
            (D.field "id" (D.oneOf [ D.int, D.string |> D.map parseInt ]))
            -- Pull out value from nested object
            (D.fieldAt [ "info", "uname" ] D.string)
            -- Arbitrarily transform value and create our own error
            (D.field "role" D.string
                (D.andThen (\value -> 
                    case value of 
                        "m" -> Ok Role.Member
                        "a" -> Ok Role.Admin
                        unknown -> Err ("Unexpected role " ++ unknown))))

That almost compiles despite shooting from the hip directly into this textarea, but the point is that it's a pleasant way to decode JSON while allowing arbitrary transformation, and I think it's the ultimate middleground between static-typing and the sort of on-the-fly decision making you get with dynamic typing.

Your example isn’t tied to static typing, it is simply but behind-the-scenes checking types at runtime. You can do that in any good dynamically typed language (e.g Common Lisp) and hide it in a similar way.

Closure has definite problems that other Lisps don’t due to a combination of leaky JVM abstractions (incomprehensible stack traces, JAVA pollution), its semantics not really being Lisp semantics and its premature optimization into one paradigm that doesn’t really fit the language or most problems very well.

Yes but once the json has been validated once at runtime, you know it's safe for the rest of your program to use.

Whereas in a dynamic language (even an immutable one), your data could be mutated during program runtime and no longer conform to the type you expected. Yes you could re-validate it everywhere, but that requires you to exercise every code path when testing at runtime to find bugs.

I think the general argument that "You can't static type everything (like your API dependencies), so there's no point in doing it all" isn't a very solid one.

Your data can't be mutated if you're using immutable data structures. Which is what Clojure defaults to, in fact.

I didn't mean literally mutating the same structure in place - but a new version of your data structure could be returned from a function which changes some of the keys/values in unsafe ways.

Sure, my example isn't tied to static-typing but static-typing is especially helpful any time you're workin with combinators/transformations at a distance. But also, statically-typed languages like the two I brought up tend to be in most need of a nice validation/codec/transformation library because you can't simply procrastinate establishing your domain models like you can in dynamically-typed languages. And this is the sort of pain that leads people to say that dynamically-typed languages are more accommodating here.

Though I will have to defend Clojure from your last paragraph. I don't think there has been a lisp that has done so much for lisps than Clojure. For example, until I used Clojure, other lisps were just like any other dynamically-typed, mutation-heavy language. The idea of just shuttling around data and transformations on that data is something that fundamentally changed my approach to writing software.

I can certainly come up with worse things to say about Clojure. I kinda grew out of dynamic-typing, for one. And this is never more obvious than if you've ever read a large or complex Clojure codebase while dying to just know what the underlying Java types are. And you're so desperate that you get excited when you see a single ^ChannelHandler type-hint.