News for hackers. Stuff that matters.
Re:State of Clojure 2020 by bitwize on Friday February 21, 2020 @01:16AM (#22381283)
Any bit of JSON either resolves to a valid object of some type in your data model, or is erroneous. "Simple and elegant traversal code" often makes assumptions about the data it operates over; when handed JSON that violates these assumptions, the result is undefined. If you use the "parse, don't validate" pattern (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...), by walking the complete data structure early on and ensuring via the type system that the assumptions hold, or failing, your JSON handling code is much clearer and more robust.

Static types are an unmitigated win, if you know how to use them.

Re:State of Clojure 2020 by bcrosby95 on Friday February 21, 2020 @01:53AM (#22381393)
Makes me think a bit of Python's "ask for forgiveness not permission".

I have a few problems with "validate then do something if its valid". On top of what that article mentions, there are concurrency issues (e.g. checking to see if a file exists, then if it does, open it... oops, race condition), and general ergonomics (if I always have to do something before I do something else, there should be a function call that effectively does both).

Re:State of Clojure 2020 by goatlover on Friday February 21, 2020 @04:25AM (#22381942)
Wouldn't work if you were implementing a library like Pandas when the data can be almost any tabular structure that you need to clean and wrangle into shape before you can be sure what types you're dealing with.
Re:State of Clojure 2020 by fauigerzigerk on Friday February 21, 2020 @04:47AM (#22382023)
Type systems give you a way to specify what you know at design time. If "has any tabular structure" is all you know then that is what you specify.

I don't think "but my data is dynamic" is a good argument for or against dynamic typing. It's an entirely separate issue.

Re:State of Clojure 2020 by dkersten on Friday February 21, 2020 @06:12AM (#22382335)
> but my data is dynamic

In my experience, most dynamically types code makes assumptions about structure that may not hold later (better have tests!) and when it breaks some kind of validation is needed anyway to select between the correct version of the different dynamic options. In static languages this is simply typically done sooner, since the type system requires it to parse the data. (Or it’s in variants that are accessed without checking that the accesses are valid, sigh)

Re:State of Clojure 2020 by goatlover on Friday February 21, 2020 @09:19AM (#22383243)
The type system depends on the kind of data you want to keep around in your columns, which could be a mix of types, and it could be all strings and NaNs, for example. This is common with DataFrames. There's a huge number of possible combinations of data you could be reading in that you need to clean up, reshape and then analyze. And you might be making decisions on how to do that in the middle of your wrangling. It's just like what you can put in Excel or a CSV.
Re:State of Clojure 2020 by fauigerzigerk on Friday February 21, 2020 @09:32AM (#22383356)
I know. I've been doing this sort of thing for most of my adult life.
Re:State of Clojure 2020 by ernst_klim on Friday February 21, 2020 @05:25AM (#22382171)
You have to be sure, like always. You can't write the code dealing with the data you have no clue about.

The type reflects invariants, constraints given for you data. The less you know, the more general type your data would have. Dynamic typing gives no advantages here safe for providing a super-general ANY type.

In case of statically typed language you would need to parse your data and construct an object representing that data just like in case of dynamically typed one. But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication.

Re:State of Clojure 2020 by goatlover on Friday February 21, 2020 @09:23AM (#22383273)
> But the more you know about your data in advance, the better, especially when you are dealing with protocols and communication.

The point of a library like Pandas is that you will be cleaning and reshaping your data until it's in a format that can be properly analyzed. And that data can be very dirty and very large.

But if you prefer to clean it up in Excel or with some bash scripts, or custom written functions in PL of choice, then go for it. You still have to deal with the cleaning and wrangling. The point of mentioning Pandas is that you don't always have a guarantee on properly structured data where you can predefine the types and read it in. And you don't always have control over how that data comes to you.

Re:State of Clojure 2020 by bitwize on Saturday February 22, 2020 @11:12AM (#22391690)
All data has SOME structural metainformation that can be captured in a type. If you are importing a CSV file, you know -- at a minimum -- that it's a CSV and can be parsed as a set of tuples of variants (or else there's something really wrong with the input and an exception continuation should be invoked to handle it). From there you can do analysis on the data to further refine its type.

But ALL data is typed, and encoding its type early in the code that processes it makes your assumptions clear and compiler-enforceable.

You're right that a typed approach to handling arbitrary input data is not right for a library like Pandas. It's a fundamentally different approach than the one Pandas takes, but it's no less valid and brings with it certain advantages.

Re:State of Clojure 2020 by dkersten on Friday February 21, 2020 @06:10AM (#22382326)
For the library, you are right, it has to support any tabular structure — but that’s what even ricks are for, to encode that there are parts that you don’t know until use.

When using the library, though, there is some level of assumed structure (otherwise your code wouldn’t work since it would be accessing fields that don’t exist. Sure data is messy in real life, so you encode that messy-ness into your structure as variants or optional that you then check, and ideally are forced to check so that you can’t silently let errors slip in).

This works in statically typed languages right now using generics, templates, types that aren’t inferred until the library is used, or whatever the specific language has to deal with this.