News for hackers. Stuff that matters.
Re:Parsix: Parse Don't Validate by didibus on Saturday May 15, 2021 @12:29PM (#27166622)
I have to be honest, I'm not seeing what problem this is trying to solve. Anyone can enlighten me?

Edit: Ok I think I understand...

It seems the problem would be that if you're implementing a function that takes a user email as a string, and that function is in a lower layer of the application, like inside the data access layer. It is difficult at this point to know if the email string you will be passed as input has already been validated or not. Thus you might be tempted to re-implement validation for it at your level inside this function as well and have an assertValidEmail check.

This can lead to a littering of validation throughout the code base, as each implemented function worries that the input isn't validated and re-validates it, possibly using slightly different rules each time.

Furthermore, if you decide to not validate it again, you might be left wondering, but am I sure it'll have been validated prior? How can I be sure? Someone in the future could easily start calling my function and forget to validate the email before calling it? This could eventually lead to a security issue or just a bug, by introducing a code path that doesn't ever validate the email string.

Thus if instead you'd re-write your function so it takes an email as a ValidEmail type (or object), and not as a string, you force the caller to for sure remember to validate the email first. And you also can safely assume if you're getting an email as a ValidEmail type that it has been validated. It could also technically allow you to localize the validation logic to the ValidEmail type constructor, avoiding possible duplicate attempts at validating email with different rules.

And it seems the latter "style" the author calls "Parsing" while the former they call "Validating", in the sense that since the function validating returns a modified structure it "parsed" it, because a string became a ValidEmail, thus parsing a string into a ValidEmail, as opposed to simply validating that the string is valid as an email.

And finally, this is a little library to help make use of this pattern in Kotlin.

Re:Parsix: Parse Don't Validate by jpeloquin on Saturday May 15, 2021 @12:50PM (#27166807)
Paraphrased from the repo's readme: Suppose you have a program that consumes user input. Users often give bad input so the program needs to validate user input before acting on it. One way to validate is to call a function (e.g., `check_input`) on the user input and if it doesn't raise an error the input is safe for consumption by the rest of the program. The repo author considers this approach to be risky because the programmer can inadvertently omit or bypass `check_input` and the program still compiles and runs without complaint.

The repo presents an alternative validation approach, which is to parse the user input into a data type (or, not quite equivalently, into a class). The parsing process serves as validation. Consumer functions are written such that they only accept the parsed data type. Therefore it is now impossible for the programmer to inadvertently omit or bypass validation of user input.

The library is a set of convenience functions for actually writing these parsing / validation functions.

Re:Parsix: Parse Don't Validate by atoav on Saturday May 15, 2021 @01:27PM (#27167119)
So in short: instead of representing user input (e.g. a Email address) as a string – which you can forget to validate – the idea here is to create a own data type for it, and use the validation step to create said data type.

The rest of your program then works with this data type instead of the string and this way you will get a type error whenever you accidentally use unvalidated data.

A nice idea that goes into a similar direction is to expand on this and create more types for different levels of trust. E.g. you could have the data types ValidatedEmail, VerifiedEmail and TrustedEmail and define precisely how one becomes the other. This way your typesystem will already tell you what is valid and what is not and you can't accidental mix them up.

Re:Parsix: Parse Don't Validate by TeMPOraL on Saturday May 15, 2021 @04:50PM (#27168893)
You can also further generalize this idea by noticing you can encode all kinds of life cycle information in your type system. As you transform some data in a sequence of steps, you can use types to document and enforce the steps are always executed in order.

In this example, the user input validation step is f(String) -> ValidatedEmail, then the process of verifying it is f(ValidatedEmail) -> VerifiedEmail. But the same principle can apply to e.g. append() operation being f(List[T], T) -> NonEmptyList[T], and you can write code accepting NonEmptyList to save yourself an emptiness check. Or, take a multi-step algorithm that gets a list of users, filters them by some criterion, sorts the list, and sends these users e-mails. Type-wise, it's a flow of Users -> EligibleUsers -> SortedEligibleUsers -> ContactedEligibleUsers.

And then, why should types be singular anyway? You should be able to tag data with properties, and then filter on or transform a subtag of them. This is the area of theory I'm not familiar with yet, but I imagine you should be able to do things like:

List[User] -> List[User, NonEmpty] -> List[User[Eligible], NonEmpty] -> List[User[Eligible], NonEmpty, Sorted[Asc]] -> List[User[Contacted], Sorted[Asc]].

Or,

Email -> Email[Validated] -> Email[Validated, Verified] -> Email[Validated, Verified, Trusted].

I'm sure there's a programming language that does that, and then there's probably lots of reasons that this doesn't work in practice. I'd love to know about them, as I haven't encountered anything like it in practice, except bits and pieces of compiler code that can sometimes propagate such information "in the background", for optimization and correctness checking.

Re:Parsix: Parse Don't Validate by rictic on Saturday May 15, 2021 @07:30PM (#27169889)
The closest I've seen is Idris, which lets you pass around data along with proofs about that data. The proofs can be generated at either runtime or compile time (the usual distinction between the kinds of things you can do at runtime vs compile time is blurred in all sorts of delightful ways in Idris).

The Idris Book (https://www.manning.com/books/type-driven-development-with-i...) is structured in a very practical, "learn some then build some" format that I found to be a joy to read.

Re:Parsix: Parse Don't Validate by akersten on Saturday May 15, 2021 @07:50PM (#27169996)
I really like this idea. It feels like weapons-grade TypeScript and could solve an entire class of logical error. If someone has a good method they're already using to encode strict typing like this I'd love to check it out.
Re:Parsix: Parse Don't Validate by greggman3 on Sunday May 16, 2021 @05:04AM (#27172326)
This site on F# used to show how to do the same in C# (Which would be more applicable to typescript?) but, the usually coding via types is best in a language designed to make that easy. AFAIK Typescript (nor C++, C#, Java) are designed for that purpose where as Haskell, F# are.

That doesn't mean you can't do it in those other languages, only that it's tedious compared to languages that are designed for it.

https://fsharpforfunandprofit.com/posts/designing-with-types...

https://fsharpforfunandprofit.com/posts/fsharp-decompiled/

Re:Parsix: Parse Don't Validate by eru on Sunday May 16, 2021 @01:03AM (#27171347)
This is pretty common practice in languages like Haskell.
Re:Parsix: Parse Don't Validate by Gregaros on Sunday May 16, 2021 @01:06AM (#27171361)
Reminds me of dependent types: https://en.wikipedia.org/wiki/Dependent_type
Re:Parsix: Parse Don't Validate by _greim_ on Saturday May 15, 2021 @05:34PM (#27169241)
To keep building on this, I think the word "parsing" is just the tip of the iceberg. Parsing is one way to port data across a type boundary, where the source and dest types are optimized for different use cases (e.g. serialization vs type-safe representation). Since the semantic Venn diagrams of any two types might have areas of non-overlap, parse-don't-validate means establishing clear boundaries in your program where those translations happen, then defining the types on either side of the boundary to rule out the possibility of nonsense states elsewhere throughout the program. The idea of nonsense states is closely related and discussed more here[0] and here[1].

[0] http://blog.jenkster.com/2016/06/how-elm-slays-a-ui-antipatt...

[1] https://kentcdodds.com/blog/make-impossible-states-impossibl...

Re:Parsix: Parse Don't Validate by _jal on Saturday May 15, 2021 @01:48PM (#27167320)
Reminds me a little of taint checking in Perl and Ruby, in reverse.
Re:Parsix: Parse Don't Validate by feurio on Sunday May 16, 2021 @04:45AM (#27172231)
Yes. Also I thought of when I have created a Domain in Postgres (https://www.postgresql.org/docs/9.5/sql-createdomain.html) with a constraint written in PL/Perl to ensure that the data going in is always valid (eg. check digits in IDs, a bit like in credit cards).
Re:Parsix: Parse Don't Validate by Smaug123 on Saturday May 15, 2021 @12:42PM (#27166741)
If you mean "why Parse, Don't Validate", you should read the original blog post, linked at the top of the article. It's… transformative, if you aren't already aware of the principle. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

If you mean "why this library", well, I guess parser combinators are nice! Some may say that a declarative statement of the parsing restrictions is better than a procedural implementation, on general principles.

Re:Parsix: Parse Don't Validate by cle on Saturday May 15, 2021 @12:53PM (#27166832)
There are lots of siblings explaining why “parse don’t validate”.

But also, it’s not always wise to take this to an extreme. I’ve seen over the years many scenarios where dev teams were over-enthusiastic about this and parsed themselves into a corner by making system components over-strict and enforcing invariants that weren’t necessary to enforce, making them much harder to change later.

The right answer is, of course, somewhere in the middle, and depends on your domain and situation.

Re:Parsix: Parse Don't Validate by Dylan16807 on Saturday May 15, 2021 @09:52PM (#27170558)
> The right answer is, of course, somewhere in the middle, and depends on your domain and situation.

I strongly disagree. The problem you describe, of being overly strict, is orthogonal to whether you validate or parse. There is no "middle" between parsing and being strict. Parse-validate and strict-nonstrict are separate axes.

Re:Parsix: Parse Don't Validate by Iazel on Saturday May 15, 2021 @02:07PM (#27167478)
hi, @cle! Curious to hear more about that, were they actually running validation/assertions in constructors?
Re:Parsix: Parse Don't Validate by cle on Saturday May 15, 2021 @03:41PM (#27168262)
That can be a case of that yeah. Using your example, a lot of devs might use that email parsing logic in various independent components of the same system. Eg if you have a reporting component that sends you business reports, that component really shouldn’t be validating the structure of email addresses…if you need to refine the parsing logic now you’ve got to do coordinated deployments, possibly backfills, etc., whereas if you just treated it as an opaque string in that system you’d be better off.

This isn’t really a criticism of the approach, it’s super useful, just that it needs to be applied judiciously. “Parse all the things” isn’t always the best advice.

Re:Parsix: Parse Don't Validate by Kranar on Saturday May 15, 2021 @07:44PM (#27169970)
Your response isn't clear.

>that component really shouldn’t be validating the structure of email addresses

That's right, as the article says you no longer validate data, instead you parse data. Components don't take strings and validate them, they take an EMail address.

>If you need to refine the parsing logic now you’ve got to do coordinated deployments

If you need to refine the parsing logic, you modify the constructor of the EMail class to do whatever refinements you need.

I have no doubt you may have seen codebases that did something wrong, and that wrong thing may have been related to parsing or validation, but nothing you've said indicates that there is something wrong with replacing validation with parsing, so that code always operates on semantically meaningful data types that are valid by construction, instead of opaque binary blobs that need to constantly be properly interpreted in ad-hoc ways.

Re:Parsix: Parse Don't Validate by cle on Saturday May 15, 2021 @09:12PM (#27170373)
I guess what I'm saying is to be judicious about what invariants a particular piece of code should care about, within the context of the larger system.

> That's right, as the article says you no longer validate data, instead you parse data. Components don't take strings and validate them, they take an EMail address.

That doesn't solve the problem, by parsing it my code is caring about its syntax and structure, even if my use case doesn't require that. That creates coupling, and in a distributed system that kind of unnecessary coupling can create major headaches when trying to change things.

An even simpler example that many people can empathize with is using strict, closed enums in a distributed system (very easy to do with e.g. Java). Safely adding a new enum value is basically impossible at that point, until you make them "open" enums that can safely parse unknown values.

Again I am not disagreeing in principle, I am only saying that this has ergonomic problems in practice and is easy to misuse.

Re:Parsix: Parse Don't Validate by Iazel on Sunday May 16, 2021 @05:49AM (#27172505)
I see, that would be problematic indeed. I strongly believe in Bounded Contexts, a concept from Domain Driven Design. As for the Ubiquitous Language (ie: product & engineers naming things the same way, code included), parsing logic should also be segregated and specific to your domain. I would argue that you should have different parsers for different purposes even inside the same system, for example one could be applied to inputs coming from an User, while another could be applied to an API and one more when coming from a DB.

I think I would stress this point more in the README, thanks for sharing :thumbsup:

Re:Parsix: Parse Don't Validate by GordonS on Saturday May 15, 2021 @02:28PM (#27167652)
If it helps, here's a related blog post but with a C# slant:

https://andrewlock.net/using-strongly-typed-entity-ids-to-av...

The author refers to using primitives everywhere as "primitive obsession", and proposed using types instead.

Re:Parsix: Parse Don't Validate by dmux on Saturday May 15, 2021 @05:29PM (#27169208)
Similar to the idea of "microtypes" (I've most often seen it used in Java circles):

https://www.markhneedham.com/blog/2009/03/10/oo-micro-types/

Re:Parsix: Parse Don't Validate by Iazel on Saturday May 15, 2021 @02:24PM (#27167619)
Cool to see you perfectly got the point in the end! I wonder though, were you confused by the README? What made it clear for you?
Re:Parsix: Parse Don't Validate by didibus on Saturday May 15, 2021 @03:10PM (#27167976)
Hum, it was the people here who replied to my question, and also reading the linked article.

I think my confusion was in trying to frame things as parsing VS validating. While I now appreciate that use of word, now that I understand, it also caused my biggest source of confusion.

That's because I think most people think of parsing as conversion, like I turn a String to an Int. Where as in your case, you're simply wanting to tag a type as having been validated, but you don't really convert the type itself, so you simply wrap it in another type in order to tag it as having been validated simply because the language offers no other way to tag the type with meta-information for the compiler to assert statically.

So because it seemed more like you're just wrapping the input, but still all code will be using the input value as it is, extracting it out of your wrapped type, the idea that you were "Parsing" and not "Validating" well just confused me.

Re:Parsix: Parse Don't Validate by sullyj3 on Saturday May 15, 2021 @11:12PM (#27170893)
It's true that here the data type was just a tag. There's always a decision to be made about how much structure you want to enforce by the construction of the data type. You'd still consider it to be "parsing" because you now statically know that an Email is valid according to the rules of the parser - in a loose sense, it's a "conversion" between a type you know nothing about, and one you know a lot about.

If you wanted to go further you could start going the "correct by construction" route - having a data type that enforces more invariants. For example, you might store the recipient name, domain name, and tld of your email in separate fields. Then your parser would more obviously be a parser. I think of the "mere tag" type as a kind of degenerate case of this, rather than something totally separate.

Re:Parsix: Parse Don't Validate by Iazel on Sunday May 16, 2021 @05:50AM (#27172508)
I see, thanks for sharing. I will improve the README by adding a clear definition for parsing, validation and deserialization, so that we can all be on the same page ;)
Re:Parsix: Parse Don't Validate by matheusmoreira on Saturday May 15, 2021 @06:57PM (#27169719)
This also has security implications. The input handling layer is critical. Bugs in parsing and validation code are responsible for a huge number of vulnerabilities.

More details: http://langsec.org/

> The Language-theoretic approach (LANGSEC) regards the Internet insecurity epidemic as a consequence of ad hoc programming of input handling at all layers of network stacks, and in other kinds of software stacks.

> LANGSEC posits that the only path to trustworthy software that takes untrusted inputs is treating all valid or expected inputs as a formal language, and the respective input-handling routines as a recognizer for that language.

Re:Parsix: Parse Don't Validate by mirekrusin on Saturday May 15, 2021 @12:56PM (#27166867)
Imagine you're writing typescript project. You type everything and have type safety. This type safety is an illusion on I/O boundaries – whenever ie. JSON.parse(...) from file/websocket/http happens. To preserve type safety, you want to use something like [0] to do runtime type assertions. Once i/o boundaries are parsing unknown types at runtime into what is defined as static types, your type safety is guaranteed.

[0] https://github.com/appliedblockchain/assert-combinators

Re:Parsix: Parse Don't Validate by TheAceOfHearts on Saturday May 15, 2021 @01:01PM (#27166909)
Refining types so they encode all desired constraints before use. This is explained in the linked article: Parse, don’t validate [0].

It helps reduce the risk of using invalid inputs by representing constraints over the value as part of the type.

For example: a common problem in web development security is that query parameters aren't properly validated which can lead to denial of service attacks. As a trivial example of this, consider a web server which paginates some data using "offset" and "limit" by passing those parameters directly to a database query; an attacker could set "limit" to some incredibly high value and cause the server to crash. If you're just doing validation on your inputs it's possible that some usage could end up being overlooked.

[0] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

Re:Parsix: Parse Don't Validate by gregors on Saturday May 15, 2021 @01:21PM (#27167069)
So real question - in the "offset" "limit" example what makes it any more safe if at first the programmer sets those types to be integers? The same problem persists does it not?

Does the explicit creation of a type add this introspection? I'm not convinced that it does. Now once you fix this bug, encoding it in a type prevents it from creeping into other parts of the the code. This seems more like DRY principles in action.

Re:Parsix: Parse Don't Validate by didibus on Saturday May 15, 2021 @02:03PM (#27167449)
Yeah, it seems to be more about guarantees as a code base grows larger and more people touch it.

If there's a Limit class whose constructor and setter all check that the range is between say 5 to 100, and all existing code that needs the limit uses the instance of Limit, it just becomes less likely a code change is made that uses the limit input as it was directly provided by the user (and thus possibly out of range).

But you'd still need to have had someone be smart enough to make sure the Limit class does prevent limits that could cause DB crashes.

In practice I'm thinking, ok, so someone must have thought... Hey we should validate this user input and put in some logic for it.

So I think what this says is, validation works by having all external input validated as they are received. But it can be easy to make a code change at the boundary where you forget to add proper validation. If all existing functions in the lower layers, like in the data access layer, are designed to take a Limit object, the person who took a limit as external input and was about to pass it to the query function will get a compile error and realize... Oh I need to first parse my integer limit into a Limit, and thus reminds them to use the thing that enforces the valid range.

If instead the code had a util function called assertValidLimit, and the query function took a limit as an integer, it be easy for that person to forget to add a call to assertValidLimit when getting the limit from the user and then pass that unvalidated to the query and possibly cause a vulnerability.

And lastly, it seems they argue, if you were to validate instead in the query function itself, thus it wouldn't matter if others forget to validate since where it matters would, but then it is hard to fail at that layer, since you might have already made other changes and that can leave your state corrupted.

So basically it seems the argument is:

"It is best to validate external input at the boundary as soon as it is received, but it can be easy to forget to do so and that's dangerous. So to help you not forget, have all implementing functions take a different type then the type of the external input, which will remind people... Oh right I need to parse this thing first and in doing so assert it's valid as well.

Re:Parsix: Parse Don't Validate by Iazel on Saturday May 15, 2021 @02:34PM (#27167711)
Well said! I would only like to add that I highly discourage adding validations/assertions in the actual data class, this often make them hard to work with and reuse. It is better to have this parsing logic as a simple function, perhaps at factory level if you prefer that kind of flavor :)
Re:Parsix: Parse Don't Validate by TheAceOfHearts on Saturday May 15, 2021 @02:40PM (#27167759)
Apologies if I did a poor job of explaining, what you wrote seems in agreement with what I was attempting to convey.

If one were only using integer types then the same problem would persist, that's correct. The problem would be solved by defining our limit type to only represent positive integers up to a specific safe value.

Type refinement is done on the input boundaries of the system during runtime to prevent errors from propagating.

Re:Parsix: Parse Don't Validate by mbildner on Saturday May 15, 2021 @02:06PM (#27167473)
This is not yet possible in Typescript, but imagine if you could define a numerical subtype that requires your input be below some threshold eg:

`type Limit = 0..100;`

See discussion here: https://github.com/Microsoft/TypeScript/issues/15480

Re:Parsix: Parse Don't Validate by steventhedev on Saturday May 15, 2021 @12:41PM (#27166730)
There's an entire class of vulnerabilities caused by having separate verification and parsing logic, typically with fields that usually only one is used, but the format supports multiple. The verifier checks the first one but the parser uses the last one.
Re:Parsix: Parse Don't Validate by StreamBright on Saturday May 15, 2021 @04:19PM (#27168611)
Interesting naming. Strongly typed languages (especially in the ML family) have best practices that include using types instead of strings as function parameters. Email type itself is enough to skip validation in each function accepting that particular type.

I think this is great first step using functional languages but you can go much much deeper than that.

https://www.slideshare.net/ScottWlaschin/the-power-of-compos...

Re:Parsix: Parse Don't Validate by finnh on Saturday May 15, 2021 @12:42PM (#27166743)
It encourages people to use strongly typed classes rather than primitives, even if the type simply wraps a primitive.

As a result you can't pass a invalid (say) accountID deep into your code, bc validity is guaranteed to be checked early when you "parse" an input string into the "AccountId" type.

So: internal interfaces defined using non-primitive types, so internal methods don't need to keep validating their input. Conversion to said types happens early and predictably, catching bad values before they (eg) hit the database.

Re:Parsix: Parse Don't Validate by jinwoo68 on Saturday May 15, 2021 @12:41PM (#27166726)
As they said in README, it’s inspired by Alex King’s Parse, don’t validate [1].

Basically, rather than write a validation function, write a parser that returns a result of a specific type and use that type everywhere else. Then you can make sure the raw inputs are always validated.

[1] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

Re:Parsix: Parse Don't Validate by anaerobicover on Saturday May 15, 2021 @01:15PM (#27167007)
Small correcting, the author is named Alexis
Re:Parsix: Parse Don't Validate by jinwoo68 on Saturday May 15, 2021 @02:15PM (#27167550)
Whoops. Thanks for correcting.
Re:Parsix: Parse Don't Validate by coderintherye on Saturday May 15, 2021 @12:45PM (#27166769)
The linked blog post explains it pretty well. Essentially, it seems to be solving for unexpected cases or incorrect validation by using of static typing and passing the expected type back in the return rather than a boolean. I'm not sure I've encountered enough issues with validation functions to use this pattern, but it does seem like a more robust way of writing them.
Re:Parsix: Parse Don't Validate by rdedev on Saturday May 15, 2021 @02:27PM (#27167643)
I find this approach combined with phantom data types really cool. Now you can easily introduce a semantic differentiation between two instances of the same data type but without much overhead