Re:Super-Structured Data: Rethinking the Schema
by HelloNurse on Tuesday May 17, 2022 @11:30AM
(#31412165)
It helps if this machinery can reject data and thus perform validation. Since recursive construction of union types (valid records can look like this, or also like that...) is trivial, a programmer somewhere has to draw the line between "loosen the schema to allow this record" and "reject this record to enforce the schema".
[ Parent
| on HN ]
Re:Super-Structured Data: Rethinking the Schema
by mccanne on Tuesday May 17, 2022 @11:41AM
(#31412323)
Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always implementation schema (or type) validation (and versioning) on top of super-structured data (as can also be done with document databases).
Re:Super-Structured Data: Rethinking the Schema
by cmollis on Tuesday May 17, 2022 @02:15PM
(#31414454)
this is a major hassle when converting from avro (from kafka which uses a schema registry, so schemas are not shipped with the avro data) and storing in parquet which requires a schema in the file but you can 'upgrade' it with another schema when reading it. It would be great to have a binary protocol-like format (schema-less avro), and a schema-less columnar storage format.. which is I guess is what these guys are doing.