gepheum
Links: - Website: https://skir.build/ -- Skir Go doc: https://skir.build/docs/go -- Skir Rust doc: https://skir.build/docs/rust -- Skir Swift doc: https://skir.build/docs/swift - Github: https://github.com/gepheum/skir - Blog post: https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t... - Discord: https://discord.gg/mruvDuybJ
As per messagepack, jsonbinpack, these seem to be layers on top of JSON to make JSON more compact. They still use field names for field identity, which I think can be problematic for long-term data persistence since it prevents renaming fields. I think the Protobuf/Thrift approach of using meaningless field numbers in serialization forms is better.
1. You can create an enum with just "wrapper" fields, that's exactly like a oneof 2. Totally fair, I'm planning to work on this later this year, probably Q3 (priority is adding support to 4 more languages, and then I'll get to it) 3. So there is introspection in the 6 targeted languages, and I think I did it a bit better than protobuf because it generally has better type safety. Example in C++: https://github.com/gepheum/skir-cc-example/blob/main/string_... Typescript: https://skir.build/docs/typescript#reflection I realize I haven't documented it in Python (although it is available and generally the same API as Typescript), will fix that However, you're right that there is no support yet for annotations. Still trying to gauge whether that's needed 4. Assuming you mean Go language: working on that now, hoping to have C#, Go, Rust and Swift in the next 2-3 months.
I completely agree with you about timing, wish I had done this 10 years ago :)
"Here is an idea to contemplate as a side gig with your favorite Ai assistant: A tool to convert proto to Skir. Or at least as much as possible. As someone who had to maintain larger and complex proto files, a lot of proto specific pain points are addressed." < I tried asking Claude: "Migrate this project from protobuf to Skir, see https://skir.build/" and it works pretty well. I created http://skir.build/llms.txt which helps with this. The pain point is data migration though, and as much as I want Skir to succeed: I cannot recommend people migrating from protobuf to Skir if they have some persisted data to migrate, the effort is probably not worth it.
``` struct User { id: int64; email: string?; name: string; } ```
You store some users in a database: [10,"[email protected]""john"], [11,"jane",null,"[email protected]"]
You remove the email field later:
``` struct User { id: int64; name: string; removed; } ```
Supposedly you remove a field after you have migrated all code that uses the field and you have deployed all binaries.
In your DB, you still have [10,[email protected]","john"], [11,null,"jane"], which you are able to deserialize fine (the email field is ignored). New values that you serialize are stored as [12,0,"jack"]. If you happen to have old binaries which still use the old email field and which are still running (which you shouldn't, but let's imagine you accidentally didn't deploy all your binaries before you removed the field), these new binaries will indeed decode the email field for new values (Jack) as an empty string instead of null.
I think the Protobuf language (which Skir is heavily influenced by) has some flaws in its core design, e.g. the enum/oneof mess, the fact that it allows spare field numbers which makes the "dense JSON" format (core feature of Skir) harder to get, the fact that it does not allow users to optionally specify a stable identifier to a message to get compatibility checks to work.
I get your point about "why building another language", but also that point taken too far means that we would all be programming in Haskell.
Copying from blog post [https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t...]:
""" Should you switch from Protobuf?
Protobuf is battle-tested and excellent. If your team already runs on Protobuf and has large amounts of persisted protobuf data in databases or on disk, a full migration is often a major effort: you have to migrate both application code and stored data safely. In many cases, that cost is not worth it.
For new projects, though, the choice is open. That is where Skir can offer a meaningful long-term advantage on developer experience, schema evolution guardrails, and day-to-day ergonomics. """
0. Yes, I looked at Avro, Ion. I like Protobuf much better because I think using field numbers for field identity, meaning being able to rename fields freely, is a must.
1. Yes. Skir also supports that with binary format (you can serialize and deserialize a Skir schema to JSON, which then allows you to convert from binary format to readable JSON). It just requires to build many layers of extra tooling which can be painful. For example, if you store your data in some SQL engine X, you won't be able to quickly visualize your data with a simple SELECT statement, you need to build the tooling which will allow you to visualize the data. Now dense JSON is obviously not idea for this use case, because you don't see the field names, but for quick debugging I find it's "good enough".
3. I agree there are definitely cases where it can be painful, but I think the cases where it actually is helpful are more numerous. One thing worth noting is that you can "opt-out" of this feature by using `ClassName.partial(...)` instead of `ClassName()` at construction time. See for example `User.partial(...)` here: https://skir.build/docs/python#frozen-structs I mostly added this feature for unit tests, where you want to easily create some objects with only some fields set and not be bothered if new fields are added to the schema.
4. Good question. I guess you mean "forward compatibility": you add a new field to the enum, not all binaries are deployed at the same time, and some old binary encounters the new enum it doesn't know about? I do like Protobuf does: I default to the UNKNOWN enum. More on this: - https://skir.build/docs/schema-evolution#adding-variants-to-... - https://skir.build/docs/schema-evolution#default-behavior-dr... - https://skir.build/docs/protobuf#implicit-unknown-variant
I agree it's important for users to understand that newer fields won't be set when they deserialize old data -- whether that's with Protobuf or Skir. I disagree with the idea that not forcing you to update all constructor call sites when you add a field will help (significantly) with that. Are you saying that because Protobuf forces you to manually search for all call sites when you add a field, it forces you to think about what happens if the field is not set at deserialization, hence, it's a good thing? I'm not sure that outweighs the cost of bugs introduced by cases where you forget to update a constructor call site when you add a field to your schema.
For dense JSON: the idea is that it is often a good "default" choice because it offers a good tradeoff across 3 properties: efficiency (where it's between binary and readable JSON), persistability (safe to evolve shema without losing backward compatibility), and readability (it's low for the reasons you mentioned, but it's not as bad as a binary string). I tried to explain this tradeoff in this table: https://skir.build/docs/serialization#serialization-formats
I hear your point about the tagline "like protos but better" which I hesitated to put because it sounds presumptuous. But I am not quite sure what idea you mean to convey by "fresh"?
Show HN: Skir – like Protocol Buffer but better
Why I built Skir: https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t...Quick start: npx skir initAll the config lives in one YML file.Website: https://skir.buildGitHub: https://github.com/gepheum/skirWould love feedback especially from teams running mixed-language stacks.