These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road.

I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.

C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.

Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.

#pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required.

> And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.

I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.

You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work.

Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.

GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.

Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself.

But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.

Person tries to find a library that does what one line of code would do (cough left pad cough and others) and then guess what, the library fails at that thing that maybe not 90% of people need but 50% need

And that even applies to some corners of famous apps or libraries big sigh

Invented here syndrome: https://dev.to/mortoray/invented-here-syndrome-4mg8

I agree with your comment. I would add that there are cases where your problem looks really easy right now, but will get increasingly complicated over time in ways you don't expect. In this case, "overkill" 3rd party solutions are actually better. For example, you could write an HTTP client using sockets pretty easily, and it will work in simple cases. But you will, over time, run into edge cases and problems that a mature library has already solved for you.

Also, this problem reminds me of having to decide whether to solve the integral, or look it up, back in the day. Solving from scratch was always preferable if you could do it, but sometimes it was just too hard.

It’s entirely possible that the new problems you encounter will be different than the ones the mature library was designed to address, and then you end up hacking around solutions to problems you don’t have in order to fix the ones you do.

We became the third company I know about to ever solve this problem, but I had built what my colleague called “the hardest to use HTTP library ever”. I was happy to move back to AFNetworking once we could do so without a regression in performance or hitting that bug.

> job scheduling systems that could be replaced by a postgres table

I've never used Postgres in my life ever.

Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#djang...

I have used nowait before to batch scheduled jobs together for a slow API call.

Moving everything to rq, and so far so good.

I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.

I wish you the best, but I cannot help but think you will regret that decision.

1. https://youtu.be/SUQYuA9AJT0?t=7546

Either Mongo, MySQL or List in Redis.

Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)

API is pretty small and comprehensible.

Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.

There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)

However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines

What about null?

JSON Schema – a whole topic unto itself – gives you further control over locking down how this can be represented (e.g. field MUST be present but can be string matching this regex or null).

Shameless plug: https://github.com/RantyDave/cppbor