tsenart

Hi. Performance testing in your CI/CD pipeline will catch performance regressions of what your team has covered with benchmarks or load tests. Anything else will fall through the cracks.

Even the most capable performance conscious teams in the world still have limited attention. Perfloop has been contributing significant performance improvements to the open source world too, across repos like ClickHouse, Prometheus, parquet-go, roaring bitmaps and more.

https://github.com/search?q=is%3Apr+is%3Amerged+author%3Aper...

This was missing in the Go world.

Proprietary.

Author here, indeed a variation of bloom filters: https://x.com/lemire/status/1971279371131646063

This is to implement a database, not use one.

Yes! A typical use case is to efficiently implement ORDER BY LIMIT N in SQL databases in a way that doesn’t require sorting the entire column just to get those first N items.

Author here! Will do eventually.

Do share your findings!

Author here. Agree 100%! It's often what didn't work that is omitted. But there's so much juice in failed experiments — it's important to share with others.

Our Go ULID package has millisecond precision + monotonic random bytes for disambiguation while preserving ordering within the same millisecond. https://github.com/oklog/ulid

This, please! Native support for read-replicas would be awesome. Ideally it would now if a query is read-only or not without application changes.

Could you elaborate on the differences a bit deeper? I’m really interested in understanding.

How does it compare to https://github.com/tdunning/t-digest?

#MakeMonolithsGreatAgain

https://twitter.com/tsenart/status/1017061272188280833

OptioPay | Engineering | Berlin, Germany | ONSITE, VISA, FULL-TIME

We're on a mission to create the most beneficial payout ecosystem in the world and fundamentally change the way people receive money!

Now hiring: Backend Engineer, Frontend Engineer, Head of IT Operations, Security Engineer, Site Reliability Engineer, Senior Data Analyst, Senior System Administrator, Senior UX Designer, Web Developer.

Take a look at our open positions in more detail below and don't hesitate to reach out to me directly: ts [at] optiopay.com

https://jobs.optiopay.com/

OptioPay | Frontend UI Engineer, Backend Engineer, SRE, Security Engineer | Berlin, Germany | ONSITE, VISA, FULL-TIME

We're on a mission to create the most beneficial payout ecosystem in the world and fundamentally change the way people receive money!

Have a look at our open positions and don't hesitate to reach out to me directly: ts [at] optiopay.com

https://www.optiopay.com/careers

> A user of these IDs my not realize that the reliability of the ordering can be substantially reduced depending on where the IDs are generated.

That can only be addressed with improved documentation and shared understanding of the subtleties and pitfalls of distributed time synchronisation.

> Some applications may be able to tolerate inconsistencies in ordering, others may not.

Indeed. Proper thought must be but into this sort of thing. ULIDs aren't an exception nor a silver bullet.

> How many bits of approximate time does one really need?

Entirely application dependent.

Fellow PHP implementation: https://github.com/Lewiscowles1986/ulid

> What is the use of 48 bites of time? It reduces the overall entropy, for what? If time is important then why not make the id literally time (i.e. UnixNano), if it isn't they why not make all bits rand?

For some designs it's useful to have identifiers have other properties than uniqueness. In this case, this property is relative lexicographical (and binary) order based on time so that you can leverage the order between the things the identifiers identify without looking at the things. The entropy is there to satisfy the uniqueness property (with some acceptable degree of collision, application dependent). The time is there to satisfy the ordering property.

Yes, there's always a negligible chance of collision.

Yes if you use a good source of entropy. In Go, the easiest and most secure is the `crypto/rand.Reader`. But it's also slow. If you don't need the security properties, an instance of `math/rand.Rand` should suffice. Otherwise, you can implement your own. It's just an io.Reader!