tanoku

> I don't swear that this is how GitHub actually works, but I've had knowledgeable friends swear up and down that it is.

All the details on how GitHub's infrastructure has evolved over the years are very publicly detailed in the GitHub engineering blog and in technical talks. There are no "secrets" or "rumors" here, all the information is one google search away. Perhaps you need to re-evaluate your priors on how knowledgeable your friends are.

- https://github.blog/engineering/architecture-optimization/in... - https://www.youtube.com/watch?v=Ri8hSZNKzu4 - https://github.blog/engineering/building-resilience-in-spoke... - https://github.blog/open-source/git/counting-objects/ - https://www.youtube.com/watch?v=DY0yNRNkYb0 - https://cursor.com/blog/git-at-any-scale

Very very seldomly! Obviously we need to be correct in 100% of the cases, which is why the de-optimizer is there, but in practice it just never triggers. Right now the things that trigger a deopt are very limited — malformed time literals, oferflowing integer negations and very little else. The performance impact is essentially zero.

Our vector implementation is fully compatible with the syntax for vector fields and comparison functions in MySQL 9 -- we have carefully backported all the changes to our version of MySQL 8 to ensure full backwards and forwards compatibility. The index syntax is specific to PlanetScale because the open-source MySQL 9 preview does not yet have support for ANN indexes on vector columns, but we're committed to being as compatible as possible in the future and to minimize fragmentation. I do agree that fragmenting the ecosystem is not great!

These are very relevant questions! Thank you!

We're storing IDs from a ghost column that is created in the table where you're inserting vector data. This works very well in practice and allows updating the value of the vectors in the table, because they're translated into a delete + insert in the vector index by updating the ghost ID.

We have abstracted away the quantization system from the index; for the initial release, vector data is stored in raw blocks, like in the paper. Query performance is good, but disk usage is high. We're actively testing different quantization algorithms to see which ones we end up offering on GA. We're hoping our beta users will help us guide this choice!

Incremental updates and MVCC are _extremely tricky_, for both correctness and performance. As you've surely noticed, the hard thing here is that the original paper is very focused on LSM trees, because it exploits the fact that LSM trees get compacted lazily to perform incremental updates to the posting lists ('merges'). MySQL (and Postgres, and all relational databases, really) are B-tree based, and in-place updates for B-trees are expensive! I think we came up with very interesting workarounds for the problem, but it's a quite a bit to drill down in a HN comment. Please stay tuned for our whitepaper. :)

Definitely yes to the first: the index implementation is fully integrated with PlanetScale's sharding layer, so you can scale horizontally as much as you need. This works very well in practice: the sharding layer, Vitess, is extremely well battle tested with petabyte-sized clusters in production, and our vector indexes are like any other table on the cluster, so it really scales horizontally with very predictable performance.

As for separating storage and compute: we don't do that. One of our key sells here is that this is vector data fully integrated into your relational database with ACID semantics. Very hard to do separate storage and compute and keep this behavior!

Hi! I'm one of the authors of this feature. It's something quite novel, because it's not just a HNSW plug-in for MySQL (like e.g. pgvector is for Postgres). It's a fully transactional index, integrated into InnoDB.

We based the implementation on two very new papers from Microsoft Research, SPANN and SPFresh. SPANN is a hybrid graph/tree algorithm that does a fantastic job of scaling larger-than-RAM indexes (https://arxiv.org/abs/2111.08566) and SPFresh expands upon it by defining a set of background operations that can be performed to maintain the index's performance and recall while it's continuously updated in-place (https://arxiv.org/html/2410.14452v1). The novel thing here is designing all the SPANN _and_ SPFresh operations to be transactional, and integrating them in MySQL's default storage engine.

This tight integration fundamentally means that inserting, updating and deleting vector data from MySQL is always reflected immediately in the index as part of committing your transaction. But it also means that the indexes are fully covered by the MySQL binlog; they recover from hard crashes just fine. They're also managed by MySQL's buffer pool, so they scale to terabytes of data, just like any other table. And also crucially, they're fully integrated with the query planner, so they can be used in any query, including JOINs and WHERE clauses (something that many other vector indexes really struggle with).

We plan to release a whitepaper on our transactionality extensions to SPFresh, which I think will be super interesting, but meanwhile please feel free to test the beta and ask me any questions (here, or by emailing PlanetScale support). Thanks!

Hey Nican! Thanks for the feedback. It wasn’t clear from the blog post, but as the sibling poster points out, the system has full eventual consistency: it behaves like a replica, but it replicates a whole cluster of MySQL instances simultaneously (i.e. your full PlanetScale database). Because of this design, we never lock or affect the performance of writes to the main database.

As for predictability, we’re working on some interesting optimizations that allow similar queries to reuse the internal state of each other, so the system becomes more efficient the more queries it’s caching. Stay tuned!

Hey, author here. Thanks for the kind words! This is a custom pipeline that I designed for the article. It's implemented as a Node.js library using SVG.js and it statically generates the interactive SVGs directly in the static site generator I was using (Eleventy) by calling out to the Go compiler and extracting assembly for any lines you mark as interesting. It turned out very handy for iterating, but it's not particularly reusable I'm afraid!

Oh OK, you have clearly spent more time thinking about this problem than the team of engineers at GitHub who've been researching code search at scale for more than four years. I bet they feel real silly right now knowing they could have shipped this search engine in a couple weeks taping together off-the-shelf libraries if they only had your talent for software architecture.

Thank you, I appreciate that. I'll point people at SunSed the next time they ask about MD alternatives. :)

Actual author of Sundown here.

> even if it was active it's not a very common Markdown implementation

First, even if this was true, it's still _not_ OK to name your project Sundown.

Second, _my_ Sundown is the most used Markdown engine in the world. It has powered all Markdown rendering in GitHub.com and Reddit.com for more than 5 years, and it has rendered billions more documents than all other Markdown parsers out there.

Could you please not be an asshole and rename your project? It's really not that hard to come up with a name for a parser. Don't steal mine.

Hi there! As the spec explains, this is a Markdown specific blacklist that prevent the tags that would otherwise "break" the content of the Markdown document.

A document that contains these tags will not be parsed properly by an HTML5 compliant parser; the parser will "swallow" other chunks of Markdown content that come after the tags. Hence, we disable the tags altogether.

This is an UX feature, not a security feature. XSS prevention, and a plethora of other security checks, are performed by our user content stack -- but this functionality is shared for all markup languages in GitHub (MD, RST, ASCIIDOC, ...), so it's not discussed in this spec.

Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is deployed (it has different input behaviors, and most importantly, it has different load and performance characteristics).

The question then becomes "why would you run these experiments offline when you can run them online?". So we simply do. I personally feel it's a game changer.

Most tech startups work in English, specially the ones with international/remote workers. Besides that, the "working language" is Catalan, because Barcelona is a city in Catalunya.

These are not vulnerabilities in Rack (necessarily), but in the way the cookie spec has been drafted and the way Chrome decides to implement it. There's nothing that can be "fixed" in Rack: the only definitive fix is a domain migration.

Oh, by the way, people were asking on Twitter, so we just enabled support for the HTML5 clipboard API. If you're using Chrome, you can now just paste an image into the comment box to upload it!

Very handy with Mac OS X screenshot shortcuts that copy straight to the clipboard.

It was done in-house. We really care about how our graphs look, because graphs represent information. It's like typography. Humans learned how to write 5000 years ago, and it took us a very long while until we mastered the art of transmitting information like that. Once we mastered it, however, we started caring about the way we present that information. For no specific reason, but we developed unique typefaces that look beautiful, different, yet transmit the same information.

Likewise, we've spent a ton of valuable time graphing and mining data from all over our stack; it's only natural that after that, we care about how we present that data. Like typography, it's hard because it's an arduous hunt for the aesthetics hidden behind information. But this hunt is what makes us human.

That's why we design beautiful graphs, just like we'd design a beautiful typeface. Not because we are GitHubbers, nor because we are designers, but because we are humans.

Life is too short to stare at ugly graphs. And yes, you should feel bad.

PS: we have no project managers at GitHub.

Sorry, what schedule?

This is Hacker News; we try to stay civil, so I don't think many people are going to "hate" you for not using (understanding?) version control.

Then again, I don't think many people will take you or your project seriously after this, either. Many of us went past folders and tarballs for version control back in 1992, and personally, there's no way I'm going to touch an OS Kernel that is distributed in a zip file. That is not how you do open source -- for what it's worth, you may want to keep your tarballs to yourself: that'd serve the same practical purpose.