News for hackers. Stuff that matters.
Re:Writing a SQL database from scratch in Go by cube2222 on Monday April 13, 2020 @02:48AM (#22853691)
Great blog post!

I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL.

And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]).

[0]: https://github.com/cube2222/octosql

Re:Writing a SQL database from scratch in Go by pizza234 on Monday April 13, 2020 @03:34AM (#22853859)
This is not the first educational project I've seen that insists on manually writing the parser.

Call me contrarian but I find the article's approach poor (at least, as it is now).

If one looks at the overall content of the two articles of the series, 70%/80% of the content (or possibly more) is parsing, which is arguably the least interesting, or at least, the most boilerplate, part of a database system.

Re:Writing a SQL database from scratch in Go by eatonphil on Monday April 13, 2020 @09:33AM (#22855669)
I like this project! I think I've asked you about it supporting parquet before... I'll add it to my list of similar projects.

Regarding goyacc and "usually", I myself was curious about this years ago and asked on Stack Exchange: Do modern languages still use parser generators? [0].

Most of the time, major languages _do_ use hand-written parsers.

Of course it may literally be the case that most other SQL projects in Go do use goyacc. My point is that in general, hand-written parsers are more common in real software than you might think.

[0] https://softwareengineering.stackexchange.com/questions/2502...

Re:Writing a SQL database from scratch in Go by cube2222 on Monday April 13, 2020 @09:41AM (#22855736)
Indeed, I remember! :) The parquet support is mostly ready as far as I know here https://github.com/cube2222/octosql/pull/153/ but I haven't tested it myself yet. However, we'll surely have it ready for the "streaming" release which is planned for May/June.

Regarding the parsing part, that's interesting. Though the cockroachdb parser is a _great_ lecture about how to add nice error messages with goyacc. (You add a special error token which captures anything after a syntax error, so the parser doesn't break down. At least something around this, haven't yet dived deep into it.)

Though I know Go itself uses a handwritten parser.

Re:Writing a SQL database from scratch in Go by tbrock on Monday April 13, 2020 @08:51AM (#22855384)
Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history.

Was it because they were the first and people just started using it or is it the best for some reason?

Re:Writing a SQL database from scratch in Go by eatonphil on Monday April 13, 2020 @09:34AM (#22855675)
SQL is a humongous spec. Any serious project would rather piggy-back off an existing parser. Most of the interesting parts for most people is implementing backends against in-memory, disk, S3, HDFS, etc.
Re:Writing a SQL database from scratch in Go by eatonphil on Monday April 13, 2020 @09:40AM (#22855725)
Also, a contributor to dolthub shared on Reddit last time this post came up that they originally wrote their own SQL frontend but gave up because it was so much to maintain and get correct. They ended up going with go-mysql-server which uses vitess.

https://www.reddit.com/r/golang/comments/fgwwlx/database_bas...

Re:Writing a SQL database from scratch in Go by cube2222 on Monday April 13, 2020 @09:36AM (#22855696)
No idea about the history.

From my perspective (as the author of OctoSQL): I need a SQL parser. Vitess has a fully fledged MySQL compatible one. I'd probably want to copy the open source vitess one and add the necessary features myself, instead of writing it from scratch. It's just a big endeavor, a few thousand lines of goyacc code and even more Go boilerplate.

I'm not saying everybody is using the same library, because the code has been heavily modified after being copied. A look at the cockroachdb parses suffices to see how much changed it is.

If you look at OctoSQL, there too have been many changes to add stuff like table valued functions.