Lets not create a false equivalency between Go and C.

Go is a good language, one of its core pieces is to increase memory safety through the use of a garbage collector.

Much of what the SQLite post states may not be safe operations in Go.

Rust would be a safer alternative that meets many of the SQLite requirements.

The thing is, SQLite is bulletproof at this point, so do we need to replace it?

> The thing is, SQLite is bulletproof at this point, so do we need to replace it?

No, probably not, it is fine as it is. Still, even SQLite started out as a "for fun" project. Who knows what might happen in the future to displace it? ;)

Sqllite didn’t start for fun it was built for navy destroyers and submarines. (1) that’s probably why parent called it bulletproof.

(1) https://en.wikipedia.org/wiki/SQLite#History

It could also be a reference to the comprehensive SQLite test suite (1) or the fact that it has been in production systems for many years, though it’s initial purpose does make bulletproof quite apt.

(1) https://sqlite.org/testing.html

> Sqllite didn’t start for fun it was built for navy destroyers and submarines.

Those aren't mutually exclusive. I for one find destroyers to be an absolute riot.

I admit that I was merely being metaphorical and referencing the comprehensive test suite, wide distribution and usage.

I wish I had intended that pun embedded in the comment, but from now on ;)

When Wikipedia simply says otherwise I wonder where you got the "for fun" reference from. I tend to almost blindly trust comments on HN but this one was quite off.

You baited me into making an account.

See https://changelog.com/podcast/201#transcript-100

>I had couple months off and I thought, "Hey, I'm just gonna go and cobble together a really quick and simple database engine that just does a few very simple SQL commands, insert the lead, update and select." No joints, wasn't trying to be efficient... All I needed to do was pull stuff off of a disk in that memory.

>And I put it out there and... I've been doing open source for years before this, putting things on my website, and people would find my thing -- or well, you know, I'd put things on my website and it'd get like five downloads per year, or something like that. I'd figured this would be just another one of those things, but for whatever reason it really resonated with people.

> The thing is, SQLite is bulletproof at this point, so do we need to replace it?

Nothing written in C is bulletproof.

You're free to write bugs in any language. C just makes them easier to write or more critical.

OpenSSL was thought to be "bulletproof" and the "go to" option. And look where we are now

SQLite has a much more thorough testing process than OpenSSL did. Their acceptance tests have 100% MC/DC coverage [1], for example. While previous things like the IOC tester did find issues with SQLite, SQLite is very aggressive about including these sorts of tools into their ongoing testing process.

[1] MC/DC coverage is a slightly more rigorous form than branch coverage. A condition if (a && b) requires that you only test one of (a false, b true) and (a false, b false) for 100% coverage, whereas MC/DC coverage would insist on both being tested.

OpenSSL has always been nowhere near the level of robustness and bulletproofness that SQLite is at. Just look at how it's tested: https://www.sqlite.org/testing.html

It's too bad some part of tests are proprietary. I think it someone would want to rewrite SQLite (keeping compatibility), those tests would be of huge value.

Yes, but that is also a source of funding for the project.

And how many more critical flaws would OpenSSL have if people kept rewriting it in the trending language of the week?

Rewriting just for the sake of rewriting or because you like another language better is an almost certain recipe for disaster.

I have a hard time believing this. It was common knowledge and lamented before Heartbleed that OpenSSL was bloated an unwieldy. People were surprised by the severity, not that it happened.

I don't think this comparison is fair. SQLite is widely regarded as one of the best codebases in the world and it has 100% test coverage.

That level of test coverage isn't as meaningful as people think. I'd bet that a significant fraction (> 10%) of those tests are redundant or cost more (in terms of maintenance, for one example) than the value they provide.

I think you should read the full write-up on sqlite testing:

https://www.sqlite.org/testing.html

There's no question that sqlite is one of the most well understood and reliable codebases in the world.

OpenSSL is known to have a poorly documented, spaghetti codebase. The same is not true for SQLite.

Bob Beck during his LibreSSL talk had a good overview of what was thought of OpenSSL https://youtu.be/GnBbhXBDmwU?t=2m11s

"we are all guilty"

I suspect there's a huge code quality difference between OpenSSL and SQLite. Also, keep in mind that SQLite was much easier to fuzz - it's basically one of the libraries that AFL has specific features for (dictionary-based coverage-guided fuzzing).

OpenSSL was never thought to be bulletproof. It is, however, the best of the worst for many scenarios.

But it was bulletproof until proven otherwise.

I suspect Go will never be an ideal choice for creating libraries which are supposed to be used from any programming language. Even while the C-compatible-library-story of Go is developing (which is great), it brings runtime machinery that just isn't there with languages like C, C++ (especially with exceptions disabled) and Rust.

I agree.

I'm curious though--is it true that Go brings its runtime? I'm of the impression that the runtime is only compiled in if you actually use it, but if your library is just `func Add(a, b int) int { return a + b }`, would linking against it still bring in the runtime?

And to reiterate, this is only a curiosity. Even if you could take care to avoid importing the runtime, I wouldn't think that it's worth the while.

You would have to start at languages that are demonstrably better than C at being C for the reasons listed as to why C is still the best language for SQLite.

In the present day context of vulnerabilities it's tempting to blame C. Yet it's still a good choice for many reasons. It's not wise to suggest that everything written in C must be re-written in some other language because of hand-waving reasons like buffer-overflows or off-by-one errors.

Maybe you could prove Rust is a good choice by writing your own SQLite implementation feature for feature? Until then I don't think there's going to be a compelling reason to re-write SQLite because it's written in C.

>It's not wise to suggest that everything written in C must be re-written in some other language because of hand-waving reasons like buffer-overflows or off-by-one errors.

These security vulnerabilities are trivial to fix at the language level but cause extremely negative consequences if exploited successfully. Yet the default stance of C programmers is to not adress them at all. A competitor like Rust absolutely becomes necessary because of the complacency of C programmers. They will then will strongly critizise the newcomer that is built on strong fundamentals as "language of the week" that is chosen by "dumber programmers" that are trying to ride the latest hype train.

Ignoring your comments about other programmers...

Engineering is trade offs. I think the SQLite programmers are well aware of the risk the use of C brings in the context of security vulnerabilities. Given the statistics collected here: https://www.cvedetails.com/vendor/9237/Sqlite.html it seems that their choice of C was not a security disaster, because C. It actually seems well within a risk tolerance threshold that I don't sleep uneasy at night recommending its use in systems. If their choice of language came with unavoidable vulnerabilities I'd expect those statistics to be much worse.

So what does a Rust implementation of SQLite offer anyone? That within 8 years there may be one less SQL injection attack or perhaps < 4 overflow vulnerabilities? Will it have better interoperability or performance than SQLite does now or will have once this new version is done? Will the market care?

I'm not saying that care shouldn't given when choosing to use C in a greenfield project today. I am saying that for many problem domains the risks associated with C are tolerable given a well trained, disciplined team.

And I doubt there's much of a market for an in-memory database system that is going to be feature compatible with SQLite in a few years that is also as performant as SQLite and can deployed on as many platforms. But that's just a prediction... maybe I'm wrong.

I've chosen Haskell on a greenfield project recently for many of the safety guarantees it brings... who knows?

You're making some mistakes here.

If the problem is complacent programmers, switching languages is going to have no effect.

You're also making blanket statements about the attitudes of C programmers which have nothing to do with the C language. That probably means your generalizations are overly broad and not true, not even for a significant fraction of C programmers.

I think you need to use some logic to argue your point, not wild claims.

There is no need for arguments here. C leads to vulnerabilities. That's self-evident. See ISO/IEC TR 24772:2013 for more information.

printf(“hello, world!”);

Please, find the vulnerability in that code, or quit making wild claims

Besides the fact that your code doesn't compile, I would argue that you should deliver real world examples to assess and then comparing it to alternativ solution.

I know, theoretically it's possible to write secure code. Even in C. Experience showed, that 99,999% of people aren't able to do that. And many reasons have its origin in the design of this bad language.

> Maybe you could prove Rust is a good choice by writing your own SQLite implementation feature for feature?

A person undertaking that task would already have a head start in that they could likely use the extensive test suite for SQLite ;-)

But... who proposed a re-write?

Exactly. At this point, the strongest reason for SQLite to remain in C is that it is in C.

It's still not there yet, but getting close. And Rust is really the only language that has come close.

In terms of performance, it's really the only language out there that can claim to be as fast as C/C++. It has a minimal runtime, with the option of no runtime. It has pretty great FFI and can produce easily callable libraries for other runtime-heavy languages.

It's not, however, as broadly compatible as C. I don't think that's a problem for most cases, as most programming is now done for (MIPS|ARM|X86|X86_64), and it can handle those well enough. But microcontrollers and OS-less embedded devices still have a ways to go before Rust beats out C.

And stability is just not there. IMO, that's a good thing. Rust is the best thing to happen to systems programming in a really long time, and there are still tons of ideas with amazing potential benefits. The Rust community has been exceptional at guiding this development. I'm sure some day it will level off, but until then, I'm happy with it changing pretty rapidly.

I don't think "stability" and "changing rapidly" are incompatible notions. Stability implies that code written today continues to work years from now on later versions of the compiler and language. Rust has always (well, since 1.0) striven to provide stability in that sense, despite the rapid pace of language improvement.

> most programming is now done for (MIPS|ARM|X86|X86_64)

but not necessarily posix/windows/etc

having C as a least-common-denominator toolchain which almost all platforms will provide is still useful for the embedded world where sqlite has a huge number of applications..

Rust has a complete #nostd mode where most of it's ergonomics are still usable without any runtime. It also supports pluggable allocators, as well as replacing most of the builtin functionality of other language components (lang items).

It's story around panic and debug handling is also improving, including cutting out formatting code for things like println! and debug! when targetting embedded platforms.

It’s also not as fast as C/C++ in all cases, only in some.

http://benchmarksgame.alioth.debian.org/u64q/rust.html

The biggest differences there are due to lack of SIMD. We expect it to be stable quite soon, and that should close the gap quite a bit.

Do you say that because you have coded those programs using SIMD and rustc nightly, and seen that using SIMD eliminates the performance difference ?

I have not personally done this, but the people who have identified that as the issue, and what we’re stabilizing is the exact same thing as what C and C++ compilers have, so there’s no reason to believe it would be different. Once it’s stable, we’ll all see!

Someone has repeatedly commented on /r/rust/ that differences are due not so much to SIMD but iirc to not triggering the same LLVM loop unrolling.

I think you're confusing loop unrolling and autovectorization. That said, it's kinda moot, once SIMD is stable, we'll find out :)

>> I think you're confusing loop unrolling and autovectorization. <<

Here's exactly what I was told -- "This was achieved by manually unrolling a 10-step loop, which compiler apparently could not optimize."

I would say C will never be as safe as Rust. Just accept it, C is crap. It's just not cool.

You may want to check out Pony :)

In 2000, the only feasible language really was C. C++ suffered from severe portability issues, which have only arguably gone away in the past few years (that said, you still do see systems with pre-libstdc++-4.9, which is still a headache for portability). Managed languages generally suffer embedability problems, and still do to this day.

The conclusion that C remains the best language is a lot harder to support. Certainly, and especially with the level of infrastructure that SQLite developed to harden its implementation, the alternatives are not so much better as to be worth the cost of migration.

> you still do see systems with pre-libstdc++-4.9

Such as macOS ;)

I'd actually say D would be best suited as a replacement because of -betterC mode. You could replace one piece, run all the tests, and then move on to the next piece. Most of the C code could even be reused with minor modifications. In fact, this is why -betterC was introduced. The only reason you wouldn't want to do this is that D is limited to the platforms targeted by LLVM.

> Most of the C code could even be reused with minor modifications

I've done such conversions with C to Rust converter (https://gitlab.com/citrus-rs/citrus), but quickly found out that the style of writing idiomatic in C is part of the problem.

To replace code function by function you're generally forced to keep the same structs and APIs (often even internal ones) for most of the time, and these require you to erase the extra type safety, degrade smart pointers and slices to plain pointers, etc.

If you just do all the same wonky stuff that C does, but only with a slightly different syntax and compiler, you don't gain that much. The value comes from using idioms of a safer language, and that's much more work, and it's especially hard if your hands are tied by the rest of the program being C-like.

I agree. The important thing for a port is to be able to do it one small chunk at a time, so that you know you're not changing behavior. Then after the port is complete, you can start adding all the bells and whistles of the new language. There's no benefit to the port if you don't take the last step.

I'm an unabashed enthusiast for high-level languages that require complicated runtimes and tooling. I thank the heavens sqlite is written in C and if anything is to ever supplant it, I hope that's then also written in C.

What, you don't want to try and figure out how to embed the Erlang VM into your Python project? ;-)

EDIT: Jokes aside, Rust sounds like the most plausible alternative, though it still lags C due to being LLVM-only, whereas C has gcc and... others.

Probably originally written as a response against C++, Java or C# yes. However it’s also been kept up to date because it mentions Swift, so I think they don’t want to rewrite it in Rust or Go. But I agree that Rust might have been a good fit if SQLite had been created today. There is the point that sibling commenter made about small embedded systems though. But some people work on making Rust useful even for microcontrollers so who knows maybe a SQLite in Rust could have been highly portable too?

Sqlite is often just dropped into a project as source code. You can't do that if it were written in Rust.

You can't do that with C code either unless your project is in C (or C++?).

Sure, but for many embedded systems you have to use C. Plus C interoperates well with every other mainstream language. Every platform supports C, so by writing it in C you can support every platform (with very few caveats).

Go and Rust is too heavy and require more dependencies for SQLite. SQLite runs even on the smallest exotic embedded systems.

You can run Rust on atmegas. It doesn't get much more small, but it can get a lot more exotic. https://github.com/avr-rust/rust

Arduino - http://jakegoulding.com/blog/2016/01/02/rust-on-an-arduino-u...

Ok, then it leaves us a performance question, right?

If Rust performance diverges significantly from C, then it's a bug. Bugs do happen! We take bug reports, please file them.

but the point is to have maximum performance. This is most deployed database in the world, the overall impact is incredible. It is not about "significantly" slower.

I said "diverges significantly", that comes on both plus and minus. Right now, Rust can be faster sometimes, but is also slower sometimes. It just depends.

No it means there should _not_ be a performance question.

It does however mean that there is an issue with supported platforms. Rust has support for all the major platforms already, but C is probably the most widely supported language in existence at all.

But then you're forced to use the rust compiler. Sqlite can be included in any project that can use a decent C compiler.

Why would you be forced to use the Rust compiler? You can generate C headers for a static Rust image.

It brings the Rust compiler in as a dependency. I'm not saying you have to use Rust for the rest of the project, but it complicates things in a way that having C source code that you drop into your project doesn't.

I think you two are talking at cross-purposes. You are correct that building the project would require Rust. Your parent is correct that, with a pre-built binary, you don't need any Rust-specific stuff installed to make this work.

Building from source is usual, of course, given the lack of a stable ABI.

The smallest known Rust ELF binary is 151 bytes.

Given your user name, I believe you completely, but can you provide a citation for how this was done?

Here's the HN discussion, which also links it https://news.ycombinator.com/item?id=8869167

Of course, it's not usual Rust code, but no binary that small is usual code, even in C.

The point is to demonstrate that you can strip Rust down to as small as you want/need, not to suggest that every single Rust program is ultra-tiny.

I'd say that at that point, it doesn't even matter that you're writing Rust code. Most of the slimming down comes from cutting out cruft that the linker puts in by default.

Well, it matters in that Rust lets you eliminate this in the first place. Not all languages let you do this kind of thing. If you want to write Ruby, you have a VM, even if your program is `a = 5`.

Rust has no runtime... so... not sure what you mean by that.

I've seen a couple of efforts to attempt to port SQLite to Go, usually with the idea of 'mechnically' porting the existing C code to Go rather than starting from scratch. Someone has already ported the SQLite3 shell: https://github.com/cznic/sqlite3shell .. you can tell the code is transpiled, however, as it's nothing like what a Go developer would write riddled with gotos and artificial variable names :-)

Personally, I think it'd be better to just start afresh with new implementations, not necessarily under the auspices of the SQLite project. Being 'compatible' is reasonable enough and you can just ditch the lesser used or out of date features if you're not claiming to be a 100% clone.

There's a lot of first-party tooling for that, though, since it's how the go compiler itself has been slowly rewritten into go.

The shell might be the least neccessary to mechanically port as it could be easily implemented in the target language and probably doesn't change that much.

The core is where the real compatibility issues lie and where it makes the most sense to translate, as well as where the core vulnerabilities that can be remotely exploited lie.

Go has a bad fitness to this kind of problem as any language with non-deterministic GC would.

Candidates may be: C, C++, Rust and maybe Swift(if the deterministic ref-counting doesnt get in the way).

Of course you can do it in other langs, but it wont be able to compete with products made in languages with better fitness to this.

Eg. CouchDB vs. MongoDB. The first invented the concept, but the later used a language with better fitness for the kind of problem, and therefore was able to create a better product.

I don’t believe that Go is useful for creating a widely-consumable library.

> Rust and possibly even Go would be strong contenders to make a new SQLite-like library/program today.

Go absolutely would not be, one of the purposes of sqlite is to be embeddable in any and all software.

You can create shared libraries in Go, which can be used as any C-implemented shared library. This would make a Go-implemented SQLite equally usable by other programming languages as a C written one.

> Especially given SQLite being started in 2000, C really was undeniably the best choice, and the existing code in C is a strong argument in favor against rewrites.

That's not much of an argument considering the CLR and the JVM are written in C.

They are both weak contenders, for the reasons laid out in the article. Go and Rust have large runtimes with lots of dependencies, and Go is not as performant as C.

Literally every system has a C compiler, except maybe for a very small number of very old and niche systems. Assuming it cound fit on the ROM, sqlite could probably be ported to my z80 calculator, which has an 8 bit processor and 128 kiolobytes of RAM - trivially! C runs literally everywhere, which is something no other programming language can lay claim to. For a tool like sqlite there is no other choice, period.

Rust has an equivalent amount of runtime to C.

Our platform support is limited by LLVM though, this is a good reason for sure.

I was under the impression that the support for unrecoverable errors via panic! creates a runtime structure (allowing for the explicit release of resources) which is not typically present in C programs (unless manually included).

Honestly, "runtime" is just a complex word in general, like, most people refer to C as having "no runtime" even though it does.

Regardless, those are called "landing pads", and you can compile with an option to turn them off. Many do.

Fair, I forgot that Rust has good freestanding support. I still don't think it'd be a good candidate for sqlite, though, for one reason in particular: Rust is not competitive with C on portability.

Yes, agreed. It's not a real answer yet, but mrustc can compile Rust to C, so if this becomes a really huge blocker, we could put in some elbow grease and make it a real answer. The question is mostly "is that worth the time or not?"

Which is why Nim would be a good candidate. Compiles to C so has the benefits of its portability, interoperability and speed.

Oh please please no! Nim is almost impossible to debug at a low level and you have no fine-grained control over its performance or behavior. It's also far less well understood than C, which is more boring and the subject of decades of conservative development. I'm also not sure what kind of support nim has for freestanding programs.

Can you elaborate? I've never had problems debugging it, you can use lldb or gdb to do it.

You have a hell of a lot of control over its performance and behaviour. Are there instances where you found that wasn't the case?

As far as the understanding of C goes, I can agree with that. But we are discussing rewriting sqlite in something else after all.

Maybe you can reach out via email, it's in my profile. I'd rather not tear into Nim in this thread, we're getting pretty off-topic.

Done.

Doesn't it come with GC? That would limit embedding.

That's true. You do have the option to disable it though.

Seeing as Go was suggested above (which is also garbage collected), I figured I would mention Nim too.

Interesting. I love programming in Nim but I'm not sure if I would consider developing something like SQLite in Nim. It still has a few rough edges IMHO.

> It still has a few rough edges

Love to hear what those are and what language you would consider developing SQLite in.

I'm nowhere near technically competent to develop something like SQLite in any language. Richard Hipp is a legend unto himself.

But, I tried working in Nim without GC and out turned out to be a bigger hassle than I expected it to be. GC was supposed to be ref counted and now they're doing memory regions? There is also a -gc:stack which I'm not sure if it works the way like in C++. Then there is the pointer free paradigm. Don't get me wrong, I'm very optimistic about Nim and would be great if it really replaces C. But I feel like it's doing too many things at once.

I personally feel like Nim dev team should stop adding new features every release and work on releasing a solid 1.0.

> That being said, Rust and possibly even Go would be strong contenders to make a new SQLite-like library/program today. At least on the Rust side, the C bindings are excellent too.

Doubtful, rust and go do not run on nearly as many platforms as c and sqlite does. I get everyone wants to use other languages, but this incessant "rewrite in rust/go/$THING" is getting annoying.

I'm going to try to coin a new online discussion rule of programming language posts:

- At some point someone is going to suggest any problems for one language can be solved by rewriting in another language.

In this case "because c" "therefore rust". Its basically godwins law for programming language discussion.

Keyword is "SQLite-like", you are the one bringing "rewrite in rust" up.

Seems like a distinction without a difference to rewrite "sqlite in c" to "not sqlite but sqlite-like in rust or go".

How is it not different? It would be a completely new library. I don't see what the argument against that is.

You're free to do so, as is anyone.

But you'll have to overcome and provide a good reason for why anyone using sqlite should move to your new unproven shiny.

If anything sqlite is a poster child for c done (and tested/validated) right. You'll have to demonstrate a lot more than you can implement some things better in the new language.

I'm going to stop here since the goalposts keep moving. My point is: no one brought "rewrite in rust" up but yourself.

Fair enough, but I guess I don't see a difference between rewrite sqlite in rust and, make something exactly like sqlite, with a c ffi but in rust.

I interpreted that commentary as referencing a new library similar in architecture, like an implementation of S3 that lives locally in the software and is modeled after SQLite. Not “let’s rewrite SQLite” like everyone else seems to have run with.