mtravis

On balance, it will probably be better for you in the long run. I am a high school and college dropout, and nearly as old as dirt. I am entirely self-taught with computers, with no academic credentials. I didn't even take CS courses while in college. I am not a failure career-wise, but I believe that a CS, EE, or similar degree, in hind-sight, would have helped me.

There's 3 main reasons for this:

1) there is a lot of conceptual and algorithm stuff that is drummed into people at school that, while possible to learn on one's own, requires time and effort. If I had time to study Knuth in depth, I would love to do so. This subject matter is very often useful--in hindsight, this is my biggest regret for not having studied CS, because there are gaps in my knowledge that would have been easy to fill had I been studying these things long ago.

2) while employers generally do not require a CS degree, it is very often that their employment screening steps will filter you out if you cannot regurgitate much of the stuff described in step 1: algorithms and coding styles taught in college, the way that colleges teach them. No matter how much PR a company claims for itself in looking for intelligence, creativity, and experience, if you haven't internalized the textbook stuff, then you'll frequently get filtered out of interview process quite early. You'll still be able to get jobs, but just not likely the ones with the biggest marquee names with the highest candidate:opening ratio.

3. If you want to do a startup, you'll want to be in the vicinity of people who also want to do a startup. That's much easier when you're in college, and you get looked upon favorably if you come from a prestigious institution. Seed and angel investors like throwing cash at pairs of kids from good schools who can make something that seems useful and is in a hot space.

So, yes, go to college. Study CS, math and also some business stuff like econ. Avoid humanities like plague.

I am having trouble locating any reports to reproducible benchmarks that back up their performance claims.

Cool. The wire protocol is PostgreSQL's, so they provide the clients (BSD).

This you? http://favstar.fm/users/hipsterhacker

Also, the main application is in C++. A python script launches the C++ daemons. Perl scripts are quick and dirty tests and deployment scripts. The main hacking I'm looking for is with C++, and I don't care so much if the other stuff gets re-implemented in some other language.

Thank you, Phil. I'm conflicted about this--I was convinced recently to move away from AGPL having to do with what I was previously unaware of as seemingly legitimate acceptance issues. I feel good about using GPL instead of AGPL.

But I'm conflicted about GPL vs Apache (or BSDish) in the sense that I'm getting the message that I have to bend over backwards just a little bit further before somebody, somewhere might be willing to use my software, maybe. Free isn't enough. I also have to let them fork it, keep it proprietary, wrap their own brand around it, before maybe they might consider using it.

That said, I really want people to use it, and of course help me hack on it. But I'm conflicted.

Good thinking, but I think that shifts the issue--namely, that each inter-thread message uses atomic compare and swap to create the message. I assume there'd be a similar bottleneck on the actor that generates the transactionid limited by the number of messages it can send & receive.

Instead, a friend and I have been thinking about how to perhaps modify MVCC to work with distinct transactionid's per partition. Namely, I'm already generating what I call "subtransactionid"'s for each partition involved in a transaction. And those must be ordered for synchronous replication, so I think the way to implement a variation on MVCC may already be mostly there.

I know I still owe you an architectural doc...fixin' ta, ya know.

For durability, check out http://www.infinisql.org/docs/overview/#idp37053600

I've thought about having an actor library, or minimally, to have the actor basis of InfiniSQL independent of specific workload, but haven't thought it through entirely. I'd be supportive of any efforts to that effect if you want to work on it!

Congratulations!

Do you have benchmark reports?

I think you mean 2PL.

It does really scale, check out the benchmark report on the blog. http://www.infinisql.org/blog/2013/1112/benchmarking-infinis...

For deadlock-prone workloads, it will likely not be as good, admittedly.

I'm considering a variation on MVCC that gets around the single transactionid bottleneck, but the currently implementation is based on 2PL. http://www.infinisql.org/docs/overview/#ftn.idp37098256

For concurrency management algorithms, there are no good ones. Only those that are less bad than others in some cases.

1) persistence: battery-backed UPS and synchronous replication. No WAL anywhere. I'm thinking about ways to do disk-based storage without synchronous IO, to provide decent performance with higher storage capacity

2) no joins supported yet. However, the benchmark that I performed (on the blog) involves 3 updates across random nodes. I designed InfiniSQL specifically to perform multi-node transactions very well, because that's the Achilles' heel of every other distributed OLTP system. I plan to implement joins, but expect them to perform decently for the workload you describe.

Thanks, edited.

Vertica's a data warehouse. InfiniSQL is geared for OLTP. --------- Thanks, jacob019. Please fork/follow on github, twitter if you're into that, etc.

On backlog to fix. But InfiniSQL is for hackers and early adopters at this stage.

The SQL support is documented (http://www.infinisql.org/docs/index/)

I assume these shops have Linux in their environments, including the GNU toolchain. There must be some contradiction somewhere that I'm not aware of.

Based on FSF feedback, I'm going to modify the license to include a Classpath-like exception. The intention is to allow people to write stored procedures that link against infinisql without triggering the copyleft. Only if the source to infinisql itself is modified (and distributed) will the copyleft apply.

I'm curious to know the rationale against the GPL in general (not just the AGPL), and how those shops allow Linux & gnu toolchains in spite of their rule against the GPL.

A few things (I'm the author of InfiniSQL)

1) I include keystore-like stored procedures in the source. They do get/set with integer key and string val. I haven't done thorough benchmarking, but I expect them to outperform the other benchmark I've published, which is quite a bit more complex workload

2) (camus2) agreed, nothing ever dies in IT. But roll back the clock a few years. How much noSQL would come into exisence if there was a free xzySQL that scaled across nodes, was fast, etc. I believe the answer is that there'd be very few network-based noSQL for operational workloads if that had been the case.

3) jwatte: Yeah! Jagged edges too!

4) stephen24: Also, I intend to change the license from AGPL to GPL next time I push out some code. No excuse not to try it out.

5) siliconc0w: There's an architectural write-up at High Scalability: http://highscalability.com/blog/2013/11/25/how-to-make-an-in... -- I believe that the actor model architecture is distinct in InfiniSQL.

6) diwu1989: Yes and no. Yes, MemSQL is more mature. No,

(a) I'm not sure how MemSQL scales horizontally (especially since that was a feature added after v1 of their code was released), and,

(b) MemSQL isn't free software

7) itsbits: for now InfiniSQL is mainly for hackers and early adopters--the dependencies are pretty clearly documented but it requires some effort to work with in its current state

Those are analytics databases, also known as data warehouses. Optimized for batch reporting. InfiniSQL is geared for operational/transactional (OLTP) kinds of workloads.

Here is some back of napkin analysis:

Starting with this benchmark report: http://www.percona.com/files/white-papers/clustrix-tpcc-mysq...

Basically, InfiniSQL does not currently support complex indices, so it can't do a TPC-like transaction.

The maximum throughput on 9 nodes is 128,114 per node per second. I don't know if that's 4 or 8 core nodes. If roughly 10% of transactions are multi-node transactions, then 12,811/node/s for multi-node, and 115,303/node/s for single-node transactions.

I don't know if full redundancy for Clustrix was configured, or a hot spare, so I don't know how many actual usable nodes were configured, but likely fewer than 9. But I don't know the precise number.

Roughly 10% of those transactions are contain records on multiple nodes. Based on 9 nodes, that means about 12811/node/s for distributed transactions combined with 115303/node/s for single node transactions.

InfiniSQL maxsed at over 530,000 multi-node transactions on 12 x 4-core nodes. http://www.infinisql.org/blog/2013/1112/benchmarking-infinis...

That's 44,167 per node.

---------

These were not apples-apples benchmarks, but Clustrix performed about 12,000 multi-node transactions per node per second, along with a whole bunch more single-node transactions.

I don't know how it would perform on the benchmark I used. And I intend to do a tpcc benchmark once InfiniSQL is capable of complex keys (among whatever else it it currently is missing).

Well, array=battery backed controller with a bunch of disks hanging off of it. Actually, there is latency associated with every sync. What I've seen on HDS arrays with Linux boxes and 4GB fibre channell adapters is about 200us per 4KB block write. That is very very good for disk. It's also slower than memory access by many orders of magnitude. This was about 3 years ago. Things are bound to be faster by now, but still not as fast as RAM.

I don't think it's unreasonable to want to write faster than an I/O subsystem can handle. Maybe it's not for every workload, but that doesn't mean it's for no workload.

The distinction I wasn't being clear about was that the storage array (the thing with the RAID controller, cache and disks hanging off of it) is not being saturated if a single transaction log file is being continuously appended to. But that headroom in the array does not translate to more throughput for the transaction log. I don't know if it's an important distinction.

Hi, eksmith. I talk a bit about plans for durability in that overview document.

I promise that I have every intention of making InfiniSQL a platform that does not lose data. I have a long career working in environments that demand 100% data integrity. If I de-emphasized it, it was not intentional.

PostgreSQL doesn't scale for OLTP workloads past a single node. There are a handful of products similar to InfiniSQL (google for the term NewSQL for a survey of them).

And yes, a redundant UPS-backed in-memory cache. I have some ideas on how to do regular disk backing as well (which I'm sure you've read).

And if a more traditional log-based storage layer is added, InfiniSQL will still scale nearly linearly across nodes horizontally. Multi-node scale and in-memory are not dependent on one another. Though I believe that redundant UPS systems managed by a quorum of administrative agents, and provide durability just like writing to disk.

Are you familiar with high end storage arrays, such as from HDS or EMC? They write to redundant memory, battery backed and managed by logic in the arrays. I'm just moving that type of design to protect the database application itself, up from the block layer.

And some people trust their datacenter power--they use pure in-memory databases without UPS already, or they do things like asynchronously write transaction log, which also sacrifices durability. For those groups, InfiniSQL ought to be just fine, without UPS systems.

I totally agree--the architecture I'm calling for is to have redundant UPS's, each managed by InfiniSQL processes--for ultimate availability. If people just want high performance but want to live with a datacenter / cloud provider's ability to maintain power, then I want to support them in that, too.

But I've suffered power outages in data centers, and they'll eventually come around to bite everybody.

Hi, Leif. It's not hard to get to the throughput limits of a single log device, even on a fast array. I've done it on Sybase, WebSphere MQ, Oracle, MySQL, basically on enough platforms that I assume it to be the general case. The log writes don't saturate the array itself--but the log file has a limit to how many blocks can be appended--even on fast arrays. But imagine getting rid of the transaction log entirely--the entire code path. That will be faster even than a transaction log write to memory-backed filesystem.

But I agree that other write (and read) activity going on in the background and foreground, also limits performance--and in fact, I've seen the index write bottleneck that you describe in real life, more-so than simple transaction log writes. So, you're correct.

I've read about Toku, but I really doubt that it writes faster to disk than writing to memory. Are you really trying to say that?

I think it would be great for InfiniSQL to be adapted to disk-backed storage, in addition to memory. The horizontal scalability will also apply, making for a very large group of fast disk-backed nodes.

I think your input is good.

Thanks!

Oh, the infinisql_*.h is because I deploy all header files as part of "make install", when what I really should do is boil it down to just the api header. The api is for stored procedure programming. Yes, I have it on backlog to fix. I give them all that name in case somebody installs to /usr/local (which you probably oughtn't) it's clear what application they all belong to. Yes, I could create a subdirectory, too. But the fix will be when I clean up api.cc to only have to pull in the one header instead of several of them.

#line statements because I get compiler messages from time to time putting things on the wrong line after having imported headers.

Your advice that I create formal academic-style paper is reasonable, and I agree that it should be something that I pursue. Will you follow me somehow (by links at http://www.infinisql.org) so that when such is produced, you'll see it? I can't guarantee getting front page here again, and don't want to be missed in the future, especially as you (and others) have been asking for this type of information.

And, yes, many thousands of transactions per node in memory is what should be expected. But scalability of ACID (lacking durable, as discussed) transactions on multiple nodes--that's the unique part. I'll try to distill that into a paper.

Hi, amalag. Yes, Clustrix is very similar to InfiniSQL (not to mention having been around longer). I believe that InfiniSQL has vastly higher performance at least for the type of workloads that InfiniSQl is currently capable of. InfiniSQL is also open source.

I hope there's room for competition in this space still.

Well, split brain means two parts think they're both active. At that point, nothing can be done (short of manual intervention), because both parts are faulty, and likely, a bunch of others are failed.

But, to avoid split brain in a failover circumstance means that a replica won't come up unless it gets a majority of votes. There's no way for any other replica to also get a majority of votes, therefore only one replica will come active. No split brain. This, of course, assumes proper operation of the cluster management protocol.

(this is CP operation)

Redundancy, quorum protocol, proactive testing, rigorous QA.

And actual 24x7 environments with important data are ludicrously expensive. I expect InfiniSQL to be less expensive since it's based on x86_64, Linux, is open source. But yeah, hardware and environment need to be right.

I'm not sure what I said wrong, but I'm nowhere claiming that split brain (or any failure scenario) can be ruled out entirely in all circumstances--but in practice, split brain is avoided 24x7 for years on end in many different architectures.

It's not magic.

Just like you can't have "enough" storage redundancy. You can have a 100-way mirror of hard drives that will still lose data if you lose 100 disks in less time than somebody replaces one of them.

Oh, I thought you said 2PC (two-phased commit).

Yes, InfiniSQL uses two phase locking. The Achilles' Heel is deadlock management. By necessity, the deadlock management will need to be single-threaded (at least as far as I can figure). No arguments there, so deadlock-prone usage patterns may definitely be problematic.

I don't think there's a perfect concurrency management protocol. MVCC is limited by transactionid generation. Predictive locking can't get rolled back once it starts, or limits practical throughput to single partitions.

2PL works. It's not ground-breaking (though incorporating it in the context of inter-thread messaging might be unique). And it will scale fine other than for a type of workload that tends to bring about a lot of deadlocks.

I'll consider this--meantime, will you please follow me on Twitter and/or sign up for the newsletter, so that you can be informed when the document is ready?

Redundant components and pathing, properly implemented and managed, are what allow enterprise storage arrays, mainframe clusters, Tandems, and the like, to operate 24x7 for years on end. Their myth seems to work.