The only thing I'd fully agree on is that using Go saved us a lot of resources as well. It's an awesome choice for stuff like this that needs to be reasonably performant as well as being simple, understandable and reasonably fast built.
The fact this had to happen in a hackathon suggests a typical disconnect between management and development (and probably poor prioritization by management). Because development knew this was a problem and how to fix it (evidenced by the fact they fixed it), but it took removing management (aka a hackathon) to give development the space to fix it. And now the company pats itself on the back for having the vision to host a hackathon instead of structuring and prioritizing correctly in the first place so this would just get fixed on the clock.
I do think the author's takeaway about the value of simplicity and pragmatism are on point, but that applies not just to code but to management as well.
It often goes like this: Oh snap, we encountered a problem! Lets find a tool, framework, language that promises to solve a similar sounding problem. Now we have a problem with a layer of abstraction on top. Soon to be two problems. Lets find a tool, framework, language to solve both of them ...
It is a spaghetti to the wall approach, where you just throw a bunch of things at your problem hoping that something sticks. And who cares how long it will stick.
Secondly as a developer I think in start-ups dedicated db experts are way underrated. Sure your fullstack devs can cobble together some tables, changing them 15 times a day to accommodate business requests and slap indexes on everything that gets slow. That is also the way to get into trouble once you scale, and instead of reflecting why this is, people reach for the bowl of pasta.
I was no different, when just starting out. I thought my biggest strength was, how quickly I can come up with easy "solutions" for any problem the company had. Took me years to realize how silly of an approach this is.
>It often goes like this: Oh snap, we encountered a problem! Lets find a tool, framework, language that promises to solve a similar sounding problem. Now we have a problem with a layer of abstraction on top. Soon to be two problems. Lets find a tool, framework, language to solve both of them ...
is absolutely real, I've actually seen in happen both in projects I was in, and heard or read about.
This Rich Hickey video is somewhat relevant, IMO:
Tech Video: Rich Hickey: Hammock-Driven Development:
https://jugad2.blogspot.in/2016/03/tech-video-rich-hickey-ha...
You have to watch it at least part way through to get some of the better points in it, although the whole thing is good.
The difficulty I find is, identifying the moment to leave the hammock again in a startup enviroment. To what degree do you need to understand a problem before you take action. If you try to understand it 100%, you'll never get anything out there.
But I'm already very happy that I was able to convince the business side of the company of the approach in a brief talk about it and they now referrer to "the hammock" themselves :)
Agreed. The problem, though, (and I'm painting with a broad brush here) is that the erring tends to be much more on the side of not trying to understand much or at all, of the problem, before jumping into action. I think a lot of it is due to peer pressure and wanting to be "seen" by peers and bosses (and VCs) to be doing stuff, as opposed to really getting things done better in the medium term, even if in the short term it looks like you are not acting but "only" thinking or analyzing or designing stuff. Hence my comment in that post I linked to, about "we have to ship next week". All too common - been there, seen a good amount of that. In fact, this subthread between HN user jacquesm and me just recently, is basically about the same point, although described in different words:
https://news.ycombinator.com/item?id=16774234
>But I'm already very happy that I was able to convince the business side of the company of the approach in a brief talk about it and they now referrer to "the hammock" themselves :)
Cool :)
e.g. I remember using a dedicated jenkins environment to run continuous, scheduled integration tests for my service. When the architect found out, he immediately sent me links to software packages that are dedicated to running continuous tests. I asked whether he had any experience running these new packages and if he would be willing to set it up/maintain it.... radio silence.
Some time ago, I thought it was <easy> to write code to do things. By now, I mostly ponder how I put things into postgres/kafka|rabbitmq|../memcache|redis|.../elasticsearch/neo4j so I can reduce everything to good queries into these systems.
The questions I usually ask myself when evaluating database solutions is:
* Does it accept invalid data?
* Does it change data on error?
* Does the query planner change drastically between minor versions?
* How strong is transaction isolation? can I create constraints, columns or tables in a transaction?
* Does it scale vertically above 40~ CPU threads and 1M IOPS?
The answer to all these questions, for MySQL is "No". You could argue the value of some of them, but a lot of them highlight architectural or development procedural misgivings.
To start with, this is still true today: https://vimeo.com/43536445 Despite being 6 years old, strict mode is still required.
Anything prior to MySQL 5.7 will accept "0000-00-00 00:00:00" as a valid date, 5.7 will not (which is sane) however this means migrating from 5.6 -> 5.7 just got a little harder.
In fact it wouldn't validate /any/ date so it would assume every year was a leap year and febuary always had 29 days.
Regarding the query planner: https://bugs.mysql.com/bug.php?id=74602 This affected my prod mysql/zabbix installation https://support.zabbix.com/browse/ZBX-10652
Regarding performance: This is what I found from my own experience: I was given the task of testing the limits of MySQL, MySQL was the chosen technology and I was no involved in making that decision so- whatever. We were given 10 servers, with 40 cores (2014-2015~) 128G of DDR3/ECC and 8 SATA SSDs in RAID-0 with 1G of RAID cache for write-back. We managed to get MySQL to bottleneck pretty quickly, our queries involved a lot of binary data so we should have been raw IOPS bound, but we weren't we were memory bound. So we replaced the memory allocator with a faster one (jemalloc) and we get a 30% performance improvement. We suspected that the kernel sockets implementation was slowing us down so we compiled a custom "fastsockets" linux kernel. The improvement was around 4%, but we were bottlenecked on memory. After doing a full trace of what MySQL was doing we saw that InnoDB was spinning on a lock quite a lot. I asked if we could try other SQL solutions (MSSQL/PostgreSQL) Postgresql was first chosen because we could just install it, no license and no OS change... it was twice as fast as the optimised MySQL installation out of the box with a stock CentOS6 kernel.
We never even bothered testing MSSQL because PostgreSQL met our performance targets, we were now IOPS bound.
-- More anecdatum:
Regarding data consistency we (tried) to migrate to postgresql for performance reasons in 2014 (my previous company), and failed because MySQL had been corrupting our data very slowly and silently for many years (corrupting meaning not honouring NOT NULL, not honouring type safety, allowing invalid dates, inserting data on error) So far in that actually reimporting the output of `mysqldump` would not work.
Isn't it? I thought that today()-(2018 years, 4 months and 10 days) would be approximately that date? Maybe you prefer +0000 vs just 0000?
'ISO 8601 prescribes, as a minimum, a four-digit year [YYYY] to avoid the year 2000 problem. It therefore represents years from 0000 to 9999, year 0000 being equal to 1 BC and all others AD. However, years prior to 1583 are not automatically allowed by the standard. Instead "values in the range [0000] through [1582] shall only be used by mutual agreement of the partners in information interchange."
To represent years before 0000 or after 9999, the standard also permits the expansion of the year representation but only by prior agreement between the sender and the receiver.[19] An expanded year representation [±YYYYY] must have an agreed-upon number of extra year digits beyond the four-digit minimum, and it must be prefixed with a + or − sign[20] instead of the more common AD/BC (or CE/BCE) notation; by convention 1 BC is labelled +0000, 2 BC is labeled −0001, and so on.'
https://en.m.wikipedia.org/wiki/ISO_8601
Now, if mysql accept, but can't store such a date, I understand that it's a problem.
I do not think MySQL is a technological debt as in 80% startups moving to the different solution is cheap and non-problematic. The LAMP is good enough and quickest/cheapest for the majority of tech companies.
You can solve issues in your application if you know there will be issues like these, knowing the pitfalls and drawbacks of a technology is certainly noble- but if you do then why not choose something that follows principle of least surprise. (There might be reasons).
I would never claim that you should move everything from MySQL if you use it. However if you care about data consistency ensure that you change the defaults, engage strict mode, ensure that your application has no bugs in handling data.
This is actually hard to do correctly, it's overhead in development that you shouldn't be caring about. Just choose something that has sane error conditions and the problem vanishes.
Most companies are not alive "because they chose mysql over something else" they're alive because they have "good enough" tech to get the job done. The job that they're trying to accomplish is the thing that makes them successful.
Uber isn't super huge because it used a specific database technology. It's huge because it's good at marketing, it's providing some value to people.
Still, if you are making choices for yourself, you don't choose mediocrity and hope to muddle through: you choose excellence. Choosing MySQL isn't choosing excellence.
Do you have objective or quantifiable data and references upon which your opinion is based, _and_ is universally applicable to any arbitrary problem that a SQL database might be an appropriate solution for?
If it silently corrupts data over years and gets discovered much later. (As was the case with my previous company, an e-commerce retailer that lost large chunks of order history)
> How do you know that similar or worse problems did not exist in alternative solutions at the time it was implemented?
Because I've been working on database solutions for over 10 years, there are problems in other software but I consider data loss to be worse than any of them. For example the autovacuum in postgresql 8.3 and before was mostly garbage which ended up bloating highly transactional databases. But deleting data when you fail a constraint is worse.
I have 15 years of experience and can built a decent clean system using "boring" technologies. But all the decent paid work where I live is maintaining big balls of mud with tech that was obviously peak hype when it was chosen, and nothing done according to best practices because of that would require sticking with a tech and learning it properly. Its quite frustrating.
Then we have the interview process where people expect me to give up my weekend for their coding test and can't even be bothered to give you feedback afterwards. Or some ridiculous algorithmic nonsense that has no relevance to the job. Getting bored of it all.
And that criteria of technologies could be summed into one sentence. Use something boring. No Hyped programming languages / DB / tools allowed.
Of course some would argue you would be doing it wrong even if it was using old tech / programming / tools. Well yes, but you have a sea of recourse and expertise there to ask for help. Instead of spending energy and time doing figuring it out.
Of course if your company is all about tech innovation, AI or something cutting edge there surely you will have to tried something new. But 80% of those startup aren't.
> Embrace Boredom.
> Let's say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet. Clearly this model is approximate, but I think it helps.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that's existed for a year or less, you just spent one of your innovation tokens. If you choose to write your own database, oh god, you're in trouble.
> Any of those choices might be sensible if you're a javascript consultancy, or a database company. But you're probably not. You're probably working for a company that is at least ostensibly rethinking global commerce or reinventing payments on the web or pursuing some other suitably epic mission. In that context, devoting any of your limited attention to innovating ssh is an excellent way to fail. Or at best, delay success.
By choosing at most one new thing, you can better control for how your stack should work and how you expect it to respond to certain unexpected circumstances, which means you should be able to more effectively solve issues as they crop up than you'd be able to if you were using multiple new technologies.
But then ... you have to find, attract and hire good developers. That's already difficult, adding an extra layer of 'boring technology' will make this task even more challenging.
The key, the whole key, and nothing but the key so help me Codd.
Also said as... "In Codd we trust."
If none of these DB jokes mean anything to you, take a DB concepts class at a CS university. There's a lot of great research going back 50 years and you can learn a great deal about why things are the way they are (tuple algebra and calculus). And before changing anything for something you think may be better, you should fully understand what you are giving up.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86....
Nontransactional DDL alone is sufficient to classify it as a toy DB for me. Yes, I've been personally bitten by it.
[1] https://stackoverflow.com/questions/4711447/oracle-ddl-and-t...
And https://mysqlserverteam.com/new-defaults-in-mysql-8-0/
Previously on HN: https://news.ycombinator.com/item?id=5122299
Personally, I only use MySQL and derivatives where I have to (basically WordPress.)
[0] - https://dev.mysql.com/doc/refman/5.7/en/cannot-roll-back.htm...
... have dedicated hundreds of engineers and millions of dollars to nothing more than keeping MySQL up, running, and not crapping the bed every time someone looks at it funny. If you can afford that resource expenditure, by all means go nuts with MySQL. Most companies can't and would be far better served by something which doesn't need that amount of handholding to serve its basic purpose.
Following on from that, I suspect a lot of large companies use MySQL because they always have, not because it's actually any good. For example, Basecamp used MySQL while I was there, but I never met a single Sysadmin there who would use it over Postgres if they were to start a new project.
MySql and PHP are good. They do the job they were designed for in a cost effective way and of course that means there will be trade offs.
Sure, you can knock up a contact form in it really quickly, but that ease of use hides significant dangers.
I've programmed it, it's a terrible language.
I think by the time Postgres sorted itself out into a more user/admin-friendly system (which is still fairly recently, really), MySQL had pretty much conquered the "quick and easy" mindshare and was deeply embedded almost everywhere.
And if you've spent millions of dollars architecting your systems such that MySQL's flaws aren't killer issues, there's very little financial benefit to switching, I guess.
Eventually Postgres caught up in most things, and the delay was in some part because of implementing those features "correctly" and with more thought, but it's still a delay that hurt the uptake in the early days.
As has PHP. "popularity" isn't really evidence for it being a "serious" tool, is it?
Most E-Shops will be fine running MySQL or MariaDB.
The one thing PG excels at however is that you can tune it much more to your workload and it allows tuning the workload much more finely than MySQL/MariaDB. That and the ability to extend PG arbitrarily (try adding native functions to mysql without recompiling) via the C-FFI offered. You can write and define your own index methods that let you use an index that is perfect for the workload or you can add a new data type to support a new input with validation.
You can sink a lot of work into getting the most out of a PG database, MySQL not so much. But again, for most people MySQL will provide the same (or even better) performance than PG. (I still trust PG over MySQL after MySQL nulled out all entries of a table with only NOTNULL columns after a nasty crash)
Like, how often one needs to rollback a DDL statement? I did that like... never.
And, what is the use case? Like, you added a column to a table by accident? Well, that will not break anything, so no harm done.
That is way different from regular dml rollback which may recover 1bn records and save your life :)
For example, you had a "color" column on a table, for a new feature youre now adding the ability to have multiple colors. You're going to create a new column, create a new table, populate that table, and drop the old column. If anything fails during that process you'd like to be able to roll back.
There is a concept or "forward-compatible change". Basically, you don't do things that will break your software.
Example, you don't add a NOT-NULL column unless you can give it a good DEFAULT value, to make it work.
Also dont' drop columns until the software is ready for it, etc.
If you have a decent ORM, it will compare your "how it needs to be" sql schema with the "how it is" schema. Then it will generate appropriate "ALTER TABLE ...." "CREATE INDEX " etc statements. Note that this is automated and you never need to type SQL statements to achieve that.
All together in the last XXX years, I did not really need to do a rollback on a dml statement.
I don't mean it to be conformist, but it's easy to forget some things are actually hard when you are very clever or old enough to forget how it was like when you were still learning too.
Not that I disagree, but to be fair, I've seen plenty of tech ignorance with experienced and older engineers as well that has been pretty crippling.
This is just trolling
What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) they move to some columnar DB no one has heard of. Nevermind that postgres and a reasonably priced DBA and a little thought put into their data model/queries could probably handle all their issues.
Sorry for the snark, cheers on a successful product.