As the complexity of the projects I work on has increased (especially in the time since I became a part of a professional dev team), I've come to realize that most of that tricky, complicated-looking code I had read years earlier was actually kind of the easy way out. When you are just trying to get stuff done, the complexity of your code mirrors the complexity of the project. There's no effort put in (and often no time to do so) to create a smoother interface to shelter the code from the complexity of the task at hand.
It's much more interesting to me now to be faced with a complex task and to figure out how to make the code simple and clear. Elegant, well-designed libraries/APIs belie the challenge in writing them. The code looks so simple that it feels immediately obvious as you're reading it. I've come to realize that reading a code base that seems dead simple -- so simple anyone could instantly understand it -- is actually often inversely related to the difficulty in creating it.
Yes, necessary complexity does exist, and sometimes there's no way around doing something fairly nasty in your code. But as an ideal to strive for, I find 'simple' to be fascinating.
That’s probably the main reason why so many C++ codebases are unnecessarily complicated. People look at what standard library developers did, and emulating it.
The complexity of the standard library is actually justified: containers need to scale from 1 to 1E9 items and more, algorithms need to work with whatever broken classes users throw at them, the whole standard library needs to be extremely generic.
None of these requirements apply to code of most applications, yet I have seen many C++ projects where people designed their code the same way the standard library is designed, without thinking about the reasons. The result is template heavy code which takes long time to compile, and hard to modify or debug.
This is also true of "its in the repo" and that doesn't get hit by "find and replace".
I don't think it should be done for EVERY case, rather the cases where clarity is replaced with something that is ambiguous but meets another need. Performance hacks are notorious for being "ugly" and "magical" and having a readable companion piece would make sense.
To clarify I don't think it replaces "prose" but should exist along side it, and be mixed into it.
It's very much not! The thing "in the repo" is of exactly the same vintage as the thing in the next commit. The thing "in the comments" is preserved in amber next to a growing thing that no longer matches it. Inevitably to understand the thing "in the comments" you have to go back to commits of the same age.
Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance.
It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster?
I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations!
If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated.
Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for.
(Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)
We're trying to use TPUs to fine tune GPT-2 1.5B. The model takes up 5.8GB memory, which is well over half of a TPUv2 core (8GB). It always OOMs when I try to do a training step, due to the gradient calculations requiring memory. It even OOMs on a TPUv3, which has 16GB per core. I've tried using bfloat16 (which ought to cut memory usage in half) and using Ada optimizer (which should be no more expensive than plain old SGD). Yet if I colocate the gradients to the same core as the model, I always OOM. (Colocation just means "don't use any memory except the memory physically on this one core.") With colocation off, I don't OOM, and I do see some speed gains using all 8 cores. But it's no more than a factor of 2x, and in fact closer to like 1.15x (i.e. it's roughly equivalent to just using larger batch sizes on a single core). And I don't understand why I'd OOM in the TPUv3 case; even with float32, the model is only using 5.8GB out of 16GB. Are gradient computations really taking up more than 10GB for the optimizer? (That leads to https://github.com/cybertronai/gradient-checkpointing and such, but I haven't tried it yet.)
If I try the same experiment with a much smaller model (117M, or about 13x smaller), I can successfully colocate the gradients onto the same core as the model. And when I use all eight cores, I'm able to get 1225 tokens/sec (roughly 1 example per second, since 1 example = 1024 tokens for GPT-2), vs the standard case of around 400 tokens/sec when using only one core. But that's still "only" a 3x speedup.
So when I see "50-100x increases," alarm bells start going off. I'm missing something fundamental here. Either you are getting 100x speedups, or I am somehow missing something fundamental.
People have even started asking me for answers regarding the TPU case, and I'm forced to be like "Yeah! I expected TPUs to be so much faster too. Everyone says they're getting 100x speed gains. Yet we're 11x slower than the GPU case, and here's a notebook showing a 11x slowdown."
https://github.com/shawwn/gpt-2/issues/5
I'm suspecting that memory bandwidth is the bottleneck here for large models. This paper even pretty much says "GPUs are more flexible and faster when memory bandwidth is an issue": https://twitter.com/mosicr/status/1196749286815481856
The closest I've come to finding an actual example of a speedup to aim for is this: https://github.com/imcaspar/gpt2-ml
They used a TPUv3-512 pod to train a GPT-2 1.5B model to 99k steps in 50 hours. If you work out the math, that's about 1 example per second. We're getting about 0.08 examples per second on a single TPUv2 core. So yes, it's a big speedup (12.5x) but certainly not 50-100x. Yet it has 64x the cores as my TPUv2; why isn't it 64x faster? And we're only using 1 core; why not 512x faster?
I have also tried this on TPUv3-8, and we're getting about the same examples/sec, further increasing the plausibility of the theory that memory bandwidth is the bottleneck.
A perfect C++ program that outperforms a simple looking python program which does the same thing isn't worse because it's harder to read or to write, or better because it's faster, it's just different.
If you need to optimize for readability, so be it. If you need to optimize for performance, so be it.
If you write code executed trillions of times in a trillion places, where power use and the expense of hardware is dependent on the performance of the software, I think it's okay for it to be as complex as it needs to be to be very optimal, at the expense of readability and other things. It should be well annotated, sure. That's always a reasonable thing to expect except in rare, extreme cases where you don't even have the time to type out the comments.
It's important to weigh the relative importance of simplicity vs speed vs memory frugality vs portability vs maintenance difficulty (dependencies and tool choice factor in here) and so on to determine the best approach.
Dogma about how best to code in a general sense is only helpful if you don't want to think or talk about what's the best fit.
To some people, C is simple. To other people, C is full of "complex and subtle" sources of pain.
To some people, a VM with a GC and a JIT is "complex and subtle". To other people, using a HLL with those features enables them to write programs which are much simpler and clearer.
These positions also change over time. A JIT is a fairly standard technique in 2019, but in 1973 it was still very much in the research phase. Even the idea of writing a kernel in such a HLL as C was once revolutionary.
I have an SDET friend who calls these types of universal truisms "apple pie", as in, you hold a meeting and say "Apple pie is good, right?" and everybody nods and says "Yes!" and then goes back to work and absolutely nothing was learned or decided.
I tend to think there is always a better way, say creating a module that contains both implementations as alternatives, but if this is the worst wart on the codebase it is probably a good codebase.
Just leaving giant blocks of commented out, trivial to recreate code as the system evolves is annoying. But I do think that the rule of not leaving old code in as comments is one where the bar to breaking it shouldn't be too terribly high. Personally I've gone overboard following it in the past, and all it did was hurt productivity. A few comments blocks aren't the end of the world.
Rebasing and other means of moving patches (email etc, adding signed-off-by) can cause the original commit id to become invalid, and eventually unretrievable.
I think there is a more universal truism here - that "complex and subtle" are sources of pain, problems and headaches.
I want to write "cool" and "magical" code as much as the next person, but that's the stuff that I look at later WTF because I am no longer in the same state of mind. Clear, simple, straight forward, plain as day are better than anything else.
And if you do have to do something "magical" something "odd" or hard to understand for the love of god please leave notes explaining what, why and how you did what you did. And if you are replacing a "clean" and "readable" version leave it there commented out. Sure it is "in the repo" but almost no one ever looks and you are just making it harder for me to figure out the original intent of whatever was there.