It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferring, the design of the hardware for training is probably similar.
Unlike Google's TPUs, Tesla have a specific use case for the hardware (computer vision for automotive), and maybe than means they can further optimize the computation pipeline with their own specialized hardware.
- It's under 100W so they can retrofit into old cars
- lower part cost, so they can do full redundancy with doubling the parts
- they estimated that 50 TOPS is needed for self-driving
- lower latency with batch size of 1 compared to TPU's 256
+ GPU for post-processing
- security: only code signed by Tesla can run on the chip
- at the time (2016) there was no neural net accelerator chips
- some part's are built from bought IP (so not reinventing them) Probably things like the 12 ARM CPUs, LP DDR4 memory, video encoder, maybe the separate post-processing GPU too...
- physical size of the board is small
- performance example: on CPU 1.5 FPS, on GPU (600 gflop) 17 FPS, on Tesla's NN accelerator 2100 FPS
- Besides the convolution even the ReLU and Pooling is implemented in hardware
- Paying attention to the energy efficiency down to the arithmetic and data type usage.
- The silicon cost is less than their previous hardware (HW 2.5)
- old hardware 110 FPS new one 2300 FPS
- 144 TOPS compared to NVidia's Drive Xavier 21 TOPS
In other words, Cloud TPUs could be the same architecture than Edge TPUs but scaled to an higher frequency and more packed.
I guess we need sources to confirm.
Also, training requires a lot more RAM per unit of compute, since it needs to store all past layer activations, whereas for inference, that is unnecessary.
As far as I know, no player who has developed dedicated ML hardware (as opposed to using GPU's) uses the same hardware for both inference and training.
I would expect their training hardware to be something specifically aimed at optimizing memory bandwidth to support distributing training of their “shared” hydra feature. It’s interesting that the shared hydra feature extractor is able to converge as they keep adding more and more output predictions under a training regime of interleaving asynchronous updates to the model from different predictor networks ...
Seems to me the formula they are pursuing with custom hardware might be to support a strategy of 1. keep adding more predictions based on same feature 2. Increase the span of time represented by batches used to train the recurrent networks
Both pursuits seem very data efficient in terms of the amount of training data they could conceivably collect per unit time of observation ...
Custom hardware with a problem specific memory architecture aimed at efficiently supporting training with very large rnn time slices could be developed that’s more about “make it possible to train this proposed model at all” rather than “make it faster/cheaper to train existing common model architectures”. When custom hardware is required to make it possible to train the model they want, the validity of the hardware development cost bet might end up being more about the effectiveness of the model they think they want than it is about maintaining general purpose performance parity vs any off the shelf hardware options ...
Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's GPUs are have to be reasonably good at training across different NNs. You could have a chip that's exceptional good at training one/two specific types of tasks.
For most companies, this would be a bad idea. However, Tesla knows how to produce hardware.
Not sure if it’s still the case today, but previously Tesla’s training was done on-prem and with their own in-house Tensorflow clone.
And yes, if you get TPUs from GCloud, you are likely to be working with their engineers to get things working. Those engineers tend not to have much business conflict of interest, though. They want to help you because your problems are likely more interesting than what they’d otherwise be assigned.
By hardware-hours, Tesla is hardly one of the top companies training deep networks. Planet Labs (satellite imaging), Netflix, Pornhub, to name a few.
What's the info they'd leak to the Google consultants? How much data or TPUs they're using? This is practically public information.
https://research.netflix.com/research-area/machine-learning
Here's another article on the subject:
https://becominghuman.ai/how-netflix-uses-ai-and-machine-lea...
Characteristics that make content successful.. here again, it's mostly decent quality content plus marketing. I doubt the scripts are reviewed by NNs.
I have no idea about the network and delivery stuff, but I guess a well designed network can take care of most of it.
My strong impression is that, similar to other well known cases (Uber, WeWork) Netflix is a mostly traditional company (a media company) that very strongly wants to be seen as a tech company.
Its a bit odd that legally one can rent out physical disks, but there is no corresponding way to legally get permission to rent out streaming content without negotiating with the rightsholder. But thats how it is...
It's hard to think of a good fair regime to do this under.
At least with a physical disk, there's a maximum reasonable rate that you can turn the disk around between users and you need to have enough copies for whatever the lifecycle peak demand is.
We do have an audio compulsory licensing system for things that are purely songs. But with video works, there's not a clear boundary for "how big" the work is-- how do you treat 30 hour anime series vs a 5 minute Pixar short? How do you treat continuing medical education videos vs. fluff amateur made content? Etc.
Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would be a good idea to upload Tesla's proprietary data into Google Cloud even if it did not. Cloud, after all, is just somebody else's computer.
I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099, it doesn't apply to TPU hardware, as is explained in the comments there.
> TPU-like stuff is ~10x the energy efficiency of GPUs
10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, that still doesn't make it a good idea to build your own chips because there will soon be many third party options to choose from. Doing your own chips is a bet that you will out execute dozens of companies ranging from startups to industry giants. Simply taking your pick of the best commercially available options is likely to be a better choice in the near future.
As for single-source, the models are written in PyTorch, not TPU machine code, and they're pretty standard models anyway (e.g. Resnet-50). They can easily transfer to other hardware if necessary. There's not a ton of lock-in there. It doesn't justify the massive costs of ASIC development just to avoid this nonexistent lock-in and imaginary TOS clause.
GPU advantage: more refined ecosystem and you can buy them for $<1000 or get laptops with them built in, and if NVDA has sweat more software engineering blood and tears than GOOG into your model's functions, it will run better on them
TPU advantage: Colab has a free tier that lets you play with them at no charge and if GOOG has sweat more software engineering blood and tears into your model's functions, it will run better on them.
All IMO of course. And deep down it can get more complicated than that, but I salute GOOG for being the first company to ship competitive AI HW, doubly so at scale.
TPU? Seems like it has a lot of potential, but not for people directly competing with them.
Waymo's Laserbear lidar? Seems like it has a lot of potential, but not for AV companies directly competing with them.
Google's playing this game pretty fiercely... which given their size is pretty bad/daunting.
could you share the stats on this? Google told me to use a K-80 for training.
Can it be true? Then again, Apple's app store behaviour seems to suggest such demands are tolerated. Antitrust is really asleep in the US, isn't it.
Also, vendor lock-in is a huge challenge in the cloud space. I don’t think Tesla would be comfortable with the fact that all their training data sits on a potential competitor’s datacenter.
Also, Tesla is/was planning on running these chips even while not in autonomous mode in order to spot new scenarios which it can then record what the car sees and what the human does in order to collect a lot more unique road scenarios to train their models with.
[0] Credit Robotbeat: https://news.ycombinator.com/item?id=19729743
Increasing their compute per watt allows them to get more compute per watt, so they will pursue it lots, low speed energy savings are a way of making the range extension seem like a big deal when it isn't (maintaining highway speeds requires tens of kilowatts).
Power efficiency does make a range difference and is worth seeking, but Tesla is exaggerating this, IMO, to conceal one of the ways that the churn-heavy cycle for FSD has imposed organizational costs (an entire chip-level hardware program, in this case).
An actual working system might slow them down, but even then, better hardware is one of the ways to make it better.
There's commodity computing of comparable throughput that other automakers can incorporate without too much cost, but it doesn't fit into the existing TM3 vehicles because it has a larger footprint in power, space, and cooling than HW2.
Here, Tesla spends a bunch of money to develop HW3 and overcome this problem, but the advantage in a new car design between HW3's efficiency and commodity hardware is limited.
The latest OTA finally brings a hardware v3 only feature, traffic cone visualization, and traffic cone automatic lane change.
I can see the case for doing their own edge hardware for the cars (barely), but I really don't think doing training hardware will pay off for them. If they're serious about it, they should spin it out as a separate business to spread the development cost over a larger customer base.
Also, I'm really curious whether the custom hardware in the cars is benefiting them at all yet. Every feature they've released so far works fine on the previous generation hardware with 1/10 the compute power. At some point won't they need to start training radically larger networks to take advantage of all that untapped compute power?