mgpc
Revised Chinchilla scaling implies higher compute requirements for small models
The UK recently announced plans to make this completely explicit, to remove any remaining doubt: "For text and data mining, we plan to introduce a new copyright and database exception which allows TDM for any purpose. Rights holders will still have safeguards to protect their content, including a requirement for lawful access."
https://www.gov.uk/government/consultations/artificial-intel...
I've been an Android user for a long time, but I think this might finally push me to switch to Apple. I'm just disgusted by this.
Some of the tools for interpretability can be useful (particularly for debugging), but I think the broader idea that we always need to be able to understand our own models is basically wrong.
For example, if you want AlphaGo to explain why it made a particular Go move, what kind of an explanation is possible? In many cases the only explanation may be that the move leads to a higher probability of a win. There simply may not be a more “compressed” or “high level” answer. Even human Go players often cannot explain why they choose particular moves, other than references to shape and feel, which is basically another way of saying their evaluation of the move leads to a higher win probability. There are a lot of domains where we may just have to accept that that _is_ the explaination.
To zoom out a bit, our greatest discoveries have historically been about finding the rare places where the universe is computationally compressible. Boiling a kettle is almost unimaginably complex to describe in terms of the interaction of elementary particles. But you can make very good predictions about that process using an equation that fits on a cocktail napkin. There may be other areas in which the universe is compressible only to a lesser extent. The parameters of AlphaGo are vanishingly small compared to the size of the Go game tree, but are very large compared to the equation we can use to predict the kettle. There may be many problems where the best descriptions lie in this intermediate domain, a domain which we have never really had access to before (except via biological brains).
So if learned models give us access to some truths without access to their (human intelligible) explanations, I think we need to just embrace that. If you allow yourself a new way of seeing, you can see new things.
As a user I do feel some privacy regulation is a good thing. There are certainly good things about GDPR (e.g. a lot of third party ad tracking deserves to die). However, I am far from convinced that the GDPR is the saviour some people think it is.
Like many universities, the value of YC to founders is mostly signalling and network. What you actually learn at YC is a minor component of the value (since you could learn most of it just by reading or watching videos)
As YC batches get larger, the signalling value of being a YC company goes down. If everyone is special then nobody is.
As YC batches get larger, I suspect the value of the YC network also goes down. This is maybe a little bit counter-intuitive, but I think a large influx of less highly-selected startups will dilute the network and prevent it from functioning effectively.
Unfortunately YC is not completely aligned with founders here. There is a strong temptation for YC to increase batch size. YC gets 7% of a company for far below market price. So increasing batch sizes makes short-term economic sense. However, if YC increases batch sizes too much and harms its brand, it will eventually break its own model. So increasing batch sizes could be seen as form of spending down the brand equity.
This all hangs on a few premises that may or may not be true: 1) You can't have larger batches while keeping quality constant. 2) YC doesn't directly influence the success of participants that much. 3) If the value of YC drops, good founders will eventually stop applying
I suspect an argument like this has been going on inside YC for quite a while. And the blog post was written by the original founders to try to settle the dispute in favour of the pro-scale faction.
http://www.educatingsilicon.com/wp-content/uploads/2013/10/p...
A lot better for text in photographs. Comparison might be different on dense document text though.
The only path I see with some prospect of success is to limit the total amount of computation available on Earth. (Basically like Vinge's slow zone). If we could engineer a pause in Moore's Law at just the right point, it would buy us time. Maybe we should put a drastic tax on computation at some stage.