News for hackers. Stuff that matters.
Re:Score matching with Langevin Sampling: a new contender to... by fxtentacle on Thursday September 03, 2020 @02:15PM (#24366951)
In short, this is a super cool approach to replace the discriminator in GANs with something that doesn't need to be trained and provably converges to the correct result.

"256×256 images cannot be done reliably without 8 V100 GPUs or more!"

That's quite sad because that means this approach is far out of reach for any hobby researcher and for most universities.

Re:Score matching with Langevin Sampling: a new contender to... by alpineidyll3 on Thursday September 03, 2020 @04:15PM (#24368376)
I think there's a pretty high chance that tricks could be brought to bear to greatly reduce the costs of what's being done here (by OOMs). There's a lot of handles for tweaking.

In particular the basic crux of this approach is a Monte-Carlo annealing of the score function. In every field of science, Monte-Carlo sampling entails a huge pre-factor cost, and in every case it only reaches it's greatest potential with importance sampling, which hasn't been applied here.

This is basically the 'brute-force' version of some future approach which would replace the diffusion kernel with another process that allows one to avoid sampling huge volumes of function space which are irrelevant for your desired P(X). This would introduce dependence of the training process on the sampling pre-conditioner, but ultimately be required for highest performance.

I suspect that these authors are already thinking of how to use invertible flows to this effect.

Re:Score matching with Langevin Sampling: a new contender to... by salty_biscuits on Thursday September 03, 2020 @10:46PM (#24371242)
I've always had this idea that you could do a dumb version of annealing by just progressively increasing the size of samples used in the minibatches. SGD gives you noise on the gradient which should get smaller as you increase the batch size. If you just adjusted the batch size with some sort of MH accept/reject logic you should get a better optimzer for mode seeking than just vanilla SGD. Like all my ideas though it has probably been thought before or is just wrong :) too damn lazy to try it as well.
Re:Score matching with Langevin Sampling: a new contender to... by fxtentacle on Thursday September 03, 2020 @07:58PM (#24370242)
The way I read it, they actually need the randomness introduced from full Monte-Carlo sampling to make sure that they explore the data distribution sufficiently far in every direction. So it might be that for this approach, importance sampling would distort the results and re-introduce problems like the chance for divergence that GANs are fighting with.
Re:Score matching with Langevin Sampling: a new contender to... by alpineidyll3 on Thursday September 03, 2020 @10:33PM (#24371194)
The miracle of importance sampling is that it doesn't formally ever prevent you from visiting the rare regions, it just might take longer. What can occur is that you will get what appears to be a well-converged estimate of an answer but really you weren't fully ergodic. Such is the curse of dimensionality. No free lunch theorem, yadda yadda.
Re:Score matching with Langevin Sampling: a new contender to... by marmaduke on Thursday September 03, 2020 @03:22PM (#24367733)
I just checked (my university account with Dell) and a dual Quadro RTX8000 (so 96 GB GPU memory) is around 10k. A V100 is 32 GB so so a 6xRTX8000 could probably cut it at price of 30k. It’s not cheap but the price of a six month of a post doc in my country.
Re:Score matching with Langevin Sampling: a new contender to... by deeeeeplearning on Thursday September 03, 2020 @03:01PM (#24367494)
AWS has V100's available so most Universities with a decent budget should be able to swing this.
Re:Score matching with Langevin Sampling: a new contender to... by notsuoh on Thursday September 03, 2020 @07:22PM (#24369964)
Not even that, spot pricing on an 8 GPU instance (the 16xl I believe, the larger one has the same number of GPUs but more memory per GPU) is something like $6/hr. I use this for personal projects sometimes, get all data in S3, a good launch template, and then spin up a spot instance and be super efficient about training quickly. I've even run evals on a separate, cheaper, machine so the 16xl can spend all its time training. It's still not "cheap", but $50 for 8 hours of training on a machine like that with $64k of GPUs on board is really not bad.
Re:Score matching with Langevin Sampling: a new contender to... by bravura on Thursday September 03, 2020 @08:03PM (#24370281)
If you wrote a blog post about how to streamline this, I would read it and upvote it.
Re:Score matching with Langevin Sampling: a new contender to... by belval on Thursday September 03, 2020 @03:09PM (#24367581)
24$/hour for 8 V100 to be exact.
Re:Score matching with Langevin Sampling: a new contender to... by sudosysgen on Thursday September 03, 2020 @05:39PM (#24369183)
It's interesting how renting 8 GPUs costing 7000$ each costs about the same as renting the services of the average US worker.
Re:Score matching with Langevin Sampling: a new contender to... by smabie on Thursday September 03, 2020 @10:17PM (#24371112)
24/hour comes to a salary of $49,920, working 8 hours a day, 5 days a week, no vacation, no holidays.

Meanwhile the GPUs cost $56,000 total. So the numbers aren't really very far off.

Re:Score matching with Langevin Sampling: a new contender to... by sudosysgen on Thursday September 03, 2020 @11:13PM (#24371364)
They kind of are, these GPUs last multiple years. Anyways, it's just an interesting observation.
Re:Score matching with Langevin Sampling: a new contender to... by smabie on Friday September 04, 2020 @01:29AM (#24371980)
I mean you also have to take into account electricity, which I assume is pretty expensive.
Re:Score matching with Langevin Sampling: a new contender to... by sudosysgen on Friday September 04, 2020 @12:59PM (#24376704)
It's only about 2.5kW, so around 1-2$ per hour for the full system
Re:Score matching with Langevin Sampling: a new contender to... by fxtentacle on Thursday September 03, 2020 @07:54PM (#24370216)
Except that cloud-ified V100s are significantly less powerful than if you have direct access to the hardware. Last time I checked, in AWS they're actually external devices mapped in over GBit ethernet, which is significantly slower than the 8GB/s that PCIe x4 has.
Re:Score matching with Langevin Sampling: a new contender to... by nl on Friday September 04, 2020 @02:58AM (#24372431)
This isn't true. AWS V100s are standard V100s.

I think you are confusing this with AWS Elastic Inference.

If you use AWS Elastic Inference, then you get networked attached devices. But these are Amazon's own (non-NVidia) devices and only used for inference, so it's not really comparable.

https://aws.amazon.com/machine-learning/elastic-inference/fa...

Re:Score matching with Langevin Sampling: a new contender to... by p1esk on Thursday September 03, 2020 @10:57PM (#24371285)
I routinely switch between AWS 8x V100 instances and on-premise 8x V100 servers and I observe no difference in speed (time per epoch).
Re:Score matching with Langevin Sampling: a new contender to... by Reelin on Thursday September 03, 2020 @11:24PM (#24371426)
Presumably that depends on maximum PCIe bandwidth consumption before your workload bottlenecks elsewhere? A 2018 benchmark (https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-...) seems to indicate that x8 isn't generally a bottleneck for common (at the time) workloads. x8 is a far cry from the claimed gigabit ethernet though!
Re:Score matching with Langevin Sampling: a new contender to... by p1esk on Friday September 04, 2020 @01:10AM (#24371878)
AWS is tricky in terms of how storage is provisioned - I don't remember details, but it's easy to put your datasets on storage that is connected to your GPU servers over 1Gb link. That could easily become a bottleneck. Datasets should live on Elastic Block Storage or something like that, over high speed links. Again, it's been a while since I looked into that, so I don't remember the details.
Re:Score matching with Langevin Sampling: a new contender to... by Reelin on Friday September 04, 2020 @02:20AM (#24372236)
The earlier comment claimed that the GPUs (!!!) were located elsewhere on the network; I suspect that the scenario you describe is what they intended to refer to.

(IIRC AWS offers compute optimized instances with a volume that's guaranteed to be backed by blocks on a local NVMe drive.)

Re:Score matching with Langevin Sampling: a new contender to... by nl on Friday September 04, 2020 @03:00AM (#24372444)
I think they are confused with AWS Elastic Inference. That is a different thing which does have network attached accelerators:

Amazon Elastic inference accelerators are GPU-powered hardware devices that are designed to work with any EC2 instance, Sagemaker instance, or ECS task to accelerate deep learning inference workloads at a low cost. When you launch an EC2 instance or an ECS task with Amazon Elastic Inference, an accelerator is provisioned and attached to the instance over the network.

https://aws.amazon.com/machine-learning/elastic-inference/fa...

Re:Score matching with Langevin Sampling: a new contender to... by marcinzm on Thursday September 03, 2020 @11:39PM (#24371490)
Citation? AWS GPU instances are cpu+gpu+memory in fixed ratio packages so I don't see why it'd be over ethernet.
Re:Score matching with Langevin Sampling: a new contender to... by twanvl on Thursday September 03, 2020 @08:42PM (#24370585)
This is not just replacing the discriminator. In this DSM-ALS method you learn a score function instead of a generator, and to generate a sample you need to evaluate the score function multiple times (it essentially gives you a direction to move in).
Re:Score matching with Langevin Sampling: a new contender to... by scoopertrooper on Friday September 04, 2020 @05:11AM (#24373179)
Oracle clouds are offering 8 A100s for $3.05 an hour! I can only assume it's some horrible trap, but it sounds like crazy good value!

https://www.oracle.com/corporate/pressrelease/oracle-cloud-i...

Re:Score matching with Langevin Sampling: a new contender to... by rajnathani on Sunday September 06, 2020 @01:53AM (#24389303)
No, they've listed "$3.05" per GPU. For 8 A100s that would be 8x$3.05 = $24.4/hour.
Re:Score matching with Langevin Sampling: a new contender to... by scoopertrooper on Monday September 07, 2020 @02:59AM (#24397209)
Ah missed that.