Re:Neurons that fire together, wire together, but how?
by blamestross on Monday June 29, 2020 @04:20PM
(#23682284)
It's not that simple. Backpropagating a bipartite graph of nodes works out to a series of matrix operations that parallelize efficiently on a GPU as long as the matrices fit into the GPU's working memory. Running a GA (part of neat) doesn't normally work well on a GPU. The good NEAT algorithms even allow different neurons to have different firing response curves. This inherently defies the "same operation multiple values" style of parallelization in GPUs. The way GPUs work just fundamentally isn't well suited to speeding up NEAT.
Re:Neurons that fire together, wire together, but how?
by jawarner on Monday June 29, 2020 @03:52PM
(#23681836)
You may be interested in this implementation [1] which builds the networks using PyTorch.
Re:Neurons that fire together, wire together, but how?
by blamestross on Monday June 29, 2020 @04:37PM
(#23682615)
It uses pytorch (and I'm probably going to use it), but doesn't effectively leverage a GPU for training.
Re:Neurons that fire together, wire together, but how?
by jawarner on Monday June 29, 2020 @05:45PM
(#23683558)
What do you think is the best way to accomplish this?
Re:Neurons that fire together, wire together, but how?
by blamestross on Wednesday July 01, 2020 @02:30PM
(#23704787)
You don't. You need a different parallelism model than a GPU provides. It could work well on machines with very high CPU count, but the speedup on GPUs is the main reason bipartite graph algorithms have seen such investment.
NEAT would totally be competitive if someone actually gets a version running in PyTorch/Tensorflow