News for hackers. Stuff that matters.
Re:Neuroevolution of augmenting topologies (NEAT algorithm) by epr on Wednesday December 04, 2024 @09:50AM (#42317951)
> The big problem is that NEAT can't leverage a GPU effectively at scale (arbitrary topologies vs bipartite graphs)

Is that true? These graphs can be transformed into a regular tensor shape with zero weights on unused connections. If you were worried about too much time/space used by these zero weights, you could introduce parsimony pressure related to the dimensions of transformed tensors rather than on the bipartite graph.

Re:Neuroevolution of augmenting topologies (NEAT algorithm) by HelloNurse on Wednesday December 04, 2024 @12:13PM (#42319552)
Another plausible strategy to neutralize arbitrary topologies: compile individual solutions or groups of similar solutions into big compute shaders that execute the network and evaluate expensive fitness functions, with parallel execution over multiple test cases (aggregated in postprocessing) and/or over different numerical parameters for the same topology.
Re:Neuroevolution of augmenting topologies (NEAT algorithm) by Y_Y on Wednesday December 04, 2024 @10:24AM (#42318349)
Or even use CuSparse, if you don't mind a little bit of extra work over normal cudnn.

https://developer.nvidia.com/cusparse

Re:Neuroevolution of augmenting topologies (NEAT algorithm) by PartiallyTyped on Wednesday December 04, 2024 @11:43AM (#42319247)
We can also show that sparse NNs under some conditions are ensembles of discrete subnets, and the authors of the original dropout paper argue that [dropout] effectively creates something akin to a forest of subnets all in "superposition".
Re:Neuroevolution of augmenting topologies (NEAT algorithm) by Rhapso on Wednesday December 04, 2024 @12:42PM (#42319883)
Just because you can pack the topology into a sparse matrix doesn't make it actually go faster.

Sparse matrices often don't see good speedup from GPUs.

In addition, each network is unique, each neuron can have an entirely different activation function, and the topology is constantly changing. You will burn a lot on constantly re-packing into matrices that then don't see the same speedups a more wasteful topology pretends to have.

On the flip-side out narrative of "speedup" is on bipartite graphs crunch faster in gpus and it might not be the same if the basis is actually utility of behaviors generated by the networks. A cousin thread explores this better.