Researchers unveil a pruning algorithm to shrink deep learning models

Re: Researchers unveil a pruning algorithm to shrink deep le... by gavinray on Friday May 08, 2020 @08:56PM (#23121181)

Why can't you model each neuron as an individual object with a unique UID in code?

I am fortunate enough to have a friend that's fairly accomplished in ML and a really nice guy. He spent a video chat + code session with me where he helped me build an NN from scratch in Typescript, we replicated the network in the paper here:

https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

Here's the Codesandbox link, if you notice the console output it (approximately) matches the figures from the paper above:

https://codesandbox.io/s/node-typescript-vqszi?file=/example...

I don't really understand much about ML or neural networks, but from a code perspective, if you have these data structures:

    class Neuron {
      id: number | void = neuronUidGen.next().value;
      incoming: Connection[];
      outgoing: Connection[];
      rate: number;
      bias: number;
      output: number;
      error: number;
    }

    class Connection {
      id: number | void = connectionUidGen.next().value;
      from: Neuron;
      to: Neuron;
      weight: number;
   }

Why couldn't you manipulate these however you see fit? There's a Layer implementation in there that's commented out because it doesn't function right that could roughly accomplish something like this theoretically, right?

[ Parent | on HN ]

Re: Researchers unveil a pruning algorithm to shrink deep le... by cs702 on Friday May 08, 2020 @10:08PM (#23121575)

You could. But that's not the hard part. The hard part, for the foreseeable future, is that if you write your code like that, you will find it very difficult to get decent performance out of existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. (Oversimplifying a bit.)

[ Parent | on HN ]

Re: Researchers unveil a pruning algorithm to shrink deep le... by gavinray on Saturday May 09, 2020 @12:28AM (#23122324)

But the Neuron and Connections are just metadata objects like pointers that hold arrays, I don't understand.

Surely in PyTorch/TF, the ND-Arrays of NN's must also be associated to an object reference?

> existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape.

    const dot = (inputs: number[], weights: number[]) => {
      let total = 0;
      const length = Math.min(inputs.length, weights.length);
      for (let i = 0; i < length; i++) {
        total += inputs[i] * weights[i];
      }
      return total;
    };

  activate(input?: number): number {
    if (!this.incoming.length) {
      this.output = input;
    } else {
      const inputs = this.incoming.map(connection => connection.from.output)
      const weights = this.incoming.map(connection => connection.weight);
      this.output = squash(dot(inputs, weights) + this.bias);
    }
    return this.output;
  }

This is on principle the same concept right?

[ Parent | on HN ]

Re: Researchers unveil a pruning algorithm to shrink deep le... by cs702 on Saturday May 09, 2020 @11:48AM (#23125688)

Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... (I oversimplified things in my comment above).

[ Parent | on HN ]

Re: Researchers unveil a pruning algorithm to shrink deep le... by goatlover on Saturday May 09, 2020 @12:00AM (#23122195)

Could there be a transpiling step that converts the objects to arrays, targeting PyTorch or TensorFlow?

[ Parent | on HN ]