Demystifying Differentiable Programming (2018)

Re:Demystifying Differentiable Programming (2018) by noelwelsh on Monday February 17, 2020 @03:05AM (#22346243)

If you speak Scala, here's code for reverse mode differentiation as a monad. The flatMap / bind operation becomes the the chain rule.

If this "explanation" (scare quotes because it's not really an explanation, just a code dump) doesn't work for you let me know and I can try a different approach.

  final case class AD[A](v: A, k: Double => Double) {
    // Chain rule
    def flatMap[B](f: A => AD[B]): AD[B] = {
      val self = this
      val next = f(v)
      AD(next.v, (d: Double) => next.k(d) * self.k(d))
    }

    def +(that: AD[Double])(implicit ev: A =:= Double): AD[Double] =
      AD(this.v + that.v, (d: Double) => this.k(d) + that.k(d))

    def *(that: AD[Double])(implicit ev: A =:= Double): AD[Double] =
      AD(this.v * that.v, (d: Double) => (this.k(d) * that.v) + (that.k(d) * this.v))

    def sin(implicit ev: A =:= Double): AD[Double] =
      this.flatMap(x => AD(Math.sin(x), (d) => Math.cos(x) * d))

    def gradient: Double =
      this.k(1.0)
  }
  object AD {
    def pure(x: Double): AD[Double] =
      AD(x, d => 1.0)
  }

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by tiarkrompf on Monday February 17, 2020 @03:26PM (#22350416)

Hi Noel, can you clarify how this corresponds to reverse mode AD? To me it seems more like a delayed or on-demand version of forward AD.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by edflsafoiewq on Monday February 17, 2020 @04:47AM (#22346573)

Oh, that's the monad — × M where M is a monoid right? M being, in this case, the monoid of functions R -> R under point-wise multiplication.

Shouldn't M actually be a monoid under composition? ie. you should compose like next.k(self.k(d)), not multiply.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by noelwelsh on Monday February 17, 2020 @06:33AM (#22346951)

Sorry, I don't understand your notation. To me a monad is defined by two functions, pure and bind (aka flatMap). I don't understand how — × M corresponds to this.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by tome on Monday February 17, 2020 @06:52AM (#22347030)

— × M is the functor. In Haskell it would generally be used as "Writer M", but (M,) can also be used.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by noelwelsh on Monday February 17, 2020 @12:09PM (#22349016)

Sorry, that doesn't help. I literally do not understand the notation. I do not know what — means in this context, I don't understand what role × plays here, etc.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by tome on Monday February 17, 2020 @12:20PM (#22349083)

"—" means "the parameter of the type constructor goes here". I believe in Scala you use underscore for this, at the value level at least. "×" is what mathematicians call the type constructor for the usual product type. In Haskell this is called (,). I don't know what it's called in Scala.

So, — × M means something like "type T a = T a M", in Haskell notation.

[ Parent | on HN ]

Re:Demystifying Differentiable Programming (2018) by edflsafoiewq on Monday February 17, 2020 @12:52PM (#22349269)

In your notation it would be something like

  final case class Mon[A](v: A, m: M) {
    def flatMap[B](f: A => Mon[B]): Mon[B] =
      Mon(f(this.v).v, f(this.v).m * this.m)  // * is M's binary operation
  }
  object Mon {
    def pure(v: A): Mon[A] =
      Mon(v, M.unit)  // M.unit is M's identity element
  }

[ Parent | on HN ]