Artificial Intelligence How Do Machines 'Grok' Data

Re:Artificial Intelligence How Do Machines 'Grok' Data by mannykannot on Sunday April 14, 2024 @01:38PM (#40032799)

Sometimes, the networks instead find what the researchers call the “pizza” algorithm. This approach imagines a pizza divided into slices and numbered in order. To add two numbers, imagine drawing arrows from the center of the pizza to the numbers in question, then calculating the line that bisects the angle formed by the first two arrows. This line passes through the middle of some slice of the pizza: The number of the slice is the sum of the two numbers.

I feel I'm looking really dumb here, but it is not obvious to me that this works - seeing as this is a pizza analogy, take, for example, 1 + 1 mod 8 = ?. I do not see how the algorithm as set out in the paper can be properly described in this way.

[ Parent | on HN ]

Re:Artificial Intelligence How Do Machines 'Grok' Data by nh23423fefe on Monday April 15, 2024 @02:31PM (#40044068)

I think its shorthand but still "accurate"

all of the following are on the same line through the origin.

    8+2 1+1 2+8 3+7 4+6 5+5 6+4 7+3

and the set of lines through the origin is double-covered by the output embeddings. which means there are twice as many slices in the output embedding as in the input embedding.

so its not that 1+1 bisector points at 1 and so the sum is 1. it points at 2 because of the double cover.

this double cover is easier to see if you examine step 2.2

    | cos(wk(a − b)/2)|(cos(wk(a + b)),sin(wk(a + b)))

and set a=b and then increment around the circle.

when a=b the leading |cos| term is 1 and the vector is roughly cos(2a),sin(2b). now go from a=b= 0 to 2pi and youll see the embedding will loop around twice as it varies from 0 to 4pi

the outside of the circle is labelled twice with 12 at top and bottom. the line through the origin picks this labelling. not the input embedding.

                     , - ~ 12~ - 1
                 , '               ' 2
               ,                       3
              ,                         4
             ,                           5
             ,                           6
             ,                           ,
              ,                         7
               ,                       8
                 ,                   9
                   ' - , _ 12_ 11 10

[ Parent | on HN ]