EDIT: Well, cross product would work, too, if you extended the 2D vectors into 3D vectors, coplanar in the screen. The orientation of the cross product into or out of the screen would also tell you the relative orientation of the two vectors. So then, you'd be looking for the two cross products with differing signs on the Z component to tell you if it's "in the triangle".
So basically, for a moving between two mouse points (Ix, Iy) and (Ax, Ay), and a top left menu corner (Bx, By) and bottom left (Cx, Cy), then:
DAx = Ax - Ix
DAy = Ay - Iy
DBx = Bx - Ix
DBy = By - Iy
DCx = Cx - Ix
DCy = Cy - Iy
Z1 = DBx * DAy - DAx * DBy
Z2 = DCx * DAy - DAx * DCy
Then, you know you're headed towards that menu if Z1 < 0 and Z2 > 0.
Could be simplified a little if you assume the X component of the top left corner of the menu will be the same as the X component of the bottom left corner, but the math was clearer to keep them separate.
Actually, you can make the dot product give you that answer, but you would need use the dot product of the mouse vector and two vectors normal to the edge vectors, which is really just a roundabout way of using the cross product.
[1] http://en.wikipedia.org/wiki/Cross_product