There is of course an alternative: use a function rather than a factory; but that's kind of the point of this discussion :-).
(Before we start a flame war: This isn't a criticsm of ruby or C# specifically, it's just something I've seen a lot in those code bases. Both languages allow writing factories or using lambdas.)
And I entirely agree with you that a PointFactory is unwanted (barring special circumstances) - though I entire disagree you'd want points to be mutable. Certain control points? Certainly - make a MutablePoint which is (conceptually) simply a reference to a point value. All points? That's just asking for pain; I really have better things than to track down with nested submodule thought it was mutating a copy but due to some optimization interaction turned out to be mutating a copy someone was actually looking on. Not to mention that reference semantics don't work very nicely with hashtables and lots of other datastructures which become a lot more complicated when the values can change right under them.
Trying to make code correspond to real world objects is annoying because after you've modeled your Car with Wheels that support Tires, and an Engine with EngineParts attached ... you then have to write a adaptation layer to make all that fit into your storage engine, your user interface, your query engine, etc ad nauseam. Strings are not "real world" objects, but we use them in our code nonetheless. They're a great UI item that lets us interface with our users. "I need input from the user. Here, type some text in this field and I shall call it String." Now you can manipulate the String with operations (methods on the object) that make sense for Strings.
Point: OO is not about "modeling the real world," it's about code organization. Some people organize with OO. Some with other means.