I think you're mixing up scales. Timestamps and numbers can be recoded to categories but when doing this you're throwing away information -- the same way you lose information when converting an integer to a string. When you work with factors you're no longer interested in qualities the data would have on its original scale like distance etc.

Could you please provide a real world example where this actually is a problem.