Ah ok, somehow misunderstood that.
However there are actually good reasons for it to be unsafe, although it is debatable if that alone should be it.
One is due to the interactions with the GC, in case it moves the data and there are references to its elements, and stack size.
One way to get around it is to use AoS instead of SoA, which is any the best option if performance is the ultimate goal.
That is exactly what I was talking about.