In a group improvisation, I understand "going out" as one person, usually a soloist, exploring another direction (a rhythmic idea, a motif, anything) where there's some risk that you won't come back, i.e. you might derail the whole thing. I grew up in New Orleans knowing lots of musicians and this is just my recollection. The closest documentation I find is about playing "outside," which is more specifically about using extended or unconventional harmonies. I've heard this usage in college settings, but I was referring to the more general concept.

"Stepping out" might be a more mainstream version, and it's apparently the title of like fifty albums.

https://en.wikipedia.org/wiki/Outside_(jazz)

https://en.wikipedia.org/wiki/Steppin%27_Out