Some implementations will group the categories into two groups, right and left. An easy way to do this is to create the groups based on whatever grouping creates the biggest decrease in the loss. There are more complicated implementations, lightgbm groups by the sum(gradient) / (sum(hessian) + smooth) where smooth is some smoothing parameter which prioritizes class size over loss decrease.

They link to this paper in their docs: https://www.tandfonline.com/doi/abs/10.1080/01621459.1958.10...