My understanding is that is they don't really care if the AI bots got the data on gitlab, but the problem is that the bots are very aggressive and have been slowing down the gitlab instance too much. I definitely agree it isn't great, but I don't know what a better option would be. Also once it passes the page once it should save a cookie and you'll be good to go for a while.