I did get panics but I haven't been able to reproduce them. The only thing I can think of is create a jail without an assigned IP, start it, assign one, shut it down and start it up again.
The bug I ran into was https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212649 which turned out to be an other symptom of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212576.
I hope we'll get VIMAGE into GENERIC for FreeBSD 11.1. He completely refactored the IP stack shutdown logic. Now the layer are shut down from top to bottom instead of the other way around. This avoids most of the nasty locking problems draining the higher levels with their pointers to the lower layer resources (e.g. routes, interfaces) first. The remaining bugs won't be found without a lot more exposure.