However, the swapping players from spectators would require them to directly connect into the game server, and simply spectate from there. I believe you can set them as coaches to prevent ghosting, as spectating directly from the server doesn't allow setting any sort of delay.
Basic guide here: https://www.reddit.com/r/GlobalOffensive/comments/2arae2/how..., might be slightly outdated.
However, the swapping players from spectators would require them to directly connect into the game server, and simply spectate from there. I believe you can set them as coaches to prevent ghosting, as spectating directly from the server doesn't allow setting any sort of delay.