That makes them merely select, like the janitor. People do commonly use elite as a pejorative stand in for select group, but elite still does mean in some way better or superior, and surely you don't mean to say they are superior in any way than position.