And, speaking as somebody who's seen a 4-digit number of interview packets: The multi-interviewer format is far from flawless, too. The capability of groups of humans to collectively fuck up without ever talking to each other is quite impressive.
If you absolutely want to invest more time, at least build a multi-hour interview that has people do their job, in a team environment. Because that's what ultimately will happen.
Don't do multiple rounds of CS trivia.
For a small company the interviewer might well be the beneficiary of your work and in the end he needs you more than you need him. So a good one won't let his bad day ruin a nice candidate.
Also, depends on culture. There are countries where 1:1 interviews with a different gender candidate are okay.