Most of the testimony given to the CAIB were sealed (very controversial move), so that's the only quote we have from him. However, he later gave an interview: http://www.spaceflightnow.com/shuttle/sts119/090327sts27/
Another account can be found in Mike Mullane's memoirs, Riding Rockets. Mullane was a mission specialist on STS-27, and operated the Canadarm during the tile inspection. He says that he kept thinking about the tiles, and had trouble sleeping. Hoot Gibson advised him: "No reason to die all tensed up."
Remember -- STS-27 was the second mission after the Challenger explosion. Not enough time yet for complacency to have set in. In addition, Mike Mullane had an it-was-absolutely-not-an-affair-we-were-just-friends relationship with Judy Resnick. So he knew very personally just how dangerous the Shuttle could be.
(Note...this isn't one of those pedantic "prove it" citation requests. I'm seriously interested...)