The clause 2 permits you to add terms that prohibit the contractors from making any copies. While technically, a contractor could request the source code by clause 13, it do not conflict with the contractually term that they have which says they are not allowed to make copies.
DannyBee said the word "access", and technically that would then be "read only" access. While I could conceive a situation in which a disgruntled contractor plus the original author could in theory make a case, read-only access is a odd thing to create a court case around and a judge decision could really go anywhere. I don't think anyone would be willing to test it or feel that the court costs would be worth paying for the privilege.
This is a fairly nasty difference from the standard GPLv3 that I don't think enough people appreciate. As you point out, companies that modify GPL code for internal use don't have to release the source code and aren't affected by most of its provisions, and this is intentionally one of the freedoms it gives users. This isn't true of the AGPLv3.