Actually, the problem is that there is no way to control the difficulty. Finding new primes would be harder and harder, but you want the difficulty to adjust up or down to target a block time. Not sure how primecoin approached that problem?

Update: found some clues here, https://www.reddit.com/r/primecoin/comments/1i71ma/strange_b...