| 1) what happens with perjury? how severe is "penalty
  | of perjury"?

Penalty of perjury probably varies from jurisdiction to jurisdiction. I'm not a lawyer, so I'm not entirely sure. I would presume that this would be at the Federal level since the DMCA is a Federal law, as is copyright law. According to Wikipedia[1]:

  > In the United States, for example, the general perjury
  > statute under Federal law classifies perjury as a felony
  > and provides for a prison sentence of up to five years.

So the person signing the DMCA takedown notice faces up to 5 years in prison and a felony conviction. That said, perjury convictions don't happen very often to my knowledge because they are so hard to prove beyond a reasonable doubt. Even if the claim looks like any normal person should have realized that it was bogus, it's possible for the claimant to attempt to say that he/she is dumber than the normal person.

  | If at point 5, LayerVault chooses not to take legal
  | action (say, because they realized their mistake),
  | did they just take some random piece of work offline
  | for 10-14 days without any consequence?

They opened themselves up to the possibility of perjury charges. The level of risk for them is debatable though. Other than that, there are no penalties to filing bogus DMCA notices. This is one of the many (valid) criticisms of the DMCA.

  | What if they do take legal action and lose?

Then whatever the courts decide at that point. Taking legal action doesn't really have much to do with the DMCA when it gets to that point. This provision of the DMCA was all about taking things down quickly so that the claimant has time to get the court system moving (rather than needing to wait a few days to take something down, time during which they could be financially taking losses[2]).

Someone taking actual legal action is pretty costly. If someone uses a DMCA takedown notice, and follows up with a lawsuit, then this process is working as intended, even if the claimant/plaintiff is in the wrong. Who is wrong, and who is right will be handled by the courts at this point.

The real danger is using the DMCA without filing a lawsuit. There are few penalties[3] for false claims, little cost and yet a big benefit to sending the takedown notice. The service provider has to comply, unlike with a C&D letter.

[1] http://en.wikipedia.org/wiki/Perjury

[2] Think someone posting a product prior to release. E.g. posting Windows 9 a week before the release date.

[3] Perjury is a serious penalty, but you're not punished until you're dragged to court over it (and convicted). Thus far, I don't believe anyone has been convicted of perjury over a false DMCA notice, despite some really egregious examples (e.g. one of the takedown notices sent to MegaUpload was for a "url" that was actually a paragraph of text containing something like 90+ words -- pretty obviously generated by an automated process, but signed off by a human).