I worked at a startup with a hardware product in 2011 (nb: I didn't work on the hardware personally, but was aware of the difficulties we faced), and the issues you describe are all very familiar.
It may be the case that the technologies you describe allow you to push the problems to later in the product development cycle, but from what I observed, they all still happen once you try to transform that product into a consumer product that can be manufactured and sold at a viable price-point.
It's true for example, that the hardware engineer founder built a prototype relatively quickly using off the shelf components, but the resulting product was so large, the housing so ugly, and those components so expensive that it would never have been economically viable.
Once they were past the prototype stage, they faced all the problems you describe: parts manufacturers and suppliers only willing to deal in huge quantities and make sales well in advance; finding manufacturers who can make plastics that are of high quality and meet safety standards is surprisingly difficult and slow [1]; much of the tooling for small, cheap, low-power-consumption microcontrollers seems to be from the dark ages [2].
What made it even worse was that many VCs we spoke to seemed to be unfamiliar with or unwilling to take the risks associated with hardware startups: much higher up-front costs to scale the business, and much longer product development times. It's possible the company I worked for didn't take advantage of all the modern developments available to hardware startups, or that they had the wrong strategy (trying to scale too fast or seeking the wrong customers, for example), but I certainly have enormous respect for the challenges hardware startups must overcome, even in these times.
1: Some anecdotes I remember from the plastics saga: the first overseas manufacturers they tried in an early attempt to make the housings cheaper absolutely would not, and maybe could not produce a housing in the correct shape. We would send them models in a form designed to be structurally sound and aesthetically appealing, and they'd come back as basically ugly square boxes. We ended up using a much more expensive US-based manufacturer for a long time because of this. (IIRC, they were more expensive mostly because their facility was really meant for rapid development and not mass production.)
I also remember a significant delay at a crucial time because we couldn't get the fireproof coloring agent we needed in the right color.
2: One of the hardware engineers spent a grueling couple of weeks hunched over an oscilloscope looking for a particular pulse that would indicate that the microcontroller had found and booted our company's software on the chip.
From recent experience working on a hardware side-project: these are extremely cool and useful development/indie tools. But when you get to the stage of large scale production we are still lacking easy entry.
I've been working on an XBEE home automation project intended to be run from a Rasp Pi "server" (but designed to run from anything that can boot linux and has GPIO). It's looks great, has a nice enclosure and is something any hobbiest could get working with the right plug and play bits.
But.
Packaging this for mainstream, where a user can simply buy "black boxes" and connect them up is tough. Circuit design and fabrication on any scale requires both skill (i.e. in designing the layout - I am an Electronic Engineer but of the solder-stuff-together ilk rather than CAD) Admittedly places such as Batchpcb etc. make prototyping a lot easier, but the hurdles you mention are still there in many forms.
The maker/hacker collective is very well serviced nowadays. Next we need someone to disrupt the small-scale development arena: off hand, a community of PCB designers/hackers available to make bespoke PCB layout which links into a batch production system would rock! I've hacked together my prototypes, but have minimal confidence they will work first time :)
I've never gone through that process. Anyone know ballpark costs for certifications?
Of course the way it's meant to work is that your company hires an approval company that knows a lot about certifications, tests the product and then approves it knowing that they're on the hook if it doesn't meet requirements and regulations.
Nowhere does it say that the approving company can't be the same company as the one making the product, so basically we bought a few CE stickers put them on the product and it was approved. By us.
* The product may be placed on the market only if it complies with the provisions of all applicable directives and if the conformity assessment procedure has been carried out accordingly.
* If stipulated in the directive(s), an authorized third party (Notified Body) must be involved in the conformity assessment procedure.
I'm not sure if the machine directive stipulates it, but "authorized third party" seems to imply that the manufacturer can't perform the certification themselves.
Even if you can count yourself as the approving company, did you make sure that your product complies with all the provisions? Or were you taking advantage of some loophole somewhere to skip the entire process?
http://en.wikipedia.org/wiki/CE_marking
EDIT: I see that you mentioned this was 15 years ago, so I'm mostly curious for historical reasons.
The "If stipulated" part seems to be the key. Below your quotes in the wikipedia entry there is a section that covers self-certification. While it doesn't list the directives comprehensively, the steps there do seem to indicate that self-certification is not unanticipated.
One of the reasons the test houses exist is that not everyone wants to assume the responsibility of self certification and so they outsource it to experts in the test industry.
The most important thing to make sure, regardless of the route you choose, is to always have your certifications and technical documents to hand should one of the monitoring bodies ask for it
Granted, we have intentional radiators (like wifi radios, cellular modems) in our products so it may be more expensive. YMMV of course.
Edit: with the comment by thomaslangston, I should clarify that the included radios already had their own FCC certifications; this was just Part 15 for the whole device.
"Agency Compliance and You: An Introduction for Small Manufacturers of OSHW" -- <http://summit.oshwa.org/agency-compliance-and-you-an-introdu...;
Slides: <http://summit.oshwa.org/schedule/oshws_presentation_fcc/>;
They don't seem to have the edited video online yet but you can look at the schedule <http://summit.oshwa.org/schedule/>; to find the session in the recorded live stream: <http://www.ustream.tv/channel/ohs2012>;
http://blog.circuits.io/post/34159982502/in-this-video-karel...
I give you, the Open Pandora project - a bunch of open-source hackers who are spearheading the hardware rennaissance and showing everyone, along the way, how to do it - and not do it (also: volcano):
One day, probably soon, there will be books written about this project. It is an awesome thing - the device itself, the people involved (EvilDragon and his merry band VERSUS Craigix' Reality Distortion Field [really, he has one!]), and the whole community. The Open Pandora Community is a very, very, very fertile y-combinator with proven track record of delivering, within its own realm of course, much value.
So, check it out, yo. Open Pandora .. 2.0 .. could be a YC project? Or .. the inevitable spinoffs? (The Open Pandora is old enough now that there will soon be splinter factions establishing new product, btw..)
[Cue Four Yorkshiremen]
Try an fabless IC company: 8-10 week turn times on fabrication, each run was 6 digits $.
Now I'm not so sure.
Our problem was twofold: First hardware development was not generally something you just did in your sparetime, it was for the big boys. This meant a lot of convoluted processes for dealing with suppliers, expensive and unreliable dev kits and tools, long lead times and all sorts of other hassles.
Second the turnaround time for a prototype was at least two weeks. If you made a small mistake you'd find out two weeks later when your prototype arrived. This adds up quickly and slows you down tremendously. Not because we made a lot of mistakes, but we had to be absolutely sure that something worked before sending it off to be prototyped. No testing a new idea in an afternoon or two.
These things have totally changed with the commodisation of hardware and the looming 3D revolution.
With a makerbot, a raspberry Pi, an arduino and a shelf full of components you're prety much ready to go and can hammer together a working prototype in no time. If you feel cheap you can buy a nice box for your arduino and call it a finished product.