It isn't quite as fixed as some measurements since the average human's vision could continue to evolve and change the curve, or because we could find out that we measured it wrong, but it's not dependent on a single physical sample like the kilogram.
http://en.wikipedia.org/wiki/File:SI_base_unit.svg
Now, in the abstract, yours is the more natural definition. But experimentally, it turns out to not give the best accuracy.
I believe one starts with (1) the meter (which is based on a second defined using a atomic transition, plus light) and (2) the Newton (which is the force required to accelerate the all-important kilogram at 1 meter per second squared). Then an Ampere is the amount of current flowing through two parallel wires required to produce a force of one Newton per meter between the wires.
http://en.wikipedia.org/wiki/Ampere#Definition
Now, the technology is apparently improving enough that it may be possible to actually count elementary charges like you suggest. See here:
http://en.wikipedia.org/wiki/Ampere#Proposed_future_definiti...
"In fact, of the seven fundamental metric units — the kilogram, meter, second, ampere, kelvin, mole, and candela — only the kilogram is still dependent on a physical artifact."
Followed by:
"And the luminosity of light, or candela, is measured in terms of power, designated in watts, or joules per second."
I'm not an SI expert, but Wikipedia suggests the candela is in fact an SI base unit with a definition based on physical phenomenon of the universe, so the second sentence seems erroneous.
[edit: clarity]