Batteries push electrons around a circuit. This is powered by a redox reaction, with half of the reaction occurring on each terminal. One half of the reaction produces electrons, the other half consumes electrons, and these electrons travel from where they are produced (the oxidation half-reaction) to where they are consumed (the reduction half-reaction).
However, one half reaction simply produced electrons forever, it would accumulate a positive charge. So something inside the battery has to also move charge around to balance things out. This thing that moves should have a positive charge, and move in the same direction as the electrons, except through the battery instead of through the circuit. Your electron moves from the battery anode (-) through the circuit to the battery cathode (+), and something else with a positive charge also moves from the battery anode (-) to the battery cathode (+), except it moves through the battery instead of moving through the circuit.
In other words, a positive ion. Lithium is the smallest positive ion you can normally work with.
You'd think hydrogen would be smaller, but it actually forms a hydronium ion in aqueous solution. Hydrogen fuel cells are a lot like batteries that use hydrogen + oxygen, but the chemistry of hydrogen fuel cells make them not suitable as batteries, and hydrogen is hard to store. I'm sure I'm getting some of this wrong, I did study some very basic electrochemistry in college, so I would encourage people interested in electrochemistry to read more about it or even do simple experiments at home (it's surprisingly easy).
Other group of very reactive elements are in the second-last column such as Fluorine but they can't be as easily used in batteries for other reasons.
Lithium is also very light - 0.534 g/cm³ compared to copper with 8.96 g/cm³.