Section 8 of the Privacy (Tax File Number) Rule 2015:
TFN recipients must only request or collect TFN information from individuals and other TFN recipients for a purpose authorised by taxation law, personal assistance law or superannuation law.
(2) When requesting an individual’s TFN, TFN recipients must take reasonable steps to ensure that:
(a) individuals are informed:
(i) of the taxation law, personal assistance law or superannuation law which authorises the TFN recipient to request or collect the TFN (ii) of the purpose(s) for which the TFN is requested or collected (iii) that declining to quote a TFN is not an offence (iv) about the consequences of declining to quote a TFN
(b) the manner of collection does not unreasonably intrude on the individual’s affairs, and
(c) the TFN recipient only requests or collects information that is necessary and relevant to the purpose of collection under applicable taxation law, personal assistance law or superannuation law.
You can just not give your employer your TFN, and they'll withhold your tax at the maximum rate, and you can get a bunch of money back after your tax return. Your TFN is just a way for the ATO to figure out how much tax to withhold so your tax return is more correct.
Anyway, Australia and New Zealand are interesting, in that there is no single number that maps to a person. You can use your drivers licence number, or your passport number, or your TFN/IRD number, depending on the situation.
When they introduced photo drivers licences in New Zealand, there was concern in them turning into a de facto national ID card.