All the interfaces to this are open: the wire protocol to the SIM and the over-the-air protocols for authentication and security (what a phone needs to do) are specified completely by freely available 3GPP/ETSI specs.
The details of how the SIM computes the derived keys and authentication response are a black box. 3GPP suggests a couple of sample algorithm sets, but there's no way to know whether the card did that unless you have the secret key (shared between the card and a box deep inside the carrier's network).
However, the phone just passes the authentication-request parameters in an authenticate command to the card, passes the result of that in an authentication-response message back to the network, and obtains integrity and ciphering keys (for GSM) from a couple of files on the card (UMTS/LTE use some keys derived from these, but again, it's documented).
An alternative, used in some older US specs like late AMPS, CDMA, and (EIA-136) TDMA is to store the secrets in the phone, generally at manufacturing time. This means that there's no real boundary between the carrier's authentication widget and "your" phone.
Much better to have the identity contained in a SIM that belongs to the carrier, that your phone talks to at arm's length.
Java seems to be for applications other than plain (U)SIM like prepaid account management. The card can ask the phone to add a menu to its interface, show messages, collect inputs, and other things. Maybe all plain USIM apps are written in java these days instead of having a masked ROM like they probably used to.
The most interesting thing your SIM card does is run arbitrary programs that can be uploaded to them, without your knowledge, by the carrier:
https://www.defcon.org/images/defcon-21/dc-21-presentations/...
The SIM card is a full computer, with its own CPU and memory, that lives inside your phone and that you have no control over.
As a result, we just recently got cheap, widely accessible SDRs, and even they only came about through more or less an accident. And their performance renders them pretty much useless for even just GSM. Now serious SDRs like USRPs have been available for a long time, and their specs make it possible to run GSM and other advanced RF protocols. So if you pay the 2k+ required to get one of these functional SDRs and have the knowledge that is predominantly still only found in dead trees or lectures in fields quite unrelated to CS, you still won't be able to get anything resembling a GSM phone.
There are the obvious legal hurdles, you are not allowed to transmit willy-nilly, not even to just talk to a public GSM network. If you make sure no RF gets out the real world, you can pickup horribly outdated, discarded GSM test equipment from eBay for more big bucks and use that. In related news, they usually have a GSM(+) test network at the CCC congress, but they won't in 2016 because the regulatory agency just auctioned off the last bit of available test spectrum in the right band you could acquire through a lengthy, convoluted petition.
And the list just goes on and on and on.. remember you'll need to first communicate with a SIM (really, Java!) card that is your key into any network. They pretty much mandated the equivalent of a DRM dongle!