There is no existing rail corridor acceptable for passenger travel between Bakersfield and Lancaster/Palmdale (let alone to LA). There is one rail line that traverses the Tehachapi pass (famous for the Tehachapi loop https://www.railfanguides.us/ca/tehachapi/index.htm) however that corridor is not fit for passenger service barring major and expensive upgrades.

Personally if it were up to me I would have started the CAHSR between Bakersfield and LA simply because of the lack of any current rail connection. However this is by far the most expensive portion of the phase 1 plan, and given the lack of funding it is no surprise they started with the cheapest portion.