Great question! My intuition tells me that L1 regularization could indeed be impacting the performance of the model, but for me personally, the only way to find out for certain would be to test this empirically.