Your argument appears to be jumping from (lack of causal associations) to (assumption that causality is in the opposite direction).