It becomes valuable if it either compensates for the lack of hyperparam tuning in all or a broad class of models, or the result is precisely presented to say something like "better than random if the model is a linear svm with C=1".

Right now we are somewhere in between where the results are presented to seem somewhat generally useful, but when you tweak the hyperparams you realize that the utility is potentially quite narrow.

We either need broad utility or increased rigor in reporting - we cannot abandon both.

Yeah that's an interesting paper [1], and label imbalance is a case we have not investigated (we were able to reproduce some results from it - the DAL experiments in Section A.3 "Reproducibility Experiments" in [2] come from there).

[1] Active Learning for BERT: An Empirical Study, https://aclanthology.org/2020.emnlp-main.638.pdf

[2] One the Fragility of Active Learners, https://arxiv.org/abs/2403.15744