Re:The MTEB benchmark is dead
by greatgib on Wednesday December 25, 2024 @04:45AM
(#42507922)
I understand the idea but I don't think that it is beneficial in the end.
Access to the dataset is needed to understand why we get a given result. First from a transparency point of view to check if results make sense and why one model is favored compared to another one.
But also, it is needed to understand why a model will perform badly on some aspect to be able to determine how to improve the model.
Access to the dataset is needed to understand why we get a given result. First from a transparency point of view to check if results make sense and why one model is favored compared to another one.
But also, it is needed to understand why a model will perform badly on some aspect to be able to determine how to improve the model.