DGoettlich

data is 100% public domain.

very interesting observation!

well put.

thanks. i think this just took on a weird dynamic. we never said we'd lock the model away. not sure how this impression seems to have emerged for some. that aside, it was an announcement of a release, not a release. the main purpose was gathering feedback on our methodology. standard procedure in our domain is to first gather criticism, incorporate it, then publish results. but i understand people just wanted to talk to it. fair enough!

i think we (whole section) are just talking past each other - we never said we'll lock it away. it was an announcement of a release, not a release. main purpose for us was getting feedback on the methodological aspects, as we clearly state. i understand you guys just wanted to talk to the thing though.

Thanks for the comment. Could you elaborate on what you find iffy about our approach? I'm sure we can improve!

valid point. its more of a stepping stone towards larger models. we're figuring out what the best way to do this is before scaling up.

we're on the same page.

exactly

Also one of our fears. What we've done so far is to drop docs where the datasource was doubtful about the date of publication, if there are multiple possible dates we take the latest to be conservative. During training, we validate that the model learns pre- but not post-cutoff facts. https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

If you have other ideas or think thats not enough, I'd be curious to know! ([email protected])

what makes you think we trained on only a few gigabytes? https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

fully understand you. we'd like to provide access but also guard against misrepresentations of our projects goals by pointing to e.g. racist generations. if you have thoughts on how we should do that, perhaps you could reach out at [email protected] ? thanks in advance!

we were considering doing that but ultimately it struck us as too sensitive wrt the exact in context examples, their ordering etc.