Benchmarking GPT-4 Turbo – A Cautionary Tale

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by m3kw9 on Thursday November 09, 2023 @09:27AM (#38205384)

Now do a programming task that requires more than 32k of context and see who’s “better”. If you don’t bench mark that you cannot get an overall pic. GitHub copilot for example could benefit big from the increased context

[ Parent | on HN ]

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by broast on Thursday November 09, 2023 @09:29AM (#38205417)

Obviously it's a drawback but the silver lining of the small context window is it forces me to decouple everything and have very sensible and strict api's where I just write the docs and it writes the code.

[ Parent | on HN ]

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by biobootloader on Thursday November 09, 2023 @11:04AM (#38206735)

we are working on creating "real world" benchmarks that require a lot of context, and will report when we have results!

[ Parent | on HN ]