Phind-70B: Closing the code quality gap with GPT-4 Turbo while running 4x faster

Re:Phind-70B: Closing the code quality gap with GPT-4 Turbo ... by devit on Thursday February 22, 2024 @09:32PM (#39476251)

How is this possible?

GPT-4 is supposed to be 8*220B = 1.7T parameters, so it seems unexpected that a 70B model can beat or match it unless it's somehow a much better algorithm or has much better data.

[ Parent | on HN ]

Re:Phind-70B: Closing the code quality gap with GPT-4 Turbo ... by benxh on Friday February 23, 2024 @01:05PM (#39484032)

If GPT4 is 220B/8 experts, that would be in-line with 3.5 Turbo being a 20B model, and GPT4 being a 55B activation out of a total 220B parameters.

It is ultimately all speculation, until Deepseek releases their own 145B MoE model, and then we can compare the activations/results

[ Parent | on HN ]

Re:Phind-70B: Closing the code quality gap with GPT-4 Turbo ... by devit on Friday February 23, 2024 @03:34PM (#39485762)

I think the conjecture is that each expert of GPT-4 has 220B parameters, for a total of 1.76T parameters.

[ Parent | on HN ]