I suspect that GPT4 started with a crazy parameter count (rumored 1.8 Trillion and 8x200B expert “sub-models”) and distilled those experts down to something below 100B. We’ve seen with Orca that a 13B model can perform at 88% the level of ChatGPT-3.5 (175B) when trained on high quality data, so there’s no reason to think that OpenAI haven’t explored this on their own and performed the same distillation techniques. OpenAI is probably also using quantization and speculative sampling to further reduce the burden, though I expect these to have less impact on real world performance.
I suspect that GPT4 started with a crazy parameter count (rumored 1.8 Trillion and 8x200B expert “sub-models”) and distilled those experts down to something below 100B. We’ve seen with Orca that a 13B model can perform at 88% the level of ChatGPT-3.5 (175B) when trained on high quality data, so there’s no reason to think that OpenAI haven’t explored this on their own and performed the same distillation techniques. OpenAI is probably also using quantization and speculative sampling to further reduce the burden, though I expect these to have less impact on real world performance.