I got a question after checking results on the open LLM leaderboard[1].
Comparing the result of NyxKrage/Microsoft_Phi-4 and microsoft/phi-4 or unsloth/phi-4, I can see fixing both the tokenizer and chat template causes the performance of both IFEval and BBH to increase. However, the performance on MATH, GPQA and MUSR degrades A LOT.
Is there any explanation on why this is happening?
According to Microsoft MATH score should be 80.4, while both original and the "fixed" models as run by unsloth only score just over 12.3. So either Microsoft made a few huge mistakes, or unsloth was not able to run their model correctly.
danielhanchen ·6 days ago
1. End of sentence should be <|im_end|> not <|endoftext|>
2. Chat template should not auto add an assistant prompt
3. Padding token should not be EOS but <|dummy_87|>
I also converted Phi-4 to Llama-arch. I uploaded GGUFs, 4bit quants, dynamic quants and all fixes to https://huggingface.co/unsloth
I also made a Colab notebook to finetune Phi-4 on a free GPU: https://colab.research.google.com/github/unslothai/notebooks...
Show replies
danielhanchen ·5 days ago
RandyOrion ·5 days ago
I got a question after checking results on the open LLM leaderboard[1].
Comparing the result of NyxKrage/Microsoft_Phi-4 and microsoft/phi-4 or unsloth/phi-4, I can see fixing both the tokenizer and chat template causes the performance of both IFEval and BBH to increase. However, the performance on MATH, GPQA and MUSR degrades A LOT.
Is there any explanation on why this is happening?
[1] https://huggingface.co/spaces/open-llm-leaderboard/open_llm_...
Show replies
t1amat ·5 days ago
Unsloth is a masterpiece, keep up the great work!
Show replies
lostmsu ·5 days ago
According to Microsoft MATH score should be 80.4, while both original and the "fixed" models as run by unsloth only score just over 12.3. So either Microsoft made a few huge mistakes, or unsloth was not able to run their model correctly.
Show replies