Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
I hope an instruction tuned will be released as well.
- 2 posts
- 4 comments
Joined 4 years ago
Cake day: November 25th, 2022
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
I find the overall parameter size interesting. On 16GB VRAM the whole thing fits at an excellent quantisation without having to offload anything to system RAM and taking hits on PP & TG.
According to the officially reported benchmarks, Ling wins on GPQA Diamond (73.4 vs. 64.6) and BFCLv4 (62.72 vs. 62.3). No other overlaps.
That being said, I’m neither trusting benchmarks alone nor have tested any of the two extensively enough to have a solid impression.
I didn’t even know they train their own models.
Have been playing around with this on my laptop’s Radeon 780M, with 16GB of RAM assigned to the GPU. llama.cpp with Vulkan reaches a comfortable 30t/s TG with up to 1000t/s PP when running Q8_0 with unquantized KV-cache.
In the few rather simple summarisation tasks I gave it so far it performed better than expected, but of course it’s very small MoE and you shouldn’t expect too much. Looking forward to evaluating this in Pi on small agentic coding tasks.
Have some of you been playing around with it?
Not necessarly, I’d say. We’ve seen many releases with multiple sizes and the 27B class has been around since at least the Gemma3 times.