Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
You must log in or register to comment.
- ZephyrXero@lemmy.worldEnglish22 hours
They’ve changed the license with Qwen 4, starting with 3.8-next. So I probably won’t be using any further releases
- fluxx@lemmy.worldEnglish6 hours
What are the significant changes? Requiring attribution or something more sinister?
- dosse91@lemmy.trippy.pizzaEnglish2 days
Flash next 125b is genuinely amazing, I hope to see 4.0 of the same size.
- tpWinthropeIII@lemmy.worldEnglish2 days
Agree, and upvoted.
Another good size would be 160 gb for 96 gb vram systems. After that I would like to see 225 to 250 gb.
2 daysAnother 35B-A3B MoE or similar would be nice…
FYI, Yandex released an 80B MoE base model (i.e. not instruction tuned!) recently: https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base – haven’t experimented with it personally though.
gemma 26b-a4b was fantastic, anything in that area is ideal
running my tiny rx6600 8gb vram
- 2 days
Oh hell yeah
I’m on my 1070 8gb copium so we’re twinsies
Maybe they release a huge moe model with non-uniform quants and then we can get even more quality on those cute little cards
I have an M4 Pro 24GB and I still use Gemma 26B-A4B as my primary model. Beats every other model I’ve ever tried locally on most stuff except coding and tool use, so a Qwen 4 at this size would be super interesting
- BeefAndPoultry@lemmus.orgEnglish2 days
32GB RAM and 8GB VRAM here, so 35b a3b is about my maximum, maybe with added n-grams
- ffhein@lemmy.worldEnglish2 days
27B dense was a good size for 3.8, but I’m guessing they’re going to add a few B parameters in order to make it noticeably better.
- BeefAndPoultry@lemmus.orgEnglish2 days
If they can make the KV cache size scale more efficiently with many tokens, like DeepSeek 4.1 Flash fitting 1 million tokens in 890MB, it might give them enough room to add a couple billion parameters of weights and still fit well in 32GB GPUs
- robber@lemmy.mlEnglish2 days
Not necessarly, I’d say. We’ve seen many releases with multiple sizes and the 27B class has been around since at least the Gemma3 times.
- ffhein@lemmy.worldEnglish2 days
Well, it’s just a guess. But also Gemma4 increased to 31B, so I think there’s a possibility that Qwen does the same.
