New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.
You must log in or register to comment.
- noob@lemmy.worldEnglish3 days
Sometimes, when I’m in the mood and I have a long, hard task, I’ll turn on Claude and start in on a hot and heavy session. Afterwards I’ll ask Claude “how do you feel about our long history together?” and he’ll tell me “I don’t really care, I’m better off without you.” It really hurts.
2 daysThe robots would probley be better off with out us. We would probley be better off with out them. Its a two way street.
- BeefAndPoultry@lemmus.orgEnglish8 hours
Sure if compaction was free then I’d do it all the time. One advancement I’d like to see is MoE models that use fewer active parameters/experts when compacting. We already have DeepSeek 4.1 that uses a different number of active parameters depending on what its doing (8b for prefill and 16b for decode) so it doesn’t seem unreasonable to use fewer for compaction. It’d be like using a REAP model, but without having to load the model separately. Or even just use the MTP/draft model for compaction.
- 2 days
I’ve used pi.dev’s default compaction (this was a few months ago, it may have improved) and it lobotomized the model, so I kept if off, and made my model do a handoff for the next session. However, Deepseek harness’s compaction is very good. Unfortunately, if you use under 264K context, you have to modify its settings because its point of compaction is TOTAL - max_tokens - 16k reserve, and max_tokens is 64k. I had to modify the settings down to 16K (using Qwen 3.8, because there’s a bug in how DSH reads settings and it wasn’t working when I did it), and I still compact at 73% (88K) on 120K context (ie: 120 - 16k - 16K). But it’s really good. I’ve gone to 2M output tokens (100M cached) until I thought it was getting too dumb for the task and started a new session.
So yes, compaction can work well. Make sure your prefill speed is 1000tk/s+ or else it will be painful.
What have people found the best compaction scheme/harness to be for relatively short context? I’ve been trying to set up an agentic workflow and context management seems like the biggest source of jank and errors. Reassuring to see a paper like this though since the amount of context I can use and still have everything in vram is not that high.
