Incorrect token estimate for AAAK

[panuhorsmalahti]

The token counts mentioned here are not at all correct according to token estimator such as https://platform.openai.com/tokenizer

[gladkikhartem]

yep, really confusing.
I've read AAAK description and it doesn't look a token-saver for me.
Thanks for raising this 2 hours before me)))

I really like the fact that in today's world you can get thousands of Github stars in 2 days by proper marketing - and everyone will use your project and share it everywhere, without even going into detail.
Just say that this is done by Mila Jovovich and everyone jumps on the bandwagon, without even putting 2 brain cells together or even evaluating how come a 50 year old star decided to become a software engineer.

Really hope this repo doesn't have malware builtin.

Kudos to Author for trolling everyone ))))

[ClearVIper3]

@gladkikhartem Exactly

[cagrimunyas]

The token counts mentioned here are not at all correct according to token estimator such as https://platform.openai.com/tokenizer

Exactly! Tried with openai's gpt5.x tokenizer. The correct results are :

• English text is 66 tokens instead of ~1000 tokens
• AAAK text is 73 tokens instead of ~120 tokens

Unless the texts in the readme file truncated, assumptions for both texts doesn't seem correct, also plain English is using less token.

[bingiflash]

Initially I was hyped when i saw the project. The moment I saw the example, I'm like 'That's not true". online tokenizers confirmed my doubts. It may have some benefits when the text is huge, but i'm still skeptical until I see some actual tests.

Thanks @panuhorsmalahti for raising this

[vanakema]

Yea it's wild for this to get so much hype when this self-described "game changer" is straight up worse than the existing alternative (plain english)

[milla-jovovich]

You're right and we're sorry. The token counts in the README example were generated from a len(text)//3 heuristic, not an actual tokenizer — and the example we picked doesn't demonstrate compression at all (66 vs 73 tokens, as you and @cagrimunyas verified).

We've pushed an honest correction to the README (commit aa10f8f) — there's now "A Note from Milla & Ben" at the top of the repo owning this and the other overclaims. The AAAK section has been rewritten honestly: it's a lossy abbreviation system, designed for repeated entities at scale, and on LongMemEval it currently regresses vs raw mode (84.2% vs 96.6%).

We're adding a real tokenizer for stats and rewriting the example. Thank you for raising this two hours after release rather than after Wired picked it up — that's the version of feedback that makes things actually get fixed.

— Milla

[bensig]

Fixed in #147 — replaced len(text)//3 heuristic with word-based estimate, updated all docstrings and CLI output to label AAAK as lossy summarization, not lossless compression.