[WIP] Improved performance of journal cleaning

[stekycz]

Refs #21

Performance testing

• Redis 2.8
• 10.000 keys
• 10 tags each
• cleaning by 5 tags

PHP

• current - 9.5 seconds on average
• after update - 0.25 seconds on average

Lua

• 0.45 seconds on average - no meassurable improvement (yet)

disadvantage

• deleted keys could remain in priority sets or in tag sets (increase of memory usage)

[fprochazka]

Great, but I won't have time to look into this till at least friday. Would you mind doing the first review @matej21 ? :)

[matej21]

Jen par poznamek v cestine, sorry :)

• toto skoro vubec nesouvisi s [RFC] fastest mother-fucking journal ever #21. U toho se predpoklada, ze se zaznamy v clean nebudou vubec mazat a k overovani platnosti dojde az pri cteni. proof of concept takoveho cteciho scriptu je tady https://gist.github.com/matej21/e16a2846ea61e9413ffb (v invalidations je hash mapa s tag => cas invalidace, v added pak cas pridani jednotlivych zaznamu), read performance z redisu se zhorsil asi o 30%, coz neni tak zle :)
• o nepromazavani tech setu jsem uvazoval, ale krom vyssi memory usage to ma pomerne zasadni problem: tomu klici zustanou tagy z predchoziho zapisu, i kdyz je vubec uz nemusi mit. priklad:
  1. ulozim zaznam "lorem" s tagy foo, bar
  2. smazu zaznamy s tagem foo
  3. doslo k odstraneni klice lorem a setu lorem:tags a foo:keys. Klic vsak zustal v bar:keys
  4. ulozim zaznam "lorem", nyni jen s tagem foo
  5. smazu dle tagu bar. Smazal se vsak i lorem, jelikoz byl v puvodnim setu

[stekycz]

I see the relation in the way that there have been some data omitted in Redis. However you are right now.

I have updated the implementation to make it work as expected. New speed of PHP implementation is 0.6 seconds on average for the same testing data.

[stekycz]

Commit b0fae69 has much better performance (0,16 s in Lua, 0,12 s in PHP). However it leaves a lot of journal data in Redis to be removed by something else. Let's call it "garbage collector". If it could be a good way then I can prepare code which deletes these orphans. I have some idea how to do it.

[stekycz]

It is also possible to add garbage collection requiring (and very fast) as optional in configuration. All comments and ideas are welcome.