Upgrade to fast IAVL with multi batches

[giskook]

When we auto upgrade our huge (289G) data. We met OOM. so splite to multi batches to upgrade the fast storage.

[robert-zaremba]

let's add a comment here

[giskook]

had add a comment

[tac0turtle]

have you tried different numbers to see results?

[giskook]

have you tried different numbers to see results?

not yet. maybe I can use a different value to comare with this. do you have any sugguestion? maybe 5,000,000?

[tac0turtle]

yea lets try it

[giskook]

with 5m, we got lower memory usage and faster speed ;P

[tac0turtle]

amazing, lets run with it, thank you for doing the research

[giskook]

ok, had change the commitGap to 5000000, and remove the timing-gc related code.

[faddat]

very interesting!

[robert-zaremba]

@p0mvn you have been in charge for the fast cache - would you have time to review this PR?

[p0mvn]

utACK.

Thank you for this change. That makes sense.

As a side note for future work, do you think we can now remove this garbage collection logic:

iavl/mutable_tree.go

Lines 688 to 712 in 807f8c5

	go func() {
	timer := time.NewTimer(time.Second)
	var m runtime.MemStats
	for {
	// Sample the current memory usage
	runtime.ReadMemStats(&m)
	if m.Alloc > 4*1024*1024*1024 {
	// If we are using more than 4GB of memory, we should trigger garbage collection
	// to free up some memory.
	runtime.GC()
	}
	select {
	case <-timer.C:
	timer.Reset(time.Second)
	case <-done:
	if !timer.Stop() {
	<-timer.C
	}
	return
	}
	}
	}()

That logic was added to mitigate OOM but I think the solution in this PR might remove the need for the garbage collection.

We suspect that this garbage collection might be increasing the setup time in our unit tests in Osmosis.

ref: osmosis-labs/osmosis#2148 (comment)

[giskook]

utACK.

Thank you for this change. That makes sense.

As a side note for future work, do you think we can now remove this garbage collection logic:

iavl/mutable_tree.go

Lines 688 to 712 in 807f8c5

	go func() {
	timer := time.NewTimer(time.Second)
	var m runtime.MemStats
	for {
	// Sample the current memory usage
	runtime.ReadMemStats(&m)
	if m.Alloc > 4*1024*1024*1024 {
	// If we are using more than 4GB of memory, we should trigger garbage collection
	// to free up some memory.
	runtime.GC()
	}
	select {
	case <-timer.C:
	timer.Reset(time.Second)
	case <-done:
	if !timer.Stop() {
	<-timer.C
	}
	return
	}
	}
	}()

That logic was added to mitigate OOM but I think the solution in this PR might remove the need for the garbage collection.

We suspect that this garbage collection might be increasing the setup time in our unit tests in Osmosis.

ref: osmosis-labs/osmosis#2148 (comment)

@p0mvn
yes, I think we can remove this garbage colection logic.

I do upgrade comparison on our aws machine(16c64g).

data set
with two tree：a tree with 166439284 nodes, the other with 16201149 nodes.

four ways to upgrade

1. multi batches
2. multi batches without timing gc
3. single batch
4. single batch without timing gc

I got the two conclusion:

1. with multi batches we got lower memory usage and faster speed.
2. timing gc has less effective when we upgrade with multi batches.

[tac0turtle]

thank you for this pr