UTxO-HD 2.0

[jasagredo]

The current design of UTxO-HD caused its concepts to leak everywhere. All downstream code needs to know about the LedgerTables and the abstract layer has to manage those.

The first question to ask would be "Why was it this way?". LedgerDB V1 kept a DbChangelog with diffs that would extend further than the immutable tip, and therefore would (in principle) have no associated LedgerState. This prompted the existence of "Diffs" and "Tables" as a concept separate from a LedgerState, needing then a definition for the HardForkBlock and using it everywhere in the codebase.

On LedgerDB V2, tables which are not ephemeral are now always Values. It just so happens that we split the steps for applying a block in tick > getKeys > read > apply > diff > applyDiff.

We are following this diagram, moving between the HF layer and the SingleEra layer:

                    |                                HFTxIn           Map (TxIn (HFB xs)) (HFTxOut xs)                          Map HFTxIn (Delta (HFTxOut era))
                    |                              Keys HFB            Values HFB                                                 Diffs HFB
        HF    Layer |  block                      neededKeys -(read)-> UTxOs                                                      Diffs     -(write)-> done
                    |      \                     /                          \                                                   /
------------------- |----------------------------------------------------------------------------------------------------------------------------------------
                    |       \                  /                             \                                                /
Current Block Layer |       eraBlock - neededKeys                             eraUTxOs -(apply)-> eraUTxOs' -(diff) -> utxoDiffs
                    |                    Keys B                                Values B           Values B             Diffs B
                    |                   SL.TxIn

But there is an important fact that we can leverage:

PROP: Tables backend when the tip is at era X is isomorphic to the backend if it was in any of the eras in {X, X+1, ...}

This means that in the LSM database, what is on the disk can be interpreted into any era as long as it doesn't contain values that come from a later era. This is guarded simply by construction.

Leveraging this, we could do the following process instead:

        HF    Layer |  block
                    |      \                                   PROP
--------------------| ---------------------------------------------------------------------------------------------------------------------------------------
                    |       \
Current Block Layer |       ( eraBlock - neededKeys            -(read)->        eraUTxOs -(apply)-> eraUTxOs' -(diff) -> utxoDiffs            -(write)-> done)
                    |                    Keys B                               Values B            Values B              Diffs B
                    |                    TxIn                                TxIn TxOut era                            Map SL.TxIn (Delta (SL.TxOut era))

Therefore eliminating the need for HF level machinery and even the machinery at the abstract layer.

---

How would this look like in the code

We could make this change in the Shelley instance of a LedgerState:

-data instance LedgerState (ShelleyBlock proto era) mk = ShelleyLedgerState
+data instance LedgerState (ShelleyBlock proto era) = ShelleyLedgerState
  { shelleyLedgerTip :: !(WithOrigin (ShelleyTip proto era))
  , shelleyLedgerState :: !(SL.NewEpochState era)  -- Ledger concept of a LedgerState
  , shelleyLedgerTransition :: !ShelleyTransition
-  , shelleyLedgerTables :: !(LedgerTables blk mk)
+  , shelleyLedgerTablesHandle :: !(LedgerTablesHandle m blk)
  , shelleyLedgerLatestPerasCertRound :: !(StrictMaybe PerasRoundNo)
  }
  deriving Generic

(Possibly hiding the m in some existential)

And then the applyHelper function for Shelley could get the block key sets, read the values, inject them into the NewEpochState, compute the diff, and create a new handle for the returned LedgerState with the diffs applied. Something like:

applyHelper f cfg blk stBefore = do
  keys <- getBlockkeySets blk
  utxos <- readUTxOs shelleyLedgerTablesHandle stBefore
  
  let TickedShelleyLedgerState
        { tickedShelleyLedgerTransition
        , tickedShelleyLedgerState
        } = stBefore

  ledgerResult <-
    f
      globals
      (tickedShelleyLedgerState `stow` utxos)
      ( let b = shelleyBlockRaw blk
            h' = mkHeaderView (SL.blockHeader b)
         in SL.Block h' (SL.blockBody b)
      )

  let track ::
        Map SL.TxIn (TxOut era) ->
        Map SL.TxIn (Delta (TxOut era))
      track = calculateDifference utxos

  newHandle <- duplicateWithDiffs shelleyLedgerTablesHandle (track newUtxos)
  return $
    ledgerResult <&> \newNewEpochState ->
      ShelleyLedgerState
            { ...
            , shelleyLedgerTablesHandle = newHandle
            , ...
            }

Further simplifications would come with this:

• No need for mk anywhere in the abstract layer
• The LedgerDB will again be a sequence of LedgerStates only
• Delete all the HF level machinery for TxIns, TxOuts, HF Queries, Traversing tables etc etc.
• No need to define any of these for any new tables that belong only to shelley.

Even if we were to do this, the handles of the InMemory backend could be also very simplified to just have a pointer to the UTxO era, and we could even inject the whole UTxO set into the Ledger in that case (by having read = \_keys -> pure wholeTables) and have the ledger translate the UTxO set for us instead of needing CanUpgradeLedgerTables.

[jasagredo]

I have been discussing this with @lehins this morning. The main conclusion is:

For tables on disk, either the values can be upgraded to newer era on decoding (and possibly updated in the backend) or the whole table will be re-created with a replay from Genesis or migration on startup

The required property is:

Values have the forwards serializability property, like the TxOuts, or values are tuples with the era tag, for example (ProtVerLow @ConwayEra, ConwayAccountState ConwayEra).

This implies the simplification described in this ticket is feasible.

[lehins]

This property that is required will be handled by the ledger on the decoder level:

Values have the forwards serializability property, like the TxOuts, or values are tuples with the era tag, for example (ProtVerLow @ConwayEra, ConwayAccountState ConwayEra).

So, we'll be able to guarantee that no extra work for Ledger tables will be necessary for era boundary transitions.

There might be cases in the future when we need to refactor what is stored in Ledger tables, but that would happen with a node version upgrade, not a protocol version upgrade and would require replay from Genesis, like it happens today in case of Ledger state serialization changes. Such occurrences should be rare, since Ledger tables are planned to store fairly stable types.