Add encoding + compression metrics to columnar shuffle

[andygrove]

What is the problem the feature request solves?

We track encoding metrics for native shuffle but not yet for columnar shuffle:

Describe the potential solution

No response

Additional context

No response

[0lai0]

Hi @andygrove , I would like to confirm that I am reproducing issue #1212 correctly.

When I run with spark.comet.shuffle.mode=jvm, I get CometColumnarExchange, and the metric shows:
encoding and compression time: 0 ms (single value) and no total (min, med, max) distribution for that metric.

When I switch to spark.comet.shuffle.mode=native, I get CometExchange, and encoding and compression time total (min, med, max) has non-zero values.

Is this is behavior that #1212 is intended to address?

spark.comet.shuffle.mode=jvm

spark.comet.shuffle.mode=native