Preserve PhysicalExpr graph in proto round trip using Arc pointers as unique identifiers (apache#20037)

Replaces apache#18192 using the APIs in apache#19437.

Similar to apache#18192 the end goal here is specifically to enable
deduplication of `DynamicFilterPhysicalExpr` so that distributed query
engines can get one step closer to using dynamic filters.

Because it's actually simpler we apply this deduplication to all
`PhysicalExpr`s with the added benefit that we more faithfully preserve
the original expression tree (instead of adding new duplicate branches)
which will have the immediate impact of e.g. not duplicating large
`InListExpr`s.

Author: adriangb

Cargo.lock

@@ -69,6 +69,7 @@ datafusion-proto-common = { workspace = true }
 object_store = { workspace = true }
 pbjson = { workspace = true, optional = true }
 prost = { workspace = true }
+rand = { workspace = true }
 serde = { version = "1.0", optional = true }
 serde_json = { workspace = true, optional = true }
 

datafusion/proto/Cargo.toml

@@ -836,6 +836,14 @@ message PhysicalExprNode {
   // Was date_time_interval_expr
   reserved 17;
 
+  // Unique identifier for this expression to do deduplication during deserialization.
+  // When serializing, this is set to a unique identifier for each combination of
+  // expression, process and serialization run.
+  // When deserializing, if this ID has been seen before, the cached Arc is returned
+  // instead of creating a new one, enabling reconstruction of referential integrity
+  // across serde roundtrips.
+  optional uint64 expr_id = 30;
+
   oneof ExprType {
     // column references
     PhysicalColumn column = 1;