Improve madvise-failure logic in the pooling allocator

In bytecodealliance#12888 the failure of `madvise` or `mmap` was improved to be handled
in the pooling allocator but this only accounted for linear memories
rather than tables as well. This meant that if `madvise` or `mmap`
failed it could be possible to have the previous contents of one table
leak over to the next. There's no known actual situation in which this
would happen (`madvise` and `mmap` failing is mostly theoretical) but
this seems best to fix nonetheless.

Author: alexcrichton

crates/wasmtime/src/runtime/vm/instance/allocator/pooling/decommit_queue.rs

@@ -177,16 +177,15 @@ impl DecommitQueue {
         // Second, restore the various entities to their associated pools' free
         // lists. This is safe, and they are ready for reuse, now that their
         // memory regions have been decommitted.
-        //
-        // Note that for memory images the images are all dropped here and
-        // ignored if any decommits failed. This signifies how the state of the
-        // slot is unknown and needs to be paved over in the future. Also note
-        // that `bytes_resident` is probably too low, but there's no other
-        // precise way to know, so it's left here as-is and it'll get reset when
-        // the slot is reused.
         let mut deallocated_any = false;
         for (allocation_index, image, bytes_resident) in self.memories {
             deallocated_any = true;
+            // Note that for memory images the images are all dropped here and
+            // ignored if any decommits failed. This signifies how the state of the
+            // slot is unknown and needs to be paved over in the future. Also note
+            // that `bytes_resident` is probably too low, but there's no other
+            // precise way to know, so it's left here as-is and it'll get reset when
+            // the slot is reused.
             let image = if decommit_succeeded {
                 Some(image)
             } else {
@@ -197,8 +196,16 @@ impl DecommitQueue {
                     .deallocate(allocation_index, image, bytes_resident);
             }
         }
-        for (allocation_index, table, bytes_resident) in self.tables {
+        for (allocation_index, mut table, bytes_resident) in self.tables {
             deallocated_any = true;
+            // Like with memories,  if any decommit failed then we need to
+            // ensure that tables are still in a defined state. Unlike memories
+            // which track this via images tables are always assumed to be
+            // all-null on returning to the allocator. If the OS couldn't do it
+            // then manually do it ourselves here.
+            if !decommit_succeeded {
+                table.manually_memset_zeros();
+            }
             unsafe {
                 pool.tables
                     .deallocate(allocation_index, table, bytes_resident);

crates/wasmtime/src/runtime/vm/table.rs

@@ -1131,6 +1131,31 @@ impl Table {
             }
         }
     }
+
+    /// Manually resets all table elements to a null bit pattern.
+    ///
+    /// Used in the pooling allocator when `madvise` fails to reset pages back
+    /// to their original contents. In such a situation the previous contents of
+    /// the table are unknown so this resets them all to defined values.
+    pub fn manually_memset_zeros(&mut self) {
+        match self.element_type() {
+            TableElementType::Func => {
+                let (funcrefs, _lazy_init) = self.funcrefs_mut();
+                funcrefs.fill(MaybeTaggedFuncRef(None));
+            }
+            TableElementType::GcRef => {
+                // Note that this explicitly contains no barriers as all table
+                // GC elements should already have been dropped before returning
+                // the table back to the pool.
+                for r in self.gc_refs_mut() {
+                    *r = None;
+                }
+            }
+            TableElementType::Cont => {
+                self.contrefs_mut().fill(None);
+            }
+        }
+    }
 }
 
 // The default table representation is an empty funcref table that cannot grow.