[Bug]: write table by FFI call from go may memory leak?

[z2665]

What happened?

memory leak when call delta-rs to write table

Expected behavior

delta-rs uses finite memory instead of infinite growth

Operating System

Linux

Binding

Rust

Bindings Version

0.29.3

Steps to reproduce

MRE: https://github.com/z2665/go_ffi_delta_rs
@abhi-airspace-intelligence

1. The program receives a -target flag (local or s3) and prepares that table once (creating or cleaning as needed).
2. For each of the ten iterations it:
   • Opens the target table.
   • Appends 100 synthetic rows.
   • Closes the table and records RSS in megabytes.
3. Iterations execute every five seconds unless interrupted with Ctrl+C.
4. After the loop completes, the program writes a single plot named <target>_memory.png.

Run the experiment with:

cd  go_ffi_delta_rs
GOOS=linux GOARCH=amd64 go run . -target local
GOOS=linux GOARCH=amd64 go run . -target s3

write to local result

write to s3 result

[rtyler]

We do not support any FFI or Golang bindings. If you are able to demonstrate the issue with a pure Rust implementation then I can take a look at it. My best guess here would be that pointers are not being cleaned up in the FFI implementation, and therefore the drop is not being invoked when it should be

[z2665]

I’m very sorry to hear this. In fact, in the MRE , the opening and closing of the FFI pointers is already managed in a very straightforward way: each time we open them, and then close them after the write is completed. That’s also the part I can’t understand—it seems as if delta-rs has some kind of global memory that keeps accumulating. We’ll continue to investigate and see whether we can find another solution.