Commit 820d0a2
authored
fix(firestore): enforce backpressure in BulkWriter (#12938)
This PR addresses a critical issue where the Firestore `BulkWriter`
could **silently drop document writes** without notifying the caller,
particularly under high load or when the process context was canceled.
## 🐛 Issue and Root Causes
Investigation revealed that the current implementation of `BulkWriter`
bypassed the client's built-in safety and resource management
mechanisms:
1. **Disabled Backpressure:** All document writes were enqueued with a
size of `0`, effectively disabling the `BufferedByteLimit`
(https://pkg.go.dev/google.golang.org/api/support/bundler#Bundler)
enforcement. This allowed the internal buffer to grow without bound,
leading to memory pressure and potential Out-of-Memory (OOM) crashes.
2. **Ignored Queuing Errors:** The internal `write` function ignored
return values from `bundler.Add`, meaning queuing failures were never
reported to the user.
## ✅ Proposed Fix
The fix moves `BulkWriter` to a managed resource model that respects
backpressure and ensures loud failures:
* **Runtime Size Calculation:** Computes the actual serialized size of
each write using `proto.Size(w)`.
* **Enforced Backpressure:** Replaces `Add(j, 0)` with `AddWait(ctx, j,
estimatedSize)`. This ensures that the producer (application code)
blocks if the internal 1GB buffer limit is reached, preventing unbounded
memory growth.
## 📌 Benefits
* **Data Integrity:** Guarantees that documents are either successfully
queued or returned with an explicit error.
* **System Stability:** Prevents OOM crashes by capping memory usage and
slowing down producers that outpace the network.
* **Alignment:** Brings the Go SDK into parity with the backpressure
behavior found in other Firestore SDKs like Java and Node.js.
#### Java Implementation
The Java SDK uses an asynchronous "task" model to manage writes.
* **Concurrency:** It leverages async threads (BulkCommitBatch) to
handle parallel requests.
* **Backpressure:** It implements a buffer limit on the number of
pending operations to prevent memory exhaustion. When this limit is
reached, subsequent attempts to queue writes will block the producer
until space is available.
#### Node.js Implementation
Node.js follows a similar pattern but is optimized for its event-driven
architecture.
* **Buffering:** It automatically buffers writes into batches and
ensures they are sent in order.
* **Memory Management:** Similar to Java, it uses an internal buffer
limit to impose backpressure on the event loop, preventing an unbounded
queue of pending promises.
#### Python Implementation
The Python SDK is designed to be user-friendly by hiding the
complexities of asynchronous execution.
* **Parallelization:** It uses a `ThreadPoolExecutor` to send batches in
parallel. This allows users to gain performance benefits without
manually managing an event loop or using `asyncio`.
* **Rate Limiting:** It includes a dedicated `RateLimiter` class to
manage the ramp-up of write traffic.
## Impact Analysis
The "breaking" change here is that Create might now block. However:
* If the user's load is within normal limits, they won't notice a
difference (the 1GB buffer is large).
* If the user's load is excessive, they are already experiencing silent
failures or OOMs. Blocking is the correct "fail-safe" state for their
application's stability.
* The BulkWriter methods already return an error. Returning a "context
deadline exceeded" error from a blocking Create call is a valid and much
more helpful response than returning nil and dropping the write.
Fixes #114221 parent 69993a5 commit 820d0a2
1 file changed
Lines changed: 14 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
297 | 298 | | |
298 | 299 | | |
299 | 300 | | |
300 | | - | |
301 | 301 | | |
302 | 302 | | |
303 | 303 | | |
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
310 | | - | |
311 | | - | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
312 | 313 | | |
313 | 314 | | |
314 | 315 | | |
| |||
360 | 361 | | |
361 | 362 | | |
362 | 363 | | |
363 | | - | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
364 | 374 | | |
365 | 375 | | |
366 | 376 | | |
| |||
0 commit comments