fix: potential buffer overflow in flood_fill due to unchecked stack size

[zendy199x]

Summary

The flood_fill function uses a stack to track fill segments without any bounds checking on stack size. If the image contains a large connected region, the stack can grow uncontrollably and lead to stack overflow or memory corruption. This is particularly dangerous in recursive-like operations where the stack depth can grow linearly with image dimensions.

Changes

• example_transcoding/utils.cpp

Add a maximum stack size limit and return an error or abort if exceeded. Consider switching to an iterative approach with heap allocation or using a more memory-efficient algorithm like scanline flood fill.

Testing

• Verified changes follow existing project conventions
• Confirmed no regressions in affected code paths

[Copilot]

Pull request overview

This PR aims to harden image_u8::flood_fill() against unbounded growth of its internal fill-segment work stack, which could otherwise lead to excessive memory usage in worst-case images.

Changes:

• Add a maximum stack size guard to image_u8::flood_fill() and return early if exceeded.
• Add a new Python test file intended to validate flood fill overflow protection, normal operation, and out-of-bounds behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
example_transcoding/utils.cpp	Adds MAX_STACK_SIZE check during the flood fill loop to cap work-stack growth.
tests/test_utils.py	Introduces pytest-style tests targeting flood fill behavior and the new stack limit behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.