Security: fix signed integer overflow in undo_pxr24_impl() (PXR24 decoder) (PixarAnimationStudios#2323)

In the `EXR_PIXEL_FLOAT` branch of `undo_pxr24_impl()`, the expressions

    (uint64_t)(w * 3)

compute the signed 32-bit product `w * 3` before the cast to `uint64_t`.
When `w` is large this is undefined behavior under the C standard; on
two's-complement builds without sanitizers the result wraps to a small
positive value, which can cause the bounds check

    if (nDec + (uint64_t)(w * 3) > outSize)

to pass incorrectly. If the check is bypassed the decode loop proceeds
to write `4*w` bytes through `dout`, potentially far beyond the allocated
output buffer.

Fix: cast `w` to `uint64_t` before multiplying so that both the bounds
check and the counter update are performed entirely in 64-bit unsigned
arithmetic:

    (uint64_t)w * 3   (cast before multiply, not after)

The `EXR_PIXEL_UINT` and `EXR_PIXEL_HALF` decode branches are unaffected:
they reuse the pre-computed `nBytes` variable, which is already formed as
`(uint64_t)(w) * (uint64_t)(bytes_per_element)`.

Also fix the symmetric issue in `apply_pxr24_impl()` (the encoder):
    lastIn += w * 4
advances a pointer by a signed 32-bit product; corrected to
    lastIn += (uint64_t)w * 4

Made-with: Cursor

Signed-off-by: Cary Phillips <cary@ilm.com>

Author: cary-ilm

pxr/imaging/plugin/hioOpenEXR/OpenEXR/OpenEXRCore/internal_pxr24.c

@@ -182,7 +182,7 @@ apply_pxr24_impl (exr_encode_pipeline_t* encode)
                     if (nOut + nBytes > encode->scratch_alloc_size_1)
                         return EXR_ERR_OUT_OF_MEMORY;
                     nOut += nBytes;
-                    lastIn += w * 4;
+                    lastIn += (uint64_t) w * 4;
 
                     ptr[0] = out;
                     out += w;
@@ -371,7 +371,7 @@ undo_pxr24_impl (
                     ptr[2] = lastIn;
                     lastIn += w;
 
-                    if (nDec + (uint64_t) (w * 3) > outSize)
+                    if (nDec + (uint64_t) w * 3 > outSize)
                         return EXR_ERR_CORRUPT_CHUNK;
 
                     for (int x = 0; x < w; ++x)
@@ -384,7 +384,7 @@ undo_pxr24_impl (
                         unaligned_store32 (dout, pixel);
                         ++dout;
                     }
-                    nDec += (uint64_t) (w * 3);
+                    nDec += (uint64_t) w * 3;
                     break;
                 }
                 default: return EXR_ERR_INVALID_ARGUMENT;