During the 0.33.0 release process, our deephaven-server PyPi upload failed as it was too large (>200MB). We're requested and were granted a size update to 300MB, pypi/support#3663. If we hit that limit, we may be able to get another limit increase if necessary, but it would be better to figure out a proper long-term solution.
Since 0.33.0, we've increased our size to 260MB. Some of this is due to fastutil dependency in #5197, and S3 dependency (#4972, #5180).
Here are the biggest offenders:
$ ls -laS server/jetty-app/build/install/server-jetty/lib/ | awk -F " " '{print $5" "$9}' | head -n 15
91225191 deephaven-web-0.34.0-SNAPSHOT.jar
23599579 fastutil-8.5.13.jar
17790220 aws-crt-0.29.7.jar
9287211 deephaven-engine-table-0.34.0-SNAPSHOT.jar
8043128 groovy-3.0.18.jar
7915023 calcite-core-1.36.0.jar
7840045 deephaven-engine-tuplesource-0.34.0-SNAPSHOT.jar
5982371 zstd-jni-1.5.5-1.jar
5261602 kafka-clients-7.6.0-ccs.jar
4470534 hadoop-common-3.3.3.jar
3544133 s3-2.23.19.jar
3494327 deephaven-proto-backplane-grpc-0.34.0-SNAPSHOT.jar
3362359 hadoop-shaded-guava-1.1.1.jar
3180995 kotlin-reflect-1.8.21.jar
We may need a better strategy to manage increasing dependency size into the future. Trim web? Remove dependencies? A better system to manage optional dependencies? Jar trimming (https://github.com/Guardsquare/proguard)? Breaking up deephaven-server PyPi package?
We might say this is only an issue for deephaven-server on PyPi; arguably, we aren't as limited for our native application tarballs and docker images. That said, it is generally nice to maintain a smaller footprint whenever possible.
During the 0.33.0 release process, our deephaven-server PyPi upload failed as it was too large (>200MB). We're requested and were granted a size update to 300MB, pypi/support#3663. If we hit that limit, we may be able to get another limit increase if necessary, but it would be better to figure out a proper long-term solution.
Since 0.33.0, we've increased our size to 260MB. Some of this is due to fastutil dependency in #5197, and S3 dependency (#4972, #5180).
Here are the biggest offenders:
We may need a better strategy to manage increasing dependency size into the future. Trim web? Remove dependencies? A better system to manage optional dependencies? Jar trimming (https://github.com/Guardsquare/proguard)? Breaking up deephaven-server PyPi package?
We might say this is only an issue for deephaven-server on PyPi; arguably, we aren't as limited for our native application tarballs and docker images. That said, it is generally nice to maintain a smaller footprint whenever possible.