Speedup using NVIDIA Apex

[llidev]

Hi,

According to PR #116, we should be able to achieve a 3-4 x speed up for both bert-base and bert-large. However, I can only achieve 2x speed up with bert-base. My docker image uses CUDA9.0 while the discussion in the PR #116 is based on CUDA10.0... I am wondering if that makes the difference....

Thanks

[thomwolf]

Maybe. You can try with pytorch docker image dockerhub 1.0-cuda10.0-cudnn7 to debug, as we did in the discussion in PR #116.

[llidev]

Just verified that CUDA10.0 makes 4x speedup. It should be better to include this in the main document.

[Oxi84]

What GPU do you run ... and how do you increase such a speedup? Is this possible with gtx 1080?

Thanks