[Discussion] Training 1.5B model on Google Cloud

[EibrielInv]

Hi, I'm looking to train GPT-2 in Spanish. And I would like to know exactly what hardware is needed.

On the readme you state that was trained Using Cloud TPU Pod v3-256 to train 22w steps.

• Are you referring to the following TPU POD?

TPU (v3) type	TPU v3 cores	TPU total memory	Available zones
v3-256	256	4 TiB	europe-west4-a

• Could you tell me how many TPU time was needed to setup and run the training?

• And do you remember what other services were needed? How much storage? And for how long?

Thank you!