Skip to content

Migrate to a new S3 bucket#1089

Merged
edmorley merged 1 commit intomainfrom
migrate-s3-bucket
Oct 6, 2020
Merged

Migrate to a new S3 bucket#1089
edmorley merged 1 commit intomainfrom
migrate-s3-bucket

Conversation

@edmorley
Copy link
Copy Markdown
Member

@edmorley edmorley commented Oct 6, 2020

Since:

  • We want the S3 bucket to be owned by a different AWS account and it's not possible to transfer ownership of an existing bucket.
  • In the future we want to rebuild some of the Python runtime archives (for example to improve the sqlite3 handling, or to tweak the compile flags used), and it will be easier to reason about the change if we can guarantee only recent buildpack versions are using the assets rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the --dryrun):

aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'

The files that were --excluded are those that are no longer used, or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future (ie: we won't be deleting it), since builds using older versions of this buildpack (either due to pinning to a tag or via a fork) will still be using assets from it.

Closes W-8060097.

@edmorley edmorley self-assigned this Oct 6, 2020
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
  not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
  (for example to improve the sqlite3 handling, or to tweak the compile
  flags used), and it will be easier to reason about the change if we
  can guarantee only recent buildpack versions are using the assets
  rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the `--dryrun`):

```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'
```

The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.

Closes W-8060097.
@edmorley edmorley marked this pull request as ready for review October 6, 2020 06:47
@edmorley edmorley requested a review from a team as a code owner October 6, 2020 06:47
@edmorley edmorley merged commit b250300 into main Oct 6, 2020
@edmorley edmorley deleted the migrate-s3-bucket branch October 6, 2020 08:23
scorphus added a commit to scorphus/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
scorphus added a commit to scorphus/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
cezarsa pushed a commit to tsuru/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
dryan pushed a commit to dryan/heroku-buildpack-python that referenced this pull request Nov 19, 2020
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
  not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
  (for example to improve the sqlite3 handling, or to tweak the compile
  flags used), and it will be easier to reason about the change if we
  can guarantee only recent buildpack versions are using the assets
  rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the `--dryrun`):

```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'
```

The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.

Closes @W-8060097@.
joaodaher pushed a commit to edukorg/tsuru-platforms that referenced this pull request Mar 1, 2021
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants