Skip to content

Handle resuming tile cache downloads#242

Open
chrowe wants to merge 1 commit into
mainfrom
codex/update-cache_city_data-for-existing-tiles
Open

Handle resuming tile cache downloads#242
chrowe wants to merge 1 commit into
mainfrom
codex/update-cache_city_data-for-existing-tiles

Conversation

@chrowe

@chrowe chrowe commented Dec 23, 2025

Copy link
Copy Markdown
Contributor

Summary

  • detect existing tile caches and resume downloading only missing tiles instead of wiping the cache
  • update tile processing notice handling and failure detection to work for both S3 and local cache paths
  • add tests covering tile ID parsing and cache resumption

Testing

  • pytest tests/test_tile_cache_resumption.py (fails: missing boto3 dependency in test environment; network access blocked when attempting to install)

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1323 to +1326
fishnet["index"] = fishnet.index
existing_tile_paths = self._list_existing_tile_filepaths(target_uri)
completed_tile_ids = self._get_completed_tile_ids(existing_tile_paths)
unretrieved_tiles = fishnet.drop(completed_tile_ids, errors="ignore")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Tile cache completeness check ignores S3 pagination

_has_incomplete_tile_cache determines whether to rebuild by listing existing tile files, but for S3 targets _list_existing_tile_filepaths delegates to _list_all_tiff_filepaths_in_s3_folder, which issues a single list_objects_v2 call (default 1,000 key limit). On caches with more than 1,000 tiles, completed_tile_ids will only include the first page of objects, so unretrieved_tiles stays non‑empty and the cache is always treated as incomplete even when fully populated, forcing unnecessary reprocessing for large cities.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant