Script ends with announcment "All photos have been downloaded" when they have not

[sailingbikeruk]

Overview

Downloading a large volume of photos, the script would occasionally end with the message "All photos have been downloaded" when they clear had not. this occurred 7 or 8 times during the download of 40,000 photos

Steps to Reproduce

1. I simply had to restart the download and wait, it would stop eventually, it couldn't be reproduced on demand

Expected Behavior

continue until authentication failed`expired or all images had been downloaded, give an appropriate message

Actual Behavior

script randomly stopped with the message "All photos have been downloaded"
EDIT: script seems to stop after downloading about half of the available images (see context)

Context

Large download over several days, happened at the following points, different totals are because of another error raised as #615
As I worked through this it appears to only download about half of what is available then stop.

13691/27286
4096/8129
2034/4033
1000/1999
500/999
299/499
100/200

[AndreyNikiforov]

I suspect that some files are skipped due to errors. If that is the case, then it is expected behavior to skip and continue. User experience could have been better in this case of course, e.g. summarizing that some files were skipped.

[AndreyNikiforov]

changing from bug to enhancement to improve summarizing output.

[haraldan]

I have the same issue every time I use the script, with no indication of errors during the download. Output simply shows each file downloaded and then deleted in iCloud, until it stops, leaving half of the photos unprocessed.

[haraldan]

@sailingbikeruk, did you find an explanation or a workaround for this issue?

[AndreyNikiforov]

I have the same issue every time I use the script, with no indication of errors during the download. Output simply shows each file downloaded and then deleted in iCloud, until it stops, leaving half of the photos unprocessed.

Do the rest of the iCloud content getting downloaded when you run icloudpd multiple times? Where do you check that half of the photos are not processed? I assume that unprocessed photos are not excluded by any filters in icloudpd. Just trying to narrow down the issue.

[haraldan]

Yes, if I run the script multiple times, I can eventually download all the content. I used it a couple of years ago with the same iCloud account, and I had the same issue then.

This time I used the --keep-icloud-recent-days flag instead of the --delete-after-download, and I think that the photos that are being kept on iCloud were downloaded completely on the first try. Approximately, but not exactly half of the ones that were being deleted were not, though. This happened every time until there were only 38 left, which were downloaded completely. The iCloud account had around 14500 photos, and here are the numbers that were downloaded each time:

7537
3513
1579
805
393
204
103
38

It definitely looks like there is a pattern. Running the script with --only-print-filenames correctly identifies all the photos that are left, but when running the actual download, it stops about halfway there.

The script also doesn't download the photos sequentially, there are seemingly random chunks of several consecutive photos each that are left in iCloud every time.

[dojoca]

Just adding my voice to this thread - I’ve got the same problem. Downloads exactly half the photos, then stops. No errors, everything is as if it’s gone perfectly.

I’ve also encountered this exact same issue when using a python script. No matter how I changed it, the script would also only download exactly half the photos. Makes me wonder if it’s an issue in pyicloud, or the icloud installed on my Ubuntu system?

[electricalgorithm]

In what sense do we identify bugs or failures as enhancements, @AndreyNikiforov? This bug must be addressed as critical since the project's main idea is "to download". If the download compilation is not finished, and the program itself even sees the files that it couldn't download (when running --only-print-filename), thee exit code shall have the error. We cannot trust this behaviour in a cronjob.

I suggest to change the label.

[iowk]

With --delete-after-download, I think when some items are downloaded and deleted, the offset of the remaining items will be forwarded.

But we still query with increased offset, causing some items to be ignored.

Let's say, we have 400 items in the photos db, with id 1~400.

We first query with offset: 0, items 1~100 are downloaded and removed.

Now, item 101 becomes the first item in the db. However, we still query with offset: 100 next.

As a result, items 201~300 are queried next, and items 101~200 are ignored completely.

Always use offset: 0 when --delete-after-download is specified could probably fix this.

@dojoca Did you run with --delete-after-download? Doesn't seem so as #616 describes.

If not, maybe there is another reason.