FileNotFoundError: [Errno 2] No such file or directory (file_size = os.stat(download_path).st_size)

[GuusDeGraeve]

Overview

Script suddenly crashes on photos of a specific type (whitagram). After downloading hundreds of photos, the script suddenly crashes (and keeps crashing) on a specific photo created with the Whitagram app (an app that adds white borders around the photo). I think it occurs when you have size set to "medium" and specifically when a version of the file does not have a medium size or something? It seems to only happen with these Whitagram photos, when I try to remove this photo from iCloud, it just crashes at the next Whitagram photo.

2021-05-31 14:42:02 INFO     Skipping IMG_1569.MOV, only downloading photos.
2021-05-31 14:42:02 INFO     Skipping IMG_1568.MOV, only downloading photos.
2021-05-31 14:42:02 INFO     Skipping 2fc5f66422a34dca9406da918aedbfd5.mov, only downloading photos.
2021-05-31 14:42:02 INFO     /data/icloud/import/XXX/2020/10/11/Whitagram-Image-medium.jpg already exists.
Traceback (most recent call last):
  File "/usr/bin/icloudpd", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/icloudpd/base.py", line 562, in main
    download_photo(consecutive_files_found, item)
  File "/usr/lib/python3.8/site-packages/icloudpd/base.py", line 456, in download_photo
    file_size = os.stat(download_path).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/data/icloud/import/XXX/2020/10/11/Whitagram-Image.jpg'

Steps to Reproduce

1. Use whitagram to edit a photo and save it to your library.
2. Run the icloudpd script
3. Wait for the error to occur.

Expected Behavior

The script should not crash in this case, and even if a photo fails, I think it should be catched and just skip the photo to continue with the next one.

Actual Behavior

The script completely crashes.

[4gra]

Also experiencing this with photos downloaded by a previous version of the application - more to do with a change in the way they're named (I am downloading originals in every case):

  [...]
  File ".local/lib/python3.7/site-packages/icloudpd/base.py", line 559, in main
    download_photo(consecutive_files_found, item)
  File ".local/lib/python3.7/site-packages/icloudpd/base.py", line 454, in download_photo
    file_size = os.stat(download_path).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/backups/2023/01/31/IMG_2793.JPG'

well, this is true, /backups/2023/01/31/IMG_2793.JPG doesn't exist, but that's because the previous version named these files like IMG_2793-original.JPG. I wonder if at this stage it's easier for me to just rename all the files ... but that's a nuisance.

[AndreyNikiforov]

@GuusDeGraeve, @4gra what version of the app are you running? How are you running it (source code, docker, exe)? Is that error persistent (reproducible all the time on your set up)? Did you try prior/other versions and/or other methods?

A lot of questions, but I hope they help narrowing down the problem.

[tinder-guusdegraeve]

It's been two years since I created this issue and I stopped using this package since I kept getting the error and there was no response on my issue here.

But to answer your question, I remember I was using the package within a docker container. I was using the latest version at the moment I created the issue (I don't remember an actual version number). I believe the issue I created explains the problem pretty well and even pinpoints a possible cause.

It could as well be that the issue I faced at the time was fixed in later versions, I can't test it at the moment.

[4gra]

@GuusDeGraeve, @4gra what version of the app are you running? How are you running it (source code, docker, exe)? Is that error persistent (reproducible all the time on your set up)? Did you try prior/other versions and/or other methods?

It's the latest version from pip, which reports icloudpd, version 1.7.3 running on Debian Buster. No docker.

Persistent on repeated runs (5 or 6?) on the same file; then as I moved individual files aside, it would pick a new 'victim' and crash in the same way. Shortly after that I wrote a small shell script to rename all the -original files (I didn't really like the naming scheme anyway and was keen to get my overdue backup complete!) so didn't investigate other versions.

I back up two libraries, both in the multiple thousands.

Another observation is that I normally run it with --until-found 1000 but after the first crash ran without this argument to fill in any gaps. This did not fix the problem.

I've only ever downloaded originals; first explicitly using the command line flat, and now implicitly since the default behaviour changed.

And once I'd renamed all files their base filenames (i.e. no longer containing -original), the crashes stopped.

The original version didn't ever have this problem, but it's very old dating from 2018-2019. I upgraded to use the new 2FA methods with the iCloud library.

A lot of questions, but I hope they help narrowing down the problem.

I'm happy to try a few things artificially (other versions etc) by renaming files back, but is there anything particularly useful to try?

[AndreyNikiforov]

@4gra pls check 1.8.1 and reopen this task if regression was not fixed.

[4gra]

Upgraded to 1.8.1, did mv IMG_1076{,-original}.HEIC, ran the downloader... no crash; all downloaded. Thank you!