improve test coverage by AndreyNikiforov · Pull Request #171 · icloud-photos-downloader/icloud_photos_downloader

AndreyNikiforov · 2020-10-10T19:17:43Z

No description provided.

coveralls · 2020-10-10T19:18:50Z

Pull Request Test Coverage Report for Build 170

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.6%) to 100.0%

Totals
Change from base Build 163:	0.6%
Covered Lines:	471
Relevant Lines:	471

💛 - Coveralls

AndreyNikiforov · 2020-10-10T21:40:35Z

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

menkej · 2020-10-11T13:55:58Z

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

This is my last change for the throttling, it seems... How could we force this exception in a test case? Just ignoring these lines (pragma: no cover) is not really a good solution...

AndreyNikiforov · 2020-10-11T17:42:22Z

Travis says that icloudpd/base.py 251 3 99% 281-284 missing coverage. Locally (in docker) coverage is 100%. Checking...

This is my last change for the throttling, it seems... How could we force this exception in a test case? Just ignoring these lines (pragma: no cover) is not really a good solution...

I added a test and now coverage is 100% in Travis.

menkej

Excellent work @AndreyNikiforov ! Makes sense to go for single threading here, even though I'm still a little unsure about the multithreaded downloads. Multithreading can be a beast and I remember we had issues with it last year. I did not find the calm yet to dive into the current implementation...

AndreyNikiforov · 2020-10-12T03:29:20Z

@menkej I am skeptical about multithreaded downloading from a single source with a goal to improve speed because 1) network speed will be a limiting factor if no throttling at the source 2) if throttled at the source, it may be too naïve to expect abuse prevention to be implemented at connection level only. I would be curious to see some test results (single threaded vs multi threaded downloading)

menkej · 2020-10-16T20:46:16Z

@menkej I am skeptical about multithreaded downloading from a single source [...] I would be curious to see some test results (single threaded vs multi threaded downloading)

I run some test downloads. I am on a 100 MBit connection. No clear lab conditions - Netflix and Skype session running. Downloading 1000 recent files.

default setting (multithreaded)
87,59s user 30,36s system 11% cpu 16:33,71 total
77,79s user 27,50s system 12% cpu 13:58,24 total
77,67s user 27,30s system 13% cpu 12:34,12 total

--threads-num 1
79,00s user 27,64s system 14% cpu 12:34,61 total
79,17s user 27,51s system 11% cpu 15:04,41 total
78,30s user 27,80s system 14% cpu 12:36,18 total

Just three runs. Just 1000 pics. Seeing not much of a difference really.

Multithreading might make sense for parallel image/video conversions and stuff. For download... not so sure...

Maybe it makes a difference on a clear line with a big pipe...

AndreyNikiforov · 2020-10-17T18:54:47Z

@menkej I am skeptical about multithreaded downloading from a single source [...] I would be curious to see some test results (single threaded vs multi threaded downloading)

I run some test downloads. I am on a 100 MBit connection. No clear lab conditions - Netflix and Skype session running. Downloading 1000 recent files.

default setting (multithreaded)
87,59s user 30,36s system 11% cpu 16:33,71 total
77,79s user 27,50s system 12% cpu 13:58,24 total
77,67s user 27,30s system 13% cpu 12:34,12 total

--threads-num 1
79,00s user 27,64s system 14% cpu 12:34,61 total
79,17s user 27,51s system 11% cpu 15:04,41 total
78,30s user 27,80s system 14% cpu 12:36,18 total

Just three runs. Just 1000 pics. Seeing not much of a difference really.

Multithreading might make sense for parallel image/video conversions and stuff. For download... not so sure...

Maybe it makes a difference on a clear line with a big pipe...

Thanks for running tests, @menkej ! We can look at deprecating multithreading support eventually to reduce complexity

boredazfcuk · 2020-10-18T22:50:17Z

I ran some tests on this tonight. My home internet connection is only 36Mbps.

Here's two log files generated:

Multi-threaded - https://pastebin.com/6RKYfgsy
Single-threaded - https://pastebin.com/7zhgtjc7

Results:

threads   start      finish      total
1         22:55:00   23:16:32    21m 32s
default   22:35:00   22:50:42    15m 42s

To sum up, I see a significant performance benefit to multi-threaded downloads. However, multi-threaded results in occasional errors e.g.:

ERROR    IOError while writing file to /home/user/iCloud/2020/camphoto_1804928587.jpg! You might have run out of disk space, or the file might be too large for your OS. Skipping this file...

AndreyNikiforov · 2020-10-19T20:57:40Z

@boredazfcuk thanks for measuring time! do you mind repeat same test multiple times to eliminate of any possible network speed fluctuations etc?

I think it would be helpful to add a summary at the end of the the icloudpd run with total bytes downloaded, overall time, avg speed, and may be something else. Will be valuable beyond just perf tests.

boredazfcuk · 2020-10-21T23:59:45Z

Sorry for not getting back sooner guys... I was busy last night and most of tonight too.

I've run the tests again, but I've only had time to do one of each. The Mrs had a late night and she kills my connection with Netflix until she falls asleep lol

I should have more time tomorrow to repeat again, hopefully two or three times.

The results were pretty much the same as yesterday, multi-threaded was significantly faster, with single threaded taking nearly twice as long:

Threads     Start       Finish      Total
Default     23:19:59    23:33:40    13m 41s
1           00:09:59    00:34:51    24m 52s

Log files:
Multi-thread - https://pastebin.com/7Dku3gUg
Single-thread - https://pastebin.com/Ya7Wi4Cs

boredazfcuk · 2020-10-23T12:05:38Z

I managed to do some more testing overnight, as I put a speed test routine into my container and kicked off 8 runs (4-multi and 4-single).

I am no longer seeing the large delays on single threaded downloads. Here are the details and times calculated:

Multi:
https://pastebin.com/94E77DJx - Start: 00:25:32 End: 00:37:42 Total: 0:12:10
https://pastebin.com/PzELtKis - Start: 23:08:16 End: 23:20:09 Total: 0:11:53
https://pastebin.com/7SU4K52e - Start: 23:33:16 End: 23:45:36 Total: 0:12:20
https://pastebin.com/JER2eKJS - Start: 23:59:21 End: 00:12:33 Total: 0:13:12

Single
https://pastebin.com/EzXds2Vz - Start: 23:45:37 End: 23:59:20 Total: 0:13:43
https://pastebin.com/XNVpPd8q - Start: 00:12:34 End: 00:25:31 Total: 0:12:57
https://pastebin.com/DT2bKR6T - Start: 23:20:10 End: 23:33:15 Total: 0:13:05
https://pastebin.com/5TQRTLkw - Start: 00:37:43 End: 00:49:22 Total: 0:11:39

There seems to be a bug in my container that calculates the total time taken incorrectly, which is why they all show as taking 20:32 in the log files. When calculating manually, the times shown above are accurate.

I'm not sure why this is different to before. As I could reliably reproduce it before under the same network conditions. These times have been taken using the latest version which defaults to single threaded.

AndreyNikiforov · 2020-10-23T15:40:18Z

@boredazfcuk thanks a lot for the test! Not we can proceed with #180 with confidence.

boredazfcuk · 2020-10-23T16:40:14Z

@boredazfcuk thanks a lot for the test! Not we can proceed with #180 with confidence.

Time calculation bug should have been fixed and I've pushed latest version to DockerHub. People should be able to replicate my tests using this script I created:

#!/bin/bash
apple_id="notmy@appleid.com"
apple_password="notmypasswordeither"
mkdir --parents ~/single ~/multi

CreateCookies(){
   docker run -it --rm \
      --name iCloudPD_Test_single \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/single:/config \
      boredazfcuk/icloudpd
   docker run -it --rm \
      --name iCloudPD_Test_multi \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env multi_thread=Enabled \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/multi:/config \
      boredazfcuk/icloudpd
}

RunSingleThread(){
   docker run --rm \
      --name iCloudPD_Test_single \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/single:/config \
      boredazfcuk/icloudpd
}

RunMultiThread(){
   docker run --rm \
      --name iCloudPD_Test_multi \
      --env apple_id="${apple_id}" \
      --env apple_password="${apple_password}" \
      --env multi_thread=Enabled \
      --env speed_test=True \
      --env TZ=Europe/London \
      --volume ~/multi:/config \
      boredazfcuk/icloudpd
}

CreateCookies
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"
RunMultiThread | tee "~/MultiThread_$(date +%H%M%S).log"
RunSingleThread | tee "~/SingleThread_$(date +%H%M%S).log"

* remove obsolete python 2 code to improve test coverage * limit tests with one thread for determinism * add test for fetching albums

remove obsolete python 2 code to improve test coverage

8b3a21e

AndreyNikiforov requested review from menkej and ndbroadbent October 10, 2020 19:17

menkej approved these changes Oct 10, 2020

View reviewed changes

limit tests with one thread for determinism

cf9a17f

AndreyNikiforov added 2 commits October 11, 2020 08:42

Merge branch 'master' into bug/fix-coverage

3b0c71f

add test for fetching albums

65ec1cc

AndreyNikiforov changed the title ~~remove obsolete python 2 code to improve test coverage~~ improve test coverage Oct 11, 2020

AndreyNikiforov requested a review from menkej October 11, 2020 17:50

menkej approved these changes Oct 11, 2020

View reviewed changes

AndreyNikiforov merged commit 09570f2 into icloud-photos-downloader:master Oct 12, 2020

AndreyNikiforov deleted the bug/fix-coverage branch October 12, 2020 03:30

AndreyNikiforov mentioned this pull request Oct 21, 2020

Remove multi-threaded downloads #180

Closed

ndbroadbent mentioned this pull request Oct 27, 2020

remove multi-threaded downloading #191

Merged

rhoopr referenced this pull request in rhoopr/kei Feb 8, 2026

improve test coverage (#171)

21ac9b2

* remove obsolete python 2 code to improve test coverage * limit tests with one thread for determinism * add test for fetching albums

Conversation

AndreyNikiforov commented Oct 10, 2020

Uh oh!

coveralls commented Oct 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 170

💛 - Coveralls

Uh oh!

AndreyNikiforov commented Oct 10, 2020

Uh oh!

menkej commented Oct 11, 2020

Uh oh!

AndreyNikiforov commented Oct 11, 2020

Uh oh!

menkej left a comment

Choose a reason for hiding this comment

Uh oh!

AndreyNikiforov commented Oct 12, 2020

Uh oh!

menkej commented Oct 16, 2020

Uh oh!

AndreyNikiforov commented Oct 17, 2020

Uh oh!

boredazfcuk commented Oct 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndreyNikiforov commented Oct 19, 2020

Uh oh!

boredazfcuk commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boredazfcuk commented Oct 23, 2020

Uh oh!

AndreyNikiforov commented Oct 23, 2020

Uh oh!

boredazfcuk commented Oct 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Oct 10, 2020 •

edited

Loading

boredazfcuk commented Oct 18, 2020 •

edited

Loading

boredazfcuk commented Oct 21, 2020 •

edited

Loading