Improve local search performance#4818
Closed
bartvanandel wants to merge 2 commits intoScoopInstaller:developfrom
Closed
Improve local search performance#4818bartvanandel wants to merge 2 commits intoScoopInstaller:developfrom
bartvanandel wants to merge 2 commits intoScoopInstaller:developfrom
Conversation
bartvanandel
force-pushed
the
fix/4239_searchPerformance
branch
from
b925a13 to
0d59e9b
Compare
rashil2000
reviewed
Mar 16, 2022
Member
|
Sorry, but this approach took longer time in my PC:
And I've tried several times. What about yours @rashil2000 ? |
Author
|
Strange. On my system, BTW due to (presumably) file system caching, next runs of the same command and on the same develop branch are significantly faster. So measuring this in a fair way may be tricky, especially since in practice, usually you don't search the same thing twice in a row. |
bartvanandel
force-pushed
the
fix/4239_searchPerformance
branch
from
0d59e9b to
00a10cc
Compare
Member
|
I'm seeing a marginal improvement.
|
Member
|
Search twice or use different methods? Same here, original method 8sec, this 13sec ( Buckets list: Name Source Updated Manifests
---- ------ ------- ---------
dorado https://github.com/h404bi/dorado 2022/3/18 8:10:29 221
extras https://github.com/ScoopInstaller/Extras 2022/3/18 8:33:48 1437
java https://github.com/ScoopInstaller/Java 2022/3/17 20:26:07 220
main https://github.com/ScoopInstaller/Main 2022/3/18 8:32:58 992
nerd-fonts https://github.com/matthewjberger/scoop-nerd-fonts 2022/3/16 3:11:59 187
nih https://github.com/niheaven/scoop-nih.git 2022/3/18 9:53:30 28
rasa https://github.com/rasa/scoops 2022/3/15 5:01:36 70
tests https://github.com/ScoopInstaller/Tests 2022/3/17 4:32:31 62
versions https://github.com/ScoopInstaller/Versions 2022/3/18 9:55:40 280 |
Member
Different methods. See |
Member
|
More feedback is needed, IMO. Don't know why this one took more time. |
Member
|
Member
|
Test it again, same result: Scoop\apps\scoop took 23s
❯ git -C "current" checkout develop
Switched to branch 'develop'
Your branch is behind 'origin/develop' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
Scoop\apps\scoop
❯ Measure-Command { scoop search rstudio }
'extras' bucket:
'versions' bucket:
Days : 0
Hours : 0
Minutes : 0
Seconds : 10
Milliseconds : 352
Ticks : 103529888
TotalDays : 0.000119826259259259
TotalHours : 0.00287583022222222
TotalMinutes : 0.172549813333333
TotalSeconds : 10.3529888
TotalMilliseconds : 10352.9888
Scoop\apps\scoop took 10s
❯ git -C "current" checkout fix/4239_searchPerformance
Switched to branch 'fix/4239_searchPerformance'
Scoop\apps\scoop
❯ Measure-Command { scoop search rstudio }
'extras' bucket:
'versions' bucket:
Days : 0
Hours : 0
Minutes : 0
Seconds : 23
Milliseconds : 881
Ticks : 238818129
TotalDays : 0.000276409871527778
TotalHours : 0.00663383691666667
TotalMinutes : 0.398030215
TotalSeconds : 23.8818129
TotalMilliseconds : 23881.8129Really strange... |
Author
|
Well, if the results are this unpredictable / unreliable, I'd suggest to not merge at the moment. I don't currently have the time to dig any further. |
5 tasks
5 tasks
Member
|
Closing in favour of #5644 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improve local search performance by pruning candidates using
git ls-filesandgit grep. This drastically reduces the number of files that are read into memory to find the desired results.The code accounts for buckets that are not maintained using
git. Not sure if that really applies, but for buckets that are not maintained usinggit, the code will fall back to the previous approach.Motivation and Context
Closes #4239
How Has This Been Tested?
Compared the output of the previous search method with the new method. The code uses standard PowerShell and git functions that have existed for a long time.
Checklist:
I have updated the documentation accordingly.N/A. Well, I sparsely documented the approach in code.I have updated the tests accordingly.N/A, there were no pre-existing tests.