Another attempt to improve the search performance

[r15ch13]

Description

This is another attempt to improve the local search performance.

• Not using the manifest() function
• Use only one loop over $apps
• Push results to $list directly
• Use [System.IO.File]::ReadAllText() instead of Get-Content

Motivation and Context

Closes #4239

Relates to #4818

How Has This Been Tested?

Measure-Command { scoop search telegram }
Measure-Command { .\bin\scoop.ps1 search telegram }

Checklist:

• I have read the Contributing Guide.
• I have ensured that I am targeting the develop branch.
• I have updated the documentation accordingly.
• I have updated the tests accordingly.
• I have added an entry in the CHANGELOG.

[niheaven]

WIP or Ready for review?

[rashil2000]

Some more areas for optimization:

• Since we are listing lots of files in function apps_in_bucket, using [System.IO.Directory]::EnumerateFiles() instead of Get-ChildItem should be much faster (ref)
• Since we are deserializing a lot of JSON, using [System.Text.Json.Utf8JsonReader] should be faster than ConvertFrom-Json
• While searching in remotes, since we are making multiple requests, a .NET method should be faster than Invoke-WebRequest
• I'm not sure I understand this, but why do we use Select-Object -ExpandProperty instead of just directly accessing the field on lines 69, 90 and 100?

[niheaven]

Just for nested manifests

[niheaven]

Since we are listing lots of files in function apps_in_bucket, using [System.IO.Directory]::EnumerateFiles() instead of Get-ChildItem should be much faster (ref)

Sorry but I found Get-ChildItem is faster than [System.IO.Directory]::EnumerateFiles() here.

[r15ch13]

• Since we are listing lots of files in function apps_in_bucket, using [System.IO.Directory]::EnumerateFiles() instead of Get-ChildItem should be much faster (ref)
  It's only run once for every bucket. Could be a problem if you have 1000 buckets. 😄
• Since we are deserializing a lot of JSON, using [System.Text.Json.Utf8JsonReader] should be faster than ConvertFrom-Json
  Is it available in PowerShell?

@niheaven
We need to take the filesystem cache and the drive type into account here.
The expensive part was running Get-ChildItem and Find-BucketDirectory in manifest_path() for every app.

[niheaven]

That's LGTM, and it needs an updated changelog :).

[rashil2000]

Is it available in PowerShell?

Yes

[rashil2000]

Sorry but I found Get-ChildItem is faster than [System.IO.Directory]::EnumerateFiles() here.

On multiple repeated runs (i.e. warm cache), EnumerateFiles is almost always twice as fast as Get-ChildItem in my testing. On cold cache, the difference is even more.

[niheaven]

https://learn.microsoft.com/en-us/dotnet/api/system.text.json.jsondocument.parse?view=net-7.0

Not available for .NET 4.5

[r15ch13]

Oh, I was searching for [System.Text.Json.Utf8JsonReader] explicitly... 😁

[niheaven]

On multiple repeated runs (i.e. warm cache), EnumerateFiles is almost always twice as fast as Get-ChildItem in my testing.

If we just run Get-ChildItem once for one bucket, that's not worthwhile to use EnumerateFiles. And the output of EnumerateFiles is string[] which doesn't have FullName or BaseName.

[niheaven]

Out of topic here, the minimal requirements of Scoop are WindowsPowerShell and .Net Framework 4.5 which really lacks some new features of modern PowerShell and .Net.

Could this be updated? (To newest PowerShell)

[rashil2000]

• I'm not sure I understand this, but why do we use Select-Object -ExpandProperty instead of just directly accessing the field on lines 69, 90 and 100?

Alright, what about this one?

[r15ch13]

It goes down to around 700ms from 1.3sec by using JsonDocument.Parse(). My C# implementation runs at around 300ms.

[rashil2000]

Since [System.Text.Json] is unavailable in PS 5.1, there's another way: we could reuse the NewtonSoft.Json DLL. As mentioned in this SO post, it is also much faster than ConvertFrom-Json.

[niheaven]

Too complicated for the JSON parsing function here... And [System.Text.Json] is not available for .Net Frameworks (4.5, 5 and all version before .Net Core)

The loop and generic list (this is not needed even, IMO. I've tried to replace all array '@()' to generic list but the time savings is 😭) are enough for scoop search, and if ConvertFrom-Json matters, let's create a new PR.

[r15ch13]

Added a fallback for when System.Text.Json is not available

[niheaven]

I still don't think we need to speed up ConvertFrom-Json. From tens of seconds to seconds is important, but from seconds to milliseconds is not.

If this is needed indeed, tweak (or migrate to) the parse_json() function is better imo.

[r15ch13]

Performance right after starting the PC still sucks.

[niheaven]

The two different searching time are all using this PR's solution? Wow...

[r15ch13]

Yes, just ran the same code twice after restarting the PC.

[bytebone]

Hello. Whats the status of this? From reading the messages here, it seems to me all painpoints have been agreed upon, no? As someone coming back to Windows from Arch, I find it super painful to have to wait multiple seconds for every search I do, and I'd much appreciate any improvement to the search speed, no matter how big or small. Just wanted to add my two cents - thanks for all the hard work!

[JVimes]

Really excited about this one 🙂 Is it still on the radar?

[r15ch13]

Messed up the branch. Created a new PR #5644