feat(sqlite): Use SQLite for caching apps

[niheaven]

Description

This should be present in the next version after 0.4.0 if we need more tests

Ultimate solution for scoop search by using SQLite cache of apps.

Add a use_sqlite_cache Scoop config option. The first time scoop config use_sqlite_cache true is run, it initializes and builds a cache file located in %SCOOP%\Scoop.db.

The app table in the database has the following structure:

name TEXT NOT NULL COLLATE NOCASE,
description TEXT NOT NULL,
version TEXT NOT NULL,
bucket VARCHAR NOT NULL,
manifest JSON NOT NULL,
binary TEXT,
shortcut TEXT,
dependency TEXT,
suggest TEXT,
PRIMARY KEY (name, version, bucket)

Motivation and Context

• Closes Improve Search Performance #4239
• Relates to Implemented verbose search #3983

To initialize the cache, simply run the command scoop config use_sqlite_cache true. This will cache all manifests in the bucket directory. If you encounter any errors, you can run the same command again to recache all manifests.

The cache is automatically updated when you run scoop update and it is used in scoop search. If a cached version is found in the database, scoop install xxx@xxx will use it to avoid error.

scoop search will search 'name', 'binary', and 'shortcut' by default, and could be easily expanded to 'description' or 'suggest'.

How Has This Been Tested?

Initialize cache and search:

Searching time:

Checklist:

• I have read the Contributing Guide.
• I have ensured that I am targeting the develop branch.
• I have updated the documentation accordingly.
• I have updated the tests accordingly.
• I have added an entry in the CHANGELOG.

[rasa]

Looks great, though I haven't tested it yet, but will soon!

[niheaven]

I've made some fixes, any idea @rasa?

[rashil2000]

Let's plan this for 0.5.0? The current release is long time pending...

[niheaven]

It's ready to merge and may not affect existing functions. So if someone will test it in recent days, I prefer publishing it in 0.4.0 (which is waiting for #5840)

[chawyehsu]

I don't think several days for testing such a major feature is enough so I'd vote deferring it to the next release.

[niheaven]

That's okay that it will go to 'develop' which I'm using.

[chawyehsu]

much errors reporting

scoop config use_sqlite_cache true

...
InvalidOperation: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:155
Line |
 155 |  …                $dbCommand.Parameters.AddWithValue("@$($_.Name)", $_.V …
     |                                                          ~~~~~~~
     | You cannot call a method on a null-valued expression.
InvalidOperation: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:157
Line |
 157 |              $dbCommand.ExecuteNonQuery() | Out-Null
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | You cannot call a method on a null-valued expression.
InvalidOperation: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:167
Line |
 167 |              $db.Dispose()
     |              ~~~~~~~~~~~~~
     | You cannot call a method on a null-valued expression.
InvalidOperation: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:164
Line |
 164 |              $dbTrans.Rollback()
     |              ~~~~~~~~~~~~~~~~~~~
     | You cannot call a method on a null-valued expression.
'use_sqlite_cache' has been set to 'true'

scoop search

MethodInvocationException: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:99
Line |
  99 |      $db.Open()
     |      ~~~~~~~~~~
     | Exception calling "Open" with "0" argument(s): "Unable to load DLL 'SQLite.Interop.dll' or one of its dependencies:
     | The specified module could not be found. (0x8007007E)"
MethodInvocationException: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:113
Line |
 113 |      $tableCommand.ExecuteNonQuery() | Out-Null
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "ExecuteNonQuery" with "0" argument(s): "Database is not open"
MethodInvocationException: C:\Users\hanabi\scoop\apps\scoop\current\lib\database.ps1:292
Line |
 292 |          [void]$dbAdapter.Fill($result)
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "Fill" with "1" argument(s): "Unable to load DLL 'SQLite.Interop.dll' or one of its dependencies:
     | The specified module could not be found. (0x8007007E)"
Results from other known buckets...

[niheaven]

It seems like SQLite .NET library is not installed correctly. Do you have supportings/sqlite folder now?

[chawyehsu]

Yeah, but I noticed there is not arm64 assembly, I'm on Windows ARM btw.

sqlite
├── x64
│   └── SQLite.Interop.dll
├── x86
│   └── SQLite.Interop.dll
└── System.Data.SQLite.dll

[niheaven]

Hmm, SQLite doesn't provide a native ARM build... I'll build it by myself or add a notice that it doesn't work on ARM machines.