[R-package] Support installing from GitHub repo via `remotes::install_github()` or `pak::pak()`

[eitsupi]

Summary

It would be useful to be able to install R packages directly from the GitHub repository with a function, allowing for releases on the R-universe.

Motivation

I am currently around R-multiverse, a new R package repository, and am wondering if this R package might be worth releasing on R-multiverse.

• Very popular.
• It is a mono repository and releases are made even if CRAN does not accept them.
• There has been a problematic period of renewal on CRAN.

However, the R-universe system behind R-multiverse uses pak to build packages, so registering this current repository with R-universe will not release the R package.

Description

The lightgbm R package cannot currently be installed directly with functions that support installation from Git remote repositories such as remotes::install_github("microsoft/LightGBM/R-package") and pak::pak("microsoft/LightGBM/R-package").
In order to build the R package correctly, the special script must be run to update the package prior to the R CMD build.

Details

> remotes::install_github("microsoft/LightGBM/R-package")
Downloading GitHub repo microsoft/LightGBM@HEAD
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/boostorg/compute /tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/compute
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/compute'...
remote: Enumerating objects: 704, done.
remote: Counting objects: 100% (704/704), done.
remote: Compressing objects: 100% (589/589), done.
remote: Total 704 (delta 305), reused 165 (delta 107), pack-reused 0 (from 0)
Receiving objects: 100% (704/704), 599.54 KiB | 2.78 MiB/s, done.
Resolving deltas: 100% (305/305), done.
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://gitlab.com/libeigen/eigen.git /tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/eigen
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/eigen'...
remote: Enumerating objects: 2023, done.
remote: Counting objects: 100% (2023/2023), done.
remote: Compressing objects: 100% (1731/1731), done.
remote: Total 2023 (delta 381), reused 1041 (delta 248), pack-reused 0 (from 0)
Receiving objects: 100% (2023/2023), 3.27 MiB | 6.40 MiB/s, done.
Resolving deltas: 100% (381/381), done.
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/fmtlib/fmt.git /tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/fmt
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/fmt'...
remote: Enumerating objects: 157, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 157 (delta 4), reused 62 (delta 3), pack-reused 0 (from 0)
Receiving objects: 100% (157/157), 720.63 KiB | 4.45 MiB/s, done.
Resolving deltas: 100% (4/4), done.
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/lemire/fast_double_parser.git /tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/fast_double_parser
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/../external_libs/fast_double_parser'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 44 (delta 3), reused 26 (delta 2), pack-reused 0 (from 0)
Receiving objects: 100% (44/44), 624.83 KiB | 3.05 MiB/s, done.
Resolving deltas: 100% (3/3), done.
Submodule 'benchmark/dependencies/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'benchmarks/dependencies/abseil-cpp'
Submodule 'benchmark/dependencies/double-conversion' (https://github.com/google/double-conversion.git) registered for path 'benchmarks/dependencies/double-conversion'
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'...
remote: Enumerating objects: 27951, done.        
remote: Counting objects: 100% (3511/3511), done.        
remote: Compressing objects: 100% (508/508), done.        
remote: Total 27951 (delta 3098), reused 3004 (delta 3003), pack-reused 24440 (from 2)        
Receiving objects: 100% (27951/27951), 15.46 MiB | 10.17 MiB/s, done.
Resolving deltas: 100% (21946/21946), done.
Cloning into '/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'...
remote: Enumerating objects: 1356, done.        
remote: Counting objects: 100% (440/440), done.        
remote: Compressing objects: 100% (117/117), done.        
remote: Total 1356 (delta 374), reused 327 (delta 323), pack-reused 916 (from 3)        
Receiving objects: 100% (1356/1356), 7.08 MiB | 11.59 MiB/s, done.
Resolving deltas: 100% (881/881), done.
Installing 1 packages: data.table
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/data.table_1.16.4.tar.gz'
Content type 'binary/octet-stream' length 2453361 bytes (2.3 MB)
==================================================
downloaded 2.3 MB

* installing *binary* package ‘data.table’ ...
* DONE (data.table)

The downloaded source packages are in
        ‘/tmp/RtmpTegeTQ/downloaded_packages’
── R CMD build ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/tmp/RtmpTegeTQ/remotes4887e581cd1/microsoft-LightGBM-425395d/R-package/DESCRIPTION’ (385ms)
─  preparing ‘lightgbm’:
E  checking DESCRIPTION meta-information ...
   Malformed package version.
   
   See section 'The DESCRIPTION file' in the 'Writing R Extensions'
   manual.
Error: Failed to install 'lightgbm' from GitHub:
  ! System command 'R' failed

> pak::pak("microsoft/LightGBM/R-package")
! Using bundled GitHub PAT. Please add your own PAT using `gitcreds::gitcreds_set()`.
✔ Updated metadata database: 3.92 MB in 7 files.                          
✔ Updating metadata database ... done                                     
Error:                                                                     
! error in pak subprocess
Caused by error: 
! invalid version specification ‘~~VERSION~~’
Type .Last.error to see the more details.

My quick testing has shown that it may be possible to support these with a few modifications to the script and the DESCRIPTION file.
https://github.com/eitsupi/rbootstraptest

• Change the version of the package before replacement to a valid version (like 0.0.9000).
• Set Config/build/bootstrap: TRUE (See the references for details)
• Add a bootstrap.R script to build the package before R CMD build (See the references for details)

References

• Run bootstrap.R file on build if present and specified in Config/build r-lib/pkgbuild#157
• https://github.com/apache/arrow-nanoarrow: A package that supports remote installation using bootstrap.R

[jameslamb]

Thanks for your interest in LightGBM.

I'm not excited about including a script like this in the repo just for a single re-packager (for reasons similar to pyodide/pyodide#5279 (comment)).

I am also nervous about taking on non-standard stuff in DESCRIPTION like this:

Config/testthat/edition: 3
Config/build/bootstrap: TRUE

(apache/nanoarrow/r/DESCRIPTION)

By "non-standard", I mean that I do not see this "Config/" structure mentioned in https://cran.r-project.org/doc/manuals/r-devel/R-exts.html. From the {pak} PR you shared, I guess that Config/build/bootstrap, for example, is a {pak}-specific convention. And by "nervous", I mean that I worry it would be a continuing source of new maintenance burden to keep up with changes in {pak}, and possibly risk CRAN submission issues for unrecognized fields in DESCRIPTION.

The requirement that packages have to be buildable by a specific third-party library and have code specific to that library in their own source control is pretty invasive. Maybe it would be better for this R-Multiverse project to instead structure packages the way that most other repackagers do... providing a "recipe" describing how to retrieve source code and build artifacts from it.

Here are some examples of LightGBM re-packagings in these types of systems:

• conda-forge: https://github.com/conda-forge/lightgbm-feedstock/blob/bca9f5edeb5f98a93af27579102a5a46b914eeed/recipe/meta.yaml
• Homebrew: https://github.com/Homebrew/homebrew-core/blob/726a413d23a94ef9576573c10af9b736834eeda5/Formula/l/lightgbm.rb#L1
• macports: https://github.com/macports/macports-ports/blob/045ad676b75d8b4d9c3b685aa79f841962ac384d/math/LightGBM/Portfile
• nix: https://github.com/NixOS/nixpkgs/blob/03ae77ee2d193531819ae43711c8f168c7051e7b/pkgs/development/python-modules/lightgbm/default.nix
• pyodide: https://github.com/pyodide/pyodide/blob/eebce17dee0205d9f24f3758e417669e3ed41696/packages/lightgbm/meta.yaml

[jameslamb]

I'm willing to consider changes to make remotes::install_github() workable for the R package, I do understand it'd be helpful. Changes described in #6774 would be a good start.

But I am not convinced by the information that you've shared that we should take on this {pak}-specific configuration and build script(s).