[python-package] provide access to constructed Dataset in numpy format

[ivannp]

There are no public APIs to directly interact with the training dataset created from users data (the result of Dataset.construct).

Provide a method to export the training dataset as a numpy array.

[jameslamb]

There are no public APIs to directly interact with the training dataset created from users data.

Provide a method to export the training dataset as a numpy array.

If you are talking about the Python package specifically (I'm assuming that, since you mentioned numpy), this statement is not true. https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Dataset.html describes many methods to interact with the constructed Dataset.

I think you mean that you'd like something like this to work:

import lightgbm as lgb
from sklearn.datasets import make_regression

dtrain = lgb.Dataset(
    data=X,
    label=y,
    free_raw_data=True
).construct()

lgb_mat = dtrain.get_dataset_data()

Where a method like get_dataset_data() returns, in numpy format, the actual training data used by LightGBM (e.g. with continuous features replaced by their bin IDs, composite features created by Exclusive Feature Bundling, etc.).

Is that what you're asking for? If not, could you please provide some more specifics around what you're looking for that LightGBM doesn't currently provide?

[ivannp]

hi James, dtrain.get_dataset_data() from your example is exactly what I was asking for.

[StrikerRUS]

@ivannp You can use get_data() method for this purpose.

import lightgbm as lgb
from sklearn.datasets import make_regression

X, y = make_regression()

dtrain = lgb.Dataset(
    data=X,
    label=y,
    free_raw_data=False
).construct()

lgb_mat = dtrain.get_data()

[StrikerRUS]

Ah, nevermind, ignore my previous comment. You need

the actual training data used by LightGBM (e.g. with continuous features replaced by their bin IDs, composite features created by Exclusive Feature Bundling, etc.)

[jameslamb]

is exactly what I was asking for

Ok thanks! We can take this as a feature request then, I think it could be really useful for understanding more about the decisions LightGBM is making.

Per this project's policy, I'm going to close this issue and add it to the list of feature requests at #2302. Anyone wanting to contribute this can leave a comment here to ask for help.

To be honest though, the level of maintainer activity in this repo has been fairly low recently and we have some other large initiatives we're trying to get in to create a v4.0.0 release (#5153), so unless you or someone else from the community wants to work on this, I don't think this will be implemented soon.

[StrikerRUS]

I'm not exactly sure, but looks like internal Dataset method _dump_text() can be almost the thing that is requested in this issue.

import lightgbm as lgb
from sklearn.datasets import make_regression

X, y = make_regression()

dtrain = lgb.Dataset(
    data=X,
    label=y,
    free_raw_data=True
).construct()

dtrain._dump_text('dataset.txt')

https://github.com/microsoft/LightGBM/blob/924d337de36352ea9f8442054df23db0624a1eb6/python-package/lightgbm/basic.py#L2526-L2529

Precisely, this part of the underlying cpp function:
https://github.com/microsoft/LightGBM/blob/417c732cc05978276d533c43070d24dfd6c892b7/src/io/dataset.cpp#L1120-L1138

[vladv14]

It would also be useful to return bin upper bounds of the constructed dataset, in a similar format to how you can provide them using forced_bins Dataset parameter.
https://github.com/microsoft/LightGBM/blob/master/examples/regression/forced_bins.json

So you can maybe have a method like get_dataset_data() in Dataset, which will return a numpy array with continuous features replaced by their bin IDs, as mentioned above, plus the bin upper bounds metadata. Or perhaps those could be split out into two separate methods like get_data_bin_ids(), returning a numpy array, and get_data_bin_upper_bounds(), returning a dictionary in the format of forced_bins.json

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.