[Feature request] Add configuration source reading from local data structure

[NicolasMelot]

I understand that the only way to define configuration values used when none of the files, environment variables or command line sources, is to use the memory source. The problem with it is hard to define complex data structures using the list of (key, value) string pairs.
I would be great if the function add_in_memory could take an existing data structure and its values, that can be used as default value until any subsequent source overrides them. For example:

use serde::Deserialize;

#[derive(Default, Deserialize)]
#[serde(rename_all(deserialize = "PascalCase"))]
struct Client {
    region: String,
    url: String,
}

#[derive(Default, Deserialize)]
#[serde(rename_all(deserialize = "PascalCase"))]
struct AppOptions {
    text: String,
    demo: bool,
    clients: Vec<Client>,
}

use config::{*, ext::*};

fn main() {
    let default = AppOptions {
        text: String::from("Default text"),
        demo: false,
        clients: Vec::new()
    };

    let config = DefaultConfigurationBuilder::new()
        .add_in_memory(default)
        .add_json_file("base.json")
        .build()
        .unwrap();
    let app: AppOptions = config.reify();
    
    if app.demo
    {
        println!("{}", &app.text);
        println!("{}", &app.clients[0].region);
        return;
    }   
    
    println!("Not a demo!");
}

[commonsensesoftware]

Interesting idea. It's worth exploring, but it is likely to be fairly difficult and would involve some rather complex macros to do. The biggest challenge is reversing the data structure into the ubiquitous configuration key/value format. It's not impossible, but it's definitely not trivial. Challenge accepted!

In the meantime, your goal is easily achievable with the features that serde already provides. The following is a minor refactoring of your example where the default values come from the struct itself before configuration.

#[derive(Default, Deserialize)]
#[serde(rename_all(deserialize = "PascalCase"))]
struct AppOptions {
    #[serde(default = "default_app_text")]
    text: String,

    #[serde(default)]
    demo: bool,

    #[serde(default)]
    clients: Vec<Client>,
}

fn default_app_text() -> String {
    "Default text".into()
}

fn main() {
    let config = DefaultConfigurationBuilder::new()
        .add_json_file("base.json".is().optional()) // you can make the JSON file optional if you want to
        .build()
        .unwrap();
    let app: AppOptions = config.reify();
    
    println!("{}", &app.text);
    
    if app.demo {
        println!("{}", &app.clients[0].region);
    } else {
        println!("Not a demo!");
    }
}

If no configuration is provided or the key Text doesn't exist in the configuration then AppOptions::text will be "Default text". This works because deserialization all happens through serde. serde tries to find a matching configuration key, which it doesn't. It then falls back to using its own default value mechanism, in this case a callback, to calculate default values.

[NicolasMelot]

I almost got a working solution using serde_merge I just create an instance of the default configuration data structure, and I merge the one from configuration to the default one after having read the multilayer configuration sources. The whole thing looks like this:

use serde::{Serialize, Deserialize};
use std::env;

#[derive(Default, Deserialize, Serialize, Clone)]
#[serde(rename_all(deserialize = "PascalCase"))]
#[serde(rename_all(serialize = "PascalCase"))]
struct Client {
    region: String,
    url: String,
}

#[derive(Default, Deserialize, Serialize, Clone)]
#[serde(rename_all(deserialize = "PascalCase"))]
#[serde(rename_all(serialize = "PascalCase"))]
struct AppOptions {
    text: String,
    demo: bool,
    clients: Vec<Client>,
}

use config::{*, ext::*};
use std::path::PathBuf;
use std::vec::Vec;
use std::string::String;

use serde_merge::tmerge;

use serde_merge::tmerge;

fn main() {
    let config = DefaultConfigurationBuilder::new()
        .add_json_file("base.json")
        .build()
        .unwrap();
    let options: AppOptions = config.reify();

    let default = AppOptions {
        text: String::from("Default text"),
        demo: false,
        clients: Vec::new()
    };

    let app: AppOptions = tmerge(default.clone(), options).unwrap_or(default);
    
    if app.demo
    {   
        println!("{}", &app.text);
        println!("{}", &app.clients[0].region);
        return;
    }   
    
    println!("Not a demo!");
}

Only trouble is, if base.json misses any value, e.g., text, then the app crashes again with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: MissingValue("Text")

EDIT: formatting issues and removing irrelevant code lines
EDIT (2): Initial code in post did not work as described. Also removed EDIT (2) details about bug in code.

[commonsensesoftware]

Hmmm... I'm not familiar with serde_merge. I get what tmerge is supposed to do, but I'd have to dig into the source to give you a better explanation.

Attached is an E2E working example based on what you've shared. Is there a reason I'm not privy to as to why the built-in default value mechanism serde provides won't work? It looks like it will.

Issue-9 : E2E Solution

Another option would be to implement serde::Deserialize::deserialize_in_place. This would allow updating the struct in-place rather than replace it and potentially ignore/skip attributes that do not have corresponding configuration values. Using serde_merge or ConfigurationBinder::bind require or otherwise result in two struct instances. You can see that ConfigurationBinder::bind calls through to config::de::bind_config, which in turn calls Deserialize::deserialize_in_place.

more-rs-config/src/binder.rs

Line 45 in f740f2a

fn bind<T: DeserializeOwned>(&self, instance: &mut T) {

more-rs-config/src/de.rs

Line 295 in f740f2a

pub fn bind_config<'a, T>(configuration: &'a dyn Configuration, data: &mut T) -> Result<(), Error>

The default implementation of deserialize_in_place, however, just replaces the entire struct. You can further confirm that it is not generated by the serde macros with cargo expand (if you have it installed).

Honestly, I'm not sure if there is a way to generate the necessary deserialization code on a user's behalf for in-place updates, but it's certainly a feature I'd been interesting in adding. I think what would probably have to happen is that more-config would have to generate the appropriate function and then you'd have to wire it up in Deserialize::deserialize_in_place. I'm not even sure how that would play with the default serde derive implementation for Deserialize. I'm happy to track that as a feature request, but it will probably be some time coming under the assumption it's even possible (though I have a feeling it may be).

[NicolasMelot]

Yeah, thought about looking for merging a default config right after I posted the feature requests. It seemed ideal and it almost is except for the case where the config source merged together still miss some value. Could be because user is fine with default value and does not want config files clobbered with settings is config file user does not care about.

The workaround suggested works. Another solution would be to always have a full config file (perhaps next to binary) and loaded first. However what is nice with more-config is that it allows multi-layer configuration with little boiler plate. These workarounds break this nice property somewhat.

I did not look at more-config code, but my guess is somewhere in the process of any source, it deals with some data structure that serde can use as base to generate the deserialized data structure. From this perspective, I imagine it should be possible to slip in a data structure that can be serialized into this intermediate representation and that more-config can take to immediately deserialize, and that would do the trick. But there might be lots of details that might get in the way of my nice little ideal scheme, or it might as well work in a very different manner. If that is a sound approach to you, I might give it a try sometime although as you said it might take some time.

Anyway, thank you for the nice crate, your interest in the feature and short answer time!

[commonsensesoftware]

The configuration layering still works, if you add it. The last set value wins. For example, adding the cmd feature back in and applying .add_command_line(), then you can set Text straight from the console:

cargo run -- --Text "From Console"

To recap, for binding, there are currently two out-of-the-box approaches:

1. Use the built-in default value mechanisms in serde
2. Use the mem feature and populate the default values in-memory (which you said you didn't want to do)

Everything you're looking for related to deserialization within the crate is in de.rs. The first part of improving binding is making sure that it is still possible to implement deserialize_in_place if the serde #[derive(Deserialize)] macro is applied. If it is, then it should possible to create a function that can be dropped into. I'm not sure at this point if that will require additional macros to generate the necessary code - possibly. Using cargo expand (you have to install the tool) will show you what #[derive(Deserialize)] generates and will help understand what's happening. I created issue #10 to track it.