New configuration file format (using libconfig)

[lminiero]

This pull request is a new effort related to the discussion in this post on the Google group. While I started working on YAML as a configuration file format and actually got quite far (some of the work I brought on here, as a matter of fact), I eventually ditched that effort, as I found the format too limiting and confusing for our needs, and I personally found the libyaml library too much of a pain. Some more research later, I found libconfig, a more reasonable format (more in line with what I needed) and a library API much easier to use.

The new configuration format looks like this:

# This is a comment
// This is also a comment
/* I can also add comments
 like this */
category: {
    name1 = "value1"
    name2 = "value2"
    subcategory = {
        // Subcategory stuff here
        cool = true
    }
    array1 = (
        // An array can contain whatever we want, even objects
        "apples",
        "oranges",
        {
            stuff = "otherfruit"
        }
    )  
}

# Another category
category2: {
    name = "value"
}

This pull request adds support for a new type of configuration files that use the libconfig format, which in Janus have the .jcfg extension, to avoid confusion with the existing INI .cfg files. As with the previous YAML effort, INI files are still supported, but they're not in the repo anymore, and have been replaced with the counterpart: core and plugins all try and load the .jcfg version by default, and fallback to .cfg if they can't find it. This means you can keep on using your existing configuration files, although you're encouraged to convert them sooner or later.

To make the conversion easier, there's a new tool called janus-cfgconv. The syntax is very simple:

janus-cfgconv source destination

where source points to an existing configuration file, and destination to a new file. You can use this tool to convert cfg to jcfg and viceversa: of course, in case the jcfg uses constructs the INI format doesn't support (e.g., arrays or lists), then that information will be lost in the process.

It's also worth noting that AudioBridge, VideoRoom and TextRoom now don't use plain room IDs as the name of the category anymore, but they need to be prefixed by room- (e.g., 1234 would become room-1234): this is needed because in libconfig setting names can only start with a letter or an asterisk. This means that, when converting, the tool will always add those prefixes for you, in case the setting name is a number. That said, obviously existing INI files will work as before, so whether you add the room- prefix or not it will still work: this means that a [1234] room category is equivalent to [room-1234], although the latter is preferred since it will be what we'll use internally from now on.

Planning to merge soon, so feedback welcome.

C API changes

This section is only relevant if you're a developer working on Janus internally, e.g., on your own plugin. Feel free to ignore all this if you're not doing that.

As I was saying, this effort incorporates some of the changes I made to support YAML in #1116. Specifically, all the refactoring of the janus_config C API is exactly the same as the one I made there, which means that if you were using our config files in your custom plugin code, you'll have to make some changes to use it now, even if you decide to stick to INI. It's not a lot of changes, but the pattern is different, and the API is now much clearer, more explicit, and with less implicit assumptions. In fact, the old config API did a lot of assumptions, and masqued several things behind the curtains. This made it quite constrained, and proved unusable for these new concepts, like subobjects or arrays. The new API is much cleaner, and is inspired from the Jansson API. For instance, to add something to a config:

1. you create a node object (item/category/array)
2. you add that node to the root or to a parent (which is another node)

Lookup is cleaner as well, and so is the removal of nodes from the object.

To understand which changes to make in your code to do all that, it's probably much easier to just look at the diffs for the plugins. Anyway, I'll try to summarize the main changes here:

• Items, categories and arrays are really all the same thing, internally, but they're identified by a new type enumerator. Typically, an item will have name and, optionally, value; objects and arrays will have a name and a list of subobjects. Each of these subobjects can be another item, category or array, which means you're free to indent and structure your config files the way you like. Since it's all GList based, this means you're also free to just crawl the config object manually yourself, if you like, starting from the root janus_config object which has a list of elements.

• A common way of getting config values was janus_config_get_item_drilldown, which is no more. I mean, you can use janus_config_search if you want, that has variable arguments that act as a "path" to the element you want to retrieve (e.g., parent->parent->item), but now it's usually best and more efficient to first pick the category you're interested in (as a cache), and then get the items that belongs to it, especially when you're getting many items from the same category (e.g., "general") in sequence. For both operations you can use janus_config_get, although for categories you might want to use janus_config_get_create instead, as that will create the category for you if it doesn't exist (possibly better, especially if you plan to then save the configuration file). Getting an element requires the type you're expecting, and the parent to get it from: if NULL you're getting from the root. This means that you'd change your code like this:

Old:
    janus_config_item *item = janus_config_get_item_drilldown(config, "general", "enabled");
    item = janus_config_get_item_drilldown(config, "general", "json");

New:
    janus_config_category *config_general = janus_config_get_create(config, NULL, janus_config_type_category, "general");
    janus_config_item *item = janus_config_get(config, config_general, janus_config_type_item, "enabled");
    item = janus_config_get(config, config_general, janus_config_type_item, "json");

• As anticipated, adding elements works in a different way too. You before you had methods like janus_config_add_category and janus_config_add_item, now you first create the element you need (janus_config_item_create for items, janus_config_category_create for categories and janus_config_array_create for arrays), and then you add it to the config with janus_config_add. As janus_config_get, this method expects a parent element to add this to, e.g., if you're adding to a category rather than to the root itself. Of course, for categories and arrays you can also just use janus_config_get_create as shown before, which returns the existing category/array if it exists and creates it for you if it doesn't. This means that you'd change your code like this:

Old:
    janus_config_add_category(config, "mycategory");
    janus_config_add_item(config, "mycategory", "name1", "value1");
    janus_config_add_item(config, "mycategory", "name2", "value2");

New:
    janus_config_category *c = janus_config_get_create(config, NULL, janus_config_type_category, cat);
    janus_config_add(config, c, janus_config_item_create("name1", "value1"));
    janus_config_add(config, c, janus_config_item_create("name2", "value2"));

• Removing an element is also a bit different. You now use janus_config_remove, which expects the name of the element to remove. You also specify the parent to remove it from: again, if NULL, this removes it from the root. If removing a category/array, all the elements contained in it are destroyed too. A simple example on how you now remove a category from the root:

Old:
    janus_config_remove_category(config, "general");

New:
    janus_config_remove(config, NULL, "general");

Summary

So, as you can see, very few changes needed probably:

1. change multiple janus_config_get_item_drilldown calls to janus_config_get_create for category plus multiple janus_config_get calls (enough for when you're only reading config files);
2. change janus_config_add_category plus multiple janus_config_add_item calls, to janus_config_get_create for category and multiple janus_config_item_create+janus_config_add for items (only needed when writing to a config file);
3. change janus_config_remove_category to janus_config_remove (with updated signature; again, only needed when writing to a config file)).

[tgabi333]

I am confused by the usage of ":" and "=", is there a difference?

[lminiero]

I am confused by the usage of ":" and "=", is there a difference?

They're equivalent, I just used the convention the library follows when writing to file itself, which is to use the ":" for objects, and the "=" for scalars/arrays/lists.

[RSATom]

@lminiero, Do you still plan merge this code? Should I use new format for my plugin configs or better use old format?

[lminiero]

@RSATom definitely planning to merge soon: I was waiting for feedback before doing that, though, as I haven't received any so far. Since you're working on a plugin, your feedback would indeed help make that process faster: in case, please let me know if by playing with this you encounter anything that needs fixing.

[lminiero]

I'll align to master later today, as I see there are a couple of conflicts.

[RSATom]

Thanks, I'll try to use it and write feedback if meet any issue.

[RSATom]

@lminiero, does it miss lists support, or it's intended?

[lminiero]

Lists are there, they're called janus_config_array in our code.

[RSATom]

ah... ok, thank you.

[lminiero]

To be precise, we support both arrays and lists when parsing, but when saving to file ourselves we always use list as we don't know if all the items will be of the same time (especially if it's objects).

[RSATom]

good point

[RSATom]

I've implemented some basic config support in my plugin. It looks like new implementation is well enough for my case. But tbh, my config is very simple atm. Just list of sources...

[lminiero]

Merging, as we'll need this for the upcoming Unified Plan work.