Cannot load library if new fields are added in sabi_trait trait

[SilverMira]

Hello, I'm currently trying out this crate to implement a plugin system.

Although the docs seem to suggest that I can load a library that's compiled against a previous minor version of the shared interface, I'm getting a AbiInstability error when trying to load.

I'm not sure whether I'm doing anything wrong which made it not work.

// interface: 0.1.0
#[repr(u8)]
#[derive(StableAbi, Debug)]
#[sabi(kind(WithNonExhaustive(size = [usize; 8], traits(Debug))))]
#[non_exhaustive]
pub enum PluginError {
    NotSupported,
}

#[sabi_trait]
pub trait Plugin {
    #[sabi(last_prefix_field)]
    fn hello(&self, value: RStr<'_>) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
}

pub type PluginBox = Plugin_TO<'static, RBox<()>>;


#[repr(C)]
#[derive(StableAbi)]
#[sabi(kind(Prefix(prefix_ref = PluginLibRef)))]
#[sabi(missing_field(panic))]
pub struct PluginLib {
    #[sabi(last_prefix_field)]
    pub new_plugin: extern "C" fn() -> PluginBox,
}

impl RootModule for PluginLibRef {
    abi_stable::declare_root_module_statics! {PluginLibRef}

    const BASE_NAME: &'static str = "plugin-interface";

    const NAME: &'static str = "plugin-interface";

    const VERSION_STRINGS: VersionStrings = package_version_strings!();
}

With the above interface crate, I'm able to compile and run hello() without issues. However if I update the Plugin trait as such and try to rebuild & run the main executable without recompiling the library, I'm getting a AbiInstability error "Too many fields" while loading the library.

// interface: 0.1.1 
#[sabi_trait]
pub trait Plugin {
    #[sabi(last_prefix_field)]
    fn hello(&self, value: RStr<'_>) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
    fn not_implemented_yet(&self) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
}

Logging the AbiInstability error

AbiInstability(Compared <this>:
    --- Type Layout ---
    type:PrefixRef<'a, PluginLib>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    line:392 mod:abi_stable::prefix_type::prefix_ref
    data:
        Struct with Fields:

    Phantom fields:

        field_name:0
        type:PluginLib
        size:8 align:8
        package:'interface' version:'0.1.1'Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:DelegateDeref { layout_index: 0 }
To <other>:
    --- Type Layout ---
    type:PrefixRef<'a, PluginLib>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    line:392 mod:abi_stable::prefix_type::prefix_ref
    data:
        Struct with Fields:

    Phantom fields:

        field_name:0
        type:PluginLib
        size:8 align:8
        package:'interface' version:'0.1.1'Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:DelegateDeref { layout_index: 0 }

0 error(s).

0 error(s)inside:
    <other>

    field_name:0
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'

Layout of expected type:
    --- Type Layout ---
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'
    line:12 mod:interface
    data:
        Prefix type:
        first_suffix_field:1
        conditional_prefix_fields:
            0
        fields:
            field_name:new_plugin
            type:AFunctionPointer
            size:8 align:8
            package:'abi_stable' version:'0.11.3'
            fn pointer(s):
                fn()->Plugin_TO<'lt>

        accessible_fields:
            [Yes]
    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:C
    Module reflection mode:Module

Layout of found type:
    --- Type Layout ---
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'
    line:12 mod:interface
    data:
        Prefix type:
        first_suffix_field:1
        conditional_prefix_fields:
            0
        fields:
            field_name:new_plugin
            type:AFunctionPointer
            size:8 align:8
            package:'abi_stable' version:'0.11.3'
            fn pointer(s):
                fn()->Plugin_TO<'lt>

        accessible_fields:
            [Yes]
    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:C
    Module reflection mode:Module


0 error(s)inside:
    <other>

    field_name:0
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'

    field_name:new_plugin
    type:AFunctionPointer
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    fn pointer(s):
        fn()->Plugin_TO<'lt>

    fn()->Plugin_TO<'lt>

    field_name:__returns
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'

Layout of expected type:
    --- Type Layout ---
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'
    line:38 mod:interface::Plugin_trait
    data:
        Struct with Fields:
            field_name:obj
            type:RObject<'lt>
            size:24 align:8
            package:'abi_stable' version:'0.11.3'
            lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

            field_name:_marker
            type:UnsafeIgnoredType
            size:0 align:1
            package:'abi_stable' version:'0.11.3'

    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:Module

Layout of found type:
    --- Type Layout ---
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'
    line:38 mod:interface::Plugin_trait
    data:
        Struct with Fields:
            field_name:obj
            type:RObject<'lt>
            size:24 align:8
            package:'abi_stable' version:'0.11.3'
            lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

            field_name:_marker
            type:UnsafeIgnoredType
            size:0 align:1
            package:'abi_stable' version:'0.11.3'

    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:Module


0 error(s)inside:
    <other>

    field_name:0
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'

    field_name:new_plugin
    type:AFunctionPointer
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    fn pointer(s):
        fn()->Plugin_TO<'lt>

    fn()->Plugin_TO<'lt>

    field_name:__returns
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'

    field_name:obj
    type:RObject<'lt>
    size:24 align:8
    package:'abi_stable' version:'0.11.3'
    lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

Layout of expected type:
    --- Type Layout ---
    type:RObject<'lt>
    size:24 align:8
    package:'abi_stable' version:'0.11.3'
    line:77 mod:abi_stable::sabi_trait::robject
    data:
        Struct with Fields:
            field_name:vtable
            type:PrefixRef<'a, VTable>
            size:8 align:8
            package:'abi_stable' version:'0.11.3'

            field_name:ptr
            type:ManuallyDrop<RBox>
            size:16 align:8
            package:'std' version:'1.0.0'

            field_name:_marker
            type:PhantomData<&'a  () , AFunctionPointer>
            size:0 align:1
            package:'std' version:'1.0.0'
            fn pointer(s):
                fn()->Plugin_Interface
            lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

    Tag:
        null
    Extra checks:
        RequiredTraits
        Auto traits:<no_traits>
        Impld traits:<no_traits>
    Repr attribute:C
    Module reflection mode:Opaque

Layout of found type:
    --- Type Layout ---
    type:RObject<'lt>
    size:24 align:8
    package:'abi_stable' version:'0.11.3'
    line:77 mod:abi_stable::sabi_trait::robject
    data:
        Struct with Fields:
            field_name:vtable
            type:PrefixRef<'a, VTable>
            size:8 align:8
            package:'abi_stable' version:'0.11.3'

            field_name:ptr
            type:ManuallyDrop<RBox>
            size:16 align:8
            package:'std' version:'1.0.0'

            field_name:_marker
            type:PhantomData<&'a  () , AFunctionPointer>
            size:0 align:1
            package:'std' version:'1.0.0'
            fn pointer(s):
                fn()->Plugin_Interface
            lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

    Tag:
        null
    Extra checks:
        RequiredTraits
        Auto traits:<no_traits>
        Impld traits:<no_traits>
    Repr attribute:C
    Module reflection mode:Opaque


0 error(s)inside:
    <other>

    field_name:0
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'

    field_name:new_plugin
    type:AFunctionPointer
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    fn pointer(s):
        fn()->Plugin_TO<'lt>

    fn()->Plugin_TO<'lt>

    field_name:__returns
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'

    field_name:obj
    type:RObject<'lt>
    size:24 align:8
    package:'abi_stable' version:'0.11.3'
    lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

    field_name:vtable
    type:PrefixRef<'a, VTable>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'

Layout of expected type:
    --- Type Layout ---
    type:PrefixRef<'a, VTable>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    line:392 mod:abi_stable::prefix_type::prefix_ref
    data:
        Struct with Fields:

    Phantom fields:

        field_name:0
        type:VTable
        size:16 align:8
        package:'interface' version:'0.1.1'Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:DelegateDeref { layout_index: 0 }

Layout of found type:
    --- Type Layout ---
    type:PrefixRef<'a, VTable>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    line:392 mod:abi_stable::prefix_type::prefix_ref
    data:
        Struct with Fields:

    Phantom fields:

        field_name:0
        type:VTable
        size:16 align:8
        package:'interface' version:'0.1.1'Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:Transparent
    Module reflection mode:DelegateDeref { layout_index: 0 }


1 error(s)inside:
    <other>

    field_name:0
    type:PluginLib
    size:8 align:8
    package:'interface' version:'0.1.1'

    field_name:new_plugin
    type:AFunctionPointer
    size:8 align:8
    package:'abi_stable' version:'0.11.3'
    fn pointer(s):
        fn()->Plugin_TO<'lt>

    fn()->Plugin_TO<'lt>

    field_name:__returns
    type:Plugin_TO<'lt>
    size:24 align:8
    package:'interface' version:'0.1.1'

    field_name:obj
    type:RObject<'lt>
    size:24 align:8
    package:'abi_stable' version:'0.11.3'
    lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] })

    field_name:vtable
    type:PrefixRef<'a, VTable>
    size:8 align:8
    package:'abi_stable' version:'0.11.3'

    field_name:0
    type:VTable
    size:16 align:8
    package:'interface' version:'0.1.1'

Layout of expected type:
    --- Type Layout ---
    type:VTable
    size:16 align:8
    package:'interface' version:'0.1.1'
    line:38 mod:interface::Plugin_trait
    data:
        Prefix type:
        first_suffix_field:3
        conditional_prefix_fields:
            0
        fields:
            field_name:_sabi_tys
            type:PhantomData<(), RBox>
            size:0 align:1
            package:'std' version:'1.0.0'

            field_name:_sabi_vtable
            type:PrefixRef<'a, RObjectVtable>
            size:8 align:8
            package:'abi_stable' version:'0.11.3'

            field_name:hello
            type:AFunctionPointer
            size:8 align:8
            package:'abi_stable' version:'0.11.3'
            fn pointer(s):
                unsafe fn(_self,param_0,: RRef<'a>, param_1: RStr<'a>)->RResult
                lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, ANONYMOUS], [NONE, NONE], [NONE, NONE]] })

            field_name:not_implemented_yet
            type:AFunctionPointer
            size:8 align:8
            package:'abi_stable' version:'0.11.3'
            fn pointer(s):
                unsafe fn(_self,: RRef<'a>)->RResult
                lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, NONE], [NONE, NONE], [NONE, NONE]] })

        accessible_fields:
            [Yes, Yes, Yes, Yes]
    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:C
    Module reflection mode:Module

Layout of found type:
    --- Type Layout ---
    type:VTable
    size:16 align:8
    package:'interface' version:'0.1.1'
    line:38 mod:interface::Plugin_trait
    data:
        Prefix type:
        first_suffix_field:3
        conditional_prefix_fields:
            0
        fields:
            field_name:_sabi_tys
            type:PhantomData<(), RBox>
            size:0 align:1
            package:'std' version:'1.0.0'

            field_name:_sabi_vtable
            type:PrefixRef<'a, RObjectVtable>
            size:8 align:8
            package:'abi_stable' version:'0.11.3'

            field_name:hello
            type:AFunctionPointer
            size:8 align:8
            package:'abi_stable' version:'0.11.3'
            fn pointer(s):
                unsafe fn(_self,param_0,: RRef<'a>, param_1: RStr<'a>)->RResult
                lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, ANONYMOUS], [NONE, NONE], [NONE, NONE]] })

        accessible_fields:
            [Yes, Yes, Yes]
    Tag:
        null
    Extra checks:
        <nothing>
    Repr attribute:C
    Module reflection mode:Module



Error:too many fields
Expected:
    4
Found:
    3
)

[thorio]

It seems to me like extensibility is there to support loading newer libraries than expected, essentially working in the wrong direction to what's needed for a plugin system.

Try compiling the plugin with interface 0.1.1, then loading it with interface 0.1.0. This works in my experience, but is really the opposite of what I wanted.

---

EDIT:
If instead of loading the plugin like this:

let lib = PluginLibRef::load_from_file(&PathBuf::from("tests/data/libexample_provider.so"))
    .expect("plugin must be loadable");

You check the version and type layout yourself, reversing the expected and actual layouts:

let header = lib_header_from_path(&PathBuf::from("tests/data/libexample_provider.so"))
    .expect("plugin library header must be loadable");

// TODO check version strings manually

if let IsLayoutChecked::Yes(layout) = header.root_mod_consts().layout() {
    // Note the full path here, the abi_checking module is hidden from documentation
    // Also note the arguments have been reversed, passing the plugin's layout first
    abi_stable::abi_stability::abi_checking::check_layout_compatibility(layout, PluginLibRef::LAYOUT)
        .expect("interface must be compatible");
};

let lib = unsafe {
    header
        .unchecked_layout::<PluginLibRef>()
        .expect("plugin broke while loading")
};

Then everything seems to work like you would want for a plugin system. I can add new methods/nonexhaustive variants to the interface and still load older libraries, but not the other way around. No clue whether this is still safe to use, but it seems to behave correctly for now.

[SilverMira]

@thorio, that's a good idea for validating the layout, in my testing I did managed to use the unsafe functions to load the library which ignored layout validation, and everything seems to work as expected. My version of ensuring compatibility was a bit different, I used the same check_layout_compatibility function without reversing the arguments but instead permissively passes the validation if the error detected is just AbiInstability::FieldCountMismatch where expected count is more than actual count

fn ensure_compatibility(
    interface: &'static TypeLayout,
    implementation: &'static TypeLayout,
) -> Result<(), AbiInstabilityErrors> {
    let compatibility = abi_stable::abi_stability::abi_checking::check_layout_compatibility(
        interface,
        implementation,
    );
    if let Err(err) = compatibility {
        let incompatibilities = err.errors.iter().filter(|e| !e.errs.is_empty());
        let fatal_incompatibilities = incompatibilities.filter(|err| {
            err.errs.iter().any(|err| {
                !matches!(
                    err,
                    AbiInstability::FieldCountMismatch(assert) if assert.expected > assert.found
                )
            })
        });
        if fatal_incompatibilities.count() > 0 {
            return Err(err);
        }
    }
    Ok(())
}

Checking the generated code for the Trait_TO, this part of the docs does work as expected (ie: default implementation does get invoked if there's no corresponding VTable entry in the library layout).

Accidentally calling newer methods on trait objects from older versions of a library will cause a panic at runtime, unless it has a default implementation (within the trait definition that #[sabi_trait] can see).

It's just the layout checking logic that seems to be contradicting what the docs say.

A library will not load (through safe means) if methods are added anywhere but the end.

[SilverMira]

For people who have stumbled upon this same plugin use case, I have since changed my approach to implementing a plugin system.

Check out remoc's rtc feature which can create the same sort of RPC capability like sabi_trait as long as there is a binary pipe between 2 ends (even across networks). In fact, you can even combine sabi with remoc, with sabi providing the abi stable pipe to a dylib using the channels feature. Since remoc is based on serde, types with Serialize and Deserialize can just work easily without plumbing StableAbi, though of course, serializing/deserializing will have more overhead over just doing pure FFI