Intermittent 'cannot copy' errors for /usr/lib/sysimage/libdnf5/packages.toml/new in dnf5-5.3.0.0-7.fc44

[AdamWill]

In recent F44 / Rawhide I'm periodically seeing openQA tests fail like this:

There doesn't seem to be any particular trigger, it just...sometimes happens.

[ppisar]

Did a previously run libdn5 application crashed, or do you run multiple libdnf5 applications in a parallel (GnomeSoftware as well as PackageKit are now libdnf5 applications too)?

There is a racy implementation in libdnf5/system/state.cpp:

static std::filesystem::path suffix_new(std::filesystem::path p) {
    p += ".new";
    return p;
}

static void remove_new_suffix(std::filesystem::path p, bool skip_missing) {
    std::filesystem::path p_target = p;
    auto p_new = suffix_new(p);

    if (std::filesystem::exists(p_new) || !skip_missing) {
        utils::fs::move_recursive(p_new, p_target);
    }
}

where utils::fs::move_recursive() is:

void move_recursive(const std::filesystem::path & src, const std::filesystem::path & dest) {
    try {
        std::filesystem::rename(src, dest);
    } catch (const std::filesystem::filesystem_error &) {
        std::filesystem::copy(src, dest, std::filesystem::copy_options::recursive);
        std::filesystem::remove_all(src);
    }
}

I don't know what the author meant with this, but a correct way is to perform rename() and ignore ENOENT error if skip_missing is true. Also the copy-and-remove fallback in move_recursive() is dubious in this case since we operate in the same directory.

[AdamWill]

Did a previously run libdn5 application crashed

No idea offhand. I've just seen this same error in various failed tests I restarted. Lots of other stuff is broken ATM, I did not have time to dig into the logs further.

do you run multiple libdnf5 applications in a parallel (GnomeSoftware as well as PackageKit are now libdnf5 applications too)?

We have tests that test GNOME Software and Cockpit (which uses PackageKit), and GNOME and KDE both do background checks for available updates while we might be doing something in the foreground with dnf, so this is entirely possible, yes. Obviously, this should not break things.

[evan-goode]

This (and many similar errors) may be fixed by #2519 which landed in DNF 5.4.0.0. Let us know if you see it on that version or newer?

[AdamWill]

OK, I'll keep an eye out. As this is affecting F44, are there plans to put DNF 5.4 in F44, or backport this fix to 5.3?

[evan-goode]

Yes, it's testing->stable here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-6072c6888a. We were a bit too late for the beta sadly.

[AdamWill]

Unfortunately looks like it's still happening in current Rawhide. That test definitely had 5.4.0.0, it started out with 5.4.0.0-3.fc45 and updated to 5.4.0.0-4.fc45 before the command that failed ran.

[AdamWill]

And still happening, e.g. here, where we were just doing dnf -y install postgresql-server postgresql-contrib on a fairly idle Server system. That case is actually pretty worrying.

[m-blaha]

I've got a reproducer (not 100%, but I guess still reliable enough). Here is what's happening:

1. a dnf5 transaction runs and attempts to save the new system state files
2. a background process (like packagekit or any other libdnf5 user) loads the system repo, which loads the system state (during libdnf5/repo/solv_repo.cpp::load_system_repo_ext() call). This triggers the recovery code to rename *.new system state files during State::load() call. Since the race window is very small, I'm running a large number of these background processes.

Here is the reproducer:

// g++ -std=c++20 -o reproducer_2601 reproducer_2601.cpp $(pkg-config --cflags --libs libdnf5)
// for i in $(seq 1 100); do echo "=== Run $i ==="; sudo ./reproducer_2601 || break; done

#include <libdnf5/base/base.hpp>
#include <libdnf5/repo/repo.hpp>

#include <signal.h>
#include <sys/wait.h>
#include <unistd.h>

#include <cstdio>
#include <cstdlib>
#include <vector>
    
static void reader_loop() {
    while (true) {
        try {
            libdnf5::Base base;
            base.setup();
            base.get_repo_sack()->load_repos(libdnf5::repo::Repo::Type::SYSTEM);
        } catch (const std::exception & e) {
            std::fprintf(stderr, "reader error: %s\n", e.what());
        }
    }
}

int main() {
    // spawn number of background processes reading the system repo in a loop
    constexpr int n_readers = 20;
    std::vector<pid_t> children;
        
    for (int i = 0; i < n_readers; ++i) {
        pid_t pid = fork();
        if (pid == 0) {
            reader_loop();
            _exit(0);
        }   
        if (pid < 0) {
            std::perror("fork");
            break;
        }
        children.push_back(pid);
    }

    // run the transaction
    int ret = std::system("dnf5 install -y acpi");
    if (ret == 0) {
        ret = std::system("dnf5 remove -y acpi");
    }

    for (pid_t pid : children) {
        kill(pid, SIGTERM);
    }
    for (pid_t pid : children) {
        waitpid(pid, nullptr, 0);
    }

    return ret != 0 ? 1 : 0;
}