performance regression (for debug & single binary) of backtrace capturing after bumping toolchain on macOS

[xxchan]

Describe the bug

I found risingwave becomes significantly slow on my mac. e.g., e2e_test/batch/basic/array.slt.part takes 30+ seconds. After git bisect, I found the regression happens after #6025.

To Reproduce

No response

Expected behavior

No response

Additional context

@skyzh can't reproduce. CI also works fine. Need someone else to confirm it. Confirmed it happens on debug & single binary mode.

• capturing stack backtrace becomes slower and sometimes segfaults on Apple Silicon rust-lang/rust#104388

[skyzh]

Can you collect some perf result on that? Thanks!

[chenzl25]

I also found this regression on debug mode and Sysbench QPS dropped to almost zero. However everything works well on release mode.

[skyzh]

single-binary doesn't work well, while multi-binary looks good. Is there anything that might go wrong with the madsim update? cc @wangrunji0408 there should be no problem with madsim

[skyzh]

Probably not, seems to be a problem with frontend node.

[skyzh]

And ok_or_else seems to consume significant amount of CPU time.

[skyzh]

... because backtrace is slow to create

[skyzh]

In multi-binary:

backtrace captured in 1ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms
backtrace captured in 0ms

In single-binary:

backtrace captured in 393ms
backtrace captured in 369ms
backtrace captured in 370ms
backtrace captured in 365ms
backtrace captured in 362ms

Probably a bug with the Rust compiler, but I would still recommend not returning so many errors.

[skyzh]

Related function:

    pub fn get_schema_by_name(&self, db_name: &str, schema_name: &str) -> Result<&SchemaCatalog> {
        self.get_database_by_name(db_name)?
            .get_schema_by_name(schema_name)
            .ok_or_else(|| {
                let x = Instant::now();
                let err = CatalogError::NotFound("schema", schema_name.to_string()).into();
                println!("backtrace captured in {}ms", x.elapsed().as_millis());
                err
            })
    }

[skyzh]

The suspicious thing is the dwarf region in the binary.

On 2022-07-29:

objdump target/debug/risingwave  --dwarf=frame | wc -l
     371

On 2022-10-16:

objdump target/debug/risingwave  --dwarf=frame | wc -l
 14671494

[skyzh]

Shall we file a bug report to rust-lang? (Though I believe this is more of an LLVM bug / LLVM upgrade)

[skyzh]

Maybe related: rust-lang/rust#98051