Thread 'main' has overflowed its stack when invoking 'connect_with_retry(...)' in debug

[KunoSayo]

I don't know what's wrong with my codes. Maybe I missed something important or else.
I copied the example codes and run it, then the main has overflowed its stack.

Some infomations are blow.

[package]
name = "test_rust"
version = "0.1.0"
edition = "2018"

[dependencies]
bililive = "*"
serde_json = "*"
tokio = {version = "*", features = ["full"]}
futures = "*"
env_logger = "*"
log = "*"

use bililive::*;
use bililive::connect::tokio::*;
use futures::StreamExt;
use serde_json::Value;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    env_logger::builder()
        .filter_level(log::LevelFilter::Trace)
        .init();
    let config = ConfigBuilder::new()
        .by_uid(19550274)
        .await?
        .fetch_conf()
        .await?
        .build()?;
    println!("connecting");
    let mut stream = connect_with_retry(config, RetryConfig::default()).await?;
    println!("connected");
    while let Some(e) = stream.next().await {
        match e {
            Ok(packet) => {
                println!("raw: {:?}", packet);
                if let Ok(json) = packet.json::<Value>() {
                    println!("json: {}", json);
                }
            }
            Err(e) => {
                println!("err: {:?}", e);
            }
        }
    }

    Ok(())
}

With debug log below:

[2021-10-12T03:08:17Z DEBUG reqwest::connect] starting new connection: https://api.live.bilibili.com/
[2021-10-12T03:08:17Z TRACE mio::poll] registering event source with poller: token=Token(0), interests=READABLE | WRITABLE
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z TRACE want] signal found waiting giver, notifying
[2021-10-12T03:08:17Z TRACE want] poll_want: taker wants!
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z DEBUG reqwest::async_impl::client] response '200 OK' for https://api.live.bilibili.com/bili/living_v2/19550274
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z TRACE want] signal: Want
[2021-10-12T03:08:17Z TRACE want] poll_want: taker wants!
[2021-10-12T03:08:17Z DEBUG reqwest::async_impl::client] response '200 OK' for https://api.live.bilibili.com/room/v1/Danmu/getConf
[2021-10-12T03:08:17Z TRACE want] signal: Want
connecting
[2021-10-12T03:08:17Z TRACE mio::poll] deregistering event source from poller
[2021-10-12T03:08:17Z TRACE want] signal: Closed
[2021-10-12T03:08:17Z TRACE mio::poll] registering event source with poller: token=Token(16777216), interests=READABLE | WRITABLE
[2021-10-12T03:08:17Z TRACE tungstenite::handshake::client] Request: "GET /sub HTTP/1.1\r\nHost: tx-gz-live-comet-03.chat.bilibili.com:443\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: FmdlkoSfbo7Ajpao59t+eQ==\r\n\r\n"
[2021-10-12T03:08:17Z TRACE tungstenite::handshake::client] Client handshake initiated.

thread 'main' has overflowed its stack
error: process didn't exit successfully: `target\debug\test_rust.exe` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

Process finished with exit code -1073741571 (0xC00000FD)

If change profile to release and the result will be different

[2021-10-12T03:13:29Z DEBUG reqwest::connect] starting new connection: https://api.live.bilibili.com/
[2021-10-12T03:13:29Z TRACE mio::poll] registering event source with poller: token=Token(0), interests=READABLE | WRITABLE
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z TRACE want] signal found waiting giver, notifying
[2021-10-12T03:13:30Z TRACE want] poll_want: taker wants!
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z DEBUG reqwest::async_impl::client] response '200 OK' for https://api.live.bilibili.com/bili/living_v2/19550274
[2021-10-12T03:13:30Z TRACE want] signal found waiting giver, notifying
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z TRACE want] poll_want: taker wants!
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z TRACE want] signal found waiting giver, notifying
[2021-10-12T03:13:30Z DEBUG reqwest::async_impl::client] response '200 OK' for https://api.live.bilibili.com/room/v1/Danmu/getConf
[2021-10-12T03:13:30Z TRACE want] signal: Want
[2021-10-12T03:13:30Z TRACE want] poll_want: taker wants!
[2021-10-12T03:13:30Z TRACE want] signal: Want
connecting
[2021-10-12T03:13:30Z TRACE mio::poll] deregistering event source from poller
[2021-10-12T03:13:30Z TRACE want] signal: Closed
[2021-10-12T03:13:30Z TRACE mio::poll] registering event source with poller: token=Token(16777216), interests=READABLE | WRITABLE
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::client] Request: "GET /sub HTTP/1.1\r\nHost: hw-sh-live-comet-02.chat.bilibili.com:443\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: VD9IQIO1IarT/iXCoRGmNQ==\r\n\r\n"
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::client] Client handshake initiated.
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::machine] Doing handshake round.
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::machine] Doing handshake round.
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::machine] Doing handshake round.
[2021-10-12T03:13:30Z TRACE tungstenite::handshake::machine] Doing handshake round.
[2021-10-12T03:13:30Z DEBUG tungstenite::handshake::client] Client handshake done.
[2021-10-12T03:13:30Z TRACE tungstenite::protocol] Frames still in queue: 0
[2021-10-12T03:13:30Z TRACE tungstenite::protocol] Frames still in queue: 1
[2021-10-12T03:13:30Z TRACE tungstenite::protocol] Sending frame: Frame { header: FrameHeader { is_final: true, rsv1: false, rsv2: false, rsv3: false, opcode: Data(Binary), mask: Some([74, 119, 221, 22]) }, payload: [0, 0, 1, 1, 0, 16, 0, 0, 0, 0, 0, 7, 0, 0, 0, 1, 123, 34, 99, 108, 105, 101, 110, 116, 118, 101, 114, 34, 58, 34, 49, 46, 56, 46, 50, 34, 44, 34, 107, 101, 121, 34, 58, 34, 86, 73, 77, 115, 68, 117, 116, 55, 112, 69, 53, 109, 113, 98, 120, 119, 103, 56, 105, 82, 107, 71, 71, 81, 52, 57, 115, 98, 117, 100, 77, 115, 103, 55, 88, 49, 71, 84, 110, 79, 113, 119, 84, 105, 69, 80, 48, 79, 75, 75, 103, 83, 70, 65, 70, 119, 101, 50, 69, 101, 113, 71, 77, 121, 121, 119, 52, 67, 71, 66, 72, 104, 100, 110, 49, 84, 70, 119, 104, 81, 67, 80, 101, 54, 99, 119, 95, 65, 120, 45, 114, 82, 122, 120, 67, 114, 108, 74, 118, 98, 52, 72, 51, 75, 119, 56, 53, 76, 83, 111, 118, 99, 90, 49, 56, 80, 75, 69, 75, 70, 73, 99, 55, 109, 84, 75, 105, 122, 55, 81, 52, 106, 103, 98, 111, 110, 80, 100, 119, 61, 34, 44, 34, 112, 108, 97, 116, 102, 111, 114, 109, 34, 58, 34, 119, 101, 98, 34, 44, 34, 112, 114, 111, 116, 111, 118, 101, 114, 34, 58, 50, 44, 34, 114, 111, 111, 109, 105, 100, 34, 58, 52, 53, 57, 54, 53, 54, 55, 44, 34, 116, 121, 112, 101, 34, 58, 50, 44, 34, 117, 105, 100, 34, 58, 49, 57, 53, 53, 48, 50, 55, 52, 125] }
[2021-10-12T03:13:30Z TRACE tungstenite::protocol::frame] writing frame 
    <FRAME>
    final: true
    reserved: false false false
    opcode: BINARY
    length: 265
    payload length: 257
    payload: 0x001101000000700017b22636c69656e74766572223a22312e382e32222c226b6579223a2256494d73447574377045356d71627877673869526b4747513439736275644d736737583147546e4f717754694550304f4b4b6753464146776532456571474d797977344347424868646e3154467768514350653663775f41782d72527a7843726c4a76623448334b7738354c536f76635a3138504b454b464963376d544b697a3751346a67626f6e5064773d222c22706c6174666f726d223a22776562222c2270726f746f766572223a322c22726f6f6d6964223a343539363536372c2274797065223a322c22756964223a31393535303237347d
                
[2021-10-12T03:13:30Z TRACE tungstenite::protocol] Frames still in queue: 0
[2021-10-12T03:13:30Z DEBUG stream_reconnect::stream] Initial connection succeeded.
connected
...

[PhotonQuantum]

That's strange... It might be a bug related to tungstenite.

Unfortunately my laptop broke down yesterday. I'll look into this problem later this week. 🥺

[PhotonQuantum]

I think I may have pinpointed the cause. It seems like an upstream bug.

The default stack size on Windows is 1MB (while on most Linux distros it is 10MB or higher). However, when performing a handshake, a stack buffer in tungstenite is (unexpectedly?) copied multiple times and draining out the stack. On my laptop, it still fits in the 1MB stack, but when I reduce the stack size, I can reproduce this bug.

I'm going to report this issue to upstream. Meanwhile, you may consider:

1. Compile the binary with increased stack size.
   See https://users.rust-lang.org/t/stack-overflow-when-compiling-on-windows-10/50818/8.

2. Spawn a new thread with explicit stack size and run your code there.

fn main() {
    std::thread::Builder::new().stack_size(10*1024*1024).spawn(||{
        tokio::runtime::Runtime::new().unwrap().block_on(run()).unwrap()
    }).unwrap().join().unwrap()
}
async fn run(){
    ...
}

3. Patch tungstenite to decrease read buffer size.
   Change this line of code to

const READ_BUFFER_CHUNK_SIZE: usize = 1024;    // 4096

or something lower. Beware that this may cause a performance regression.

[PhotonQuantum]

Could you please try out the master branch? I've slightly optimized the stack usage, which might help.

[PhotonQuantum]

Upstream PR: snapview/tungstenite-rs#241

[KunoSayo]

I have tried the master branch if there is no wrong with Cargo.toml, but the problem still exists.

bililive = {git = "https://github.com/PhotonQuantum/bililive-rs", branch = "master"}

Then, I changed the stack size and found that 1MB will be enough to run the above codes.
I think the problem is that my default thread stack size is not 1MB for my RAM is 8GB or something else.
The idea of changing stack size gives me a resolution and thanks for report the issue to upstream.

[PhotonQuantum]

And what about this tungstenite patch? You may add it to your project by appending the following lines to your Cargo.toml.

[patch.crates-io]
tungstenite = { git = "https://github.com/PhotonQuantum/tungstenite-rs" }

[KunoSayo]

The patch works fine (not change the thread stack size).

[PhotonQuantum]

Upstream PR merged.

The latest beta version (bililive 0.2.0-beta.4) now contains this fix. It's also backported to bililive v0.1.2. Please update your dependencies and this issue should now be fully solved.