[Architecture] Migrating thread-per-connection to libev : DB Layer Blocking Strategy

[dreamer-00]

Context

I am opening this issue to discuss the architectural approach for the GSoC '26 project: Implementing event-driven concurrency in OLTP test tool (DBT-5).

While tracing the connection flow in the BrokerageHouse module, I mapped out the current concurrency model and wanted to align on the strategy for handling the database layer before drafting a PoC patch.

Current State

Currently, the listener accepts a socket, and entryWorkerThread() spawns a detached workerThread() per connection. Each thread maintains its own DB connection and blocking receive loop.

The libev Refactor & The Bottleneck

A straightforward refactor to libev would replace the pthread_create path with per-connection ev_io watchers, moving the socket I/O entirely into the event loop.

The architectural challenge: The transaction handlers ultimately invoke synchronous, blocking database calls. If these are executed directly inside the libev event callback, a single slow DB query will block the entire event loop, freezing all other network I/O.

Discussion: Direction for the DB Layer

To prevent the event loop from blocking, how are we envisioning the database layer transition?

Are we leaning toward:

1. Fully Asynchronous: A transition toward asynchronous DB drivers (e.g., using non-blocking calls like PQsendQuery if targeting PostgreSQL directly, wrapping them in their own libev watchers).
2. Hybrid Thread-Pool Model: libev strictly handles network I/O (accepting connections and reading/writing buffers), but dispatches the actual blocking DB transactions to a bounded, fixed-size worker thread pool.

Would love to get the maintainers' thoughts on which direction aligns best with the long-term goals for DBT-5.

[markwkm]

BrokerageHouse is basically a custom transaction manager, using a threaded model. What I think we want to do here is to implement a multi-process event-driven model, and not use threads at all. Similar to what was done here for the Client component of the dbt-2 (fair-use implementation of the TPC-C).

It may be more accurate to think of it as a re-implementation than a refactor, because it's not about using event-driven code to do what threads are doing. It's about writing a transaction manager to properly use event-driven code.

[dreamer-00]

Hi Mark,

Understood completely. A multi-process, event-driven architecture (master/worker model) makes perfect sense here to completely eliminate thread overhead and isolate any synchronous DB blocking to individual processes.

Treating this as a ground-up re-implementation of the transaction manager rather than a simple refactor provides a much cleaner architectural path.

I will pull the dbt-2 repository today, specifically dissect the Client component's process management and libev implementation, and use that as the blueprint for mapping out the new BrokerageHouse architecture. I'll follow up with a structural outline once I've fully analyzed the dbt-2 approach.

[dreamer-00]

Hi Mark,

Following up on my architectural analysis and seeing the parallel work starting on the Driver component, I have mapped out the re-implementation strategy for the BrokerageHouse (acting as the equivalent to the dbt-2 Client component).

Since the Driver is moving to a multi-process model generating the load, the BrokerageHouse needs a matching multi-process transaction manager to ingest it without thread contention.

Proposed BrokerageHouse Re-implementation (BrokerageHouse2.cpp):

1. Multi-Process Master/Worker Architecture

Instead of pthread_create per connection, the main BrokerageHouse process will fork() a fixed pool of Worker processes (likely mapped to CPU cores via sched_setaffinity).

2. The Event Loop (Per-Worker)

Each Worker process initializes its own isolated ev_default_loop.

Workers will monitor the incoming sockets via ev_io (EV_READ) watchers.

3. Resolving the DB Blocking Bottleneck

This multi-process architecture natively solves the database blocking issue I raised earlier.

Because each Worker is an entirely separate OS process, if Worker A executes a synchronous/blocking ODBC transaction against PostgreSQL, it only stalls Worker A's specific event loop. Workers B, C, and D continue processing network I/O and executing DB transactions concurrently.

We bypass the need to rewrite the entire ODBC layer into asynchronous drivers, achieving high concurrency through OS-level process isolation rather than thread-switching.

I plan to isolate this into a new BrokerageHouse2.cpp alongside the existing implementation to ensure we don't break the current threaded benchmark while building out the libev architecture.

Let me know if you agree with this mapping for the server-side component, and I will start drafting the process-spawning skeleton.

[markwkm]

Since the Driver is moving to a multi-process model generating the load, the BrokerageHouse needs a matching multi-process transaction manager to ingest it without thread contention.

That's not accurate. The concurrency model of each individual program should not affect its ability to interact with the other programs. If it does, that something else has been broken. :)

Instead of pthread_create per connection, the main BrokerageHouse process will fork() a fixed pool of Worker processes (likely mapped to CPU cores via sched_setaffinity).

The processor affinity (fyi, it's actually mapped to logical processor, i am picky about using cpu/core/processor/etc terms correctly :) ) is not necessary for the event-driven related changes, so let's keep it simple for now.

This multi-process architecture natively solves the database blocking issue I raised earlier.

While this has some truth in it, it's not completely accurate to think of it as solving a blocking issue. It's more about applying a different concurrency model that can reduce system utilization. Event-driven code, to put it crudely, changes what to do based on whether it thinks it has to wait for something to finish, as opposed to running for various slices of time.

[dreamer-00]

Thanks for the corrections, Mark. That makes perfect sense.

Understood on the decoupling. The network boundary isolates the concurrency models entirely.

I appreciate the terminology correction on logical processors vs. physical cores. I will drop any sched_setaffinity logic to keep the initial re-implementation strictly focused on the event-driven architecture and process spawning.

That philosophy on system utilization is a helpful framing. Moving from time-sliced OS thread management to state-based yielding within the event loop is the primary goal.

I am dissecting client2.c from dbt-2 now to understand the exact mechanics of the fork() and ev_default_loop setup you used there, and will begin drafting BrokerageHouse2.cpp based on that bare-bones structure.

[markwkm]

I am dissecting client2.c from dbt-2 now to understand the exact mechanics of the fork() and ev_default_loop setup you used there, and will begin drafting BrokerageHouse2.cpp based on that bare-bones structure.

I wouldn't call dbt-2's client2.c a bare-bones structure. It's a fully implemented and working program.

If you're looking for a bare-bones structure, any "hello world" type of example would be more appropriate.

[dreamer-00]

Hi Mark,

Fair point on the terminology. "Bare-bones" was definitely the wrong word for a production client—I meant extracting the underlying master/worker skeleton from it. Starting with a "hello world" prototype is exactly the right move.

After dissecting the process management in client2.c, here is the architectural blueprint I am targeting for BrokerageHouse2.cpp:

Phase 1: Pre-Forking Process Management

Master Process: The main process initializes the network listener on the designated port.

Worker Pool: Instead of spawning threads, the Master executes a pre-fork model (like startup() in client2.c), calling fork() to spawn a fixed pool of Worker processes. (Omitting sched_setaffinity for now to keep it simple).

Phase 2: Master/Worker Event Loops & Socket Passing

Master Loop: The parent runs ev_default_loop to accept() incoming connections. It then passes the accepted file descriptor to an available Worker via a Unix domain socketpair.

Worker Loop: Each spawned Worker runs its own isolated ev_default_loop. It listens on the socketpair for new connections handed down by the Master, and sets up an ev_io watcher (EV_READ) to handle the transaction stream via state-based yielding.

Phase 3: Database Isolation & System Utilization

Because each Worker is an entirely isolated OS process, if a Worker executes a synchronous ODBC transaction and waits, it only stalls that specific Worker's event loop.

The Master and all other Workers remain unblocked, maximizing CPU efficiency by eliminating thread context-switching overhead while avoiding a total rewrite of the ODBC layer.

Next Step (The "Hello World" Prototype):

Taking your advice, before touching any DBT-5 database logic, I am going to build a standalone "hello world" prototype of Phase 1 and Phase 2. It will just be a Master process that forks Workers, accepts a socket, passes it via socketpair, and the Worker echoes a message.

If this structural outline looks mathematically correct to you, I will begin writing that prototype today.

[markwkm]

Some of the terminology is not 100% accurate, but I'll worry about that more depending on whether that is getting in the way of understanding what needs to be done.

I feel like you are asking two things here, so I will answer as I perceive it.

One question that I think you are asking, is if you understand what to do. It sounds like you are understanding some of the concepts. You're not going to be able to implement it in phases (this is one example of what I meant about terminology) as you outlined it. It all has to be done altogether in order to function. The other thing about your plan is that one step seems to be about what to do, and another step isn't a step, but a rationale for how the program already works. Thus it sounds like you might understand some of what needs to be done.

The other question that I think you are asking is how you should learn how to do what you propose to do. With respect to this, this issue should just be about proposing what you will do. We are extra sensitive because much of the increased activity here has been about getting accepted as a GSoC contributor, at which point we will be prepared to instruct as much as we are to mentor on how to work in open source software development, so this issue should not be about that.

I'm not sure what the best advice is to give you. If you think hashing out the terminology enables you to better to do the work, then I think you should take another stab at showing you understand how the thread model is used in the brokerage house, and what needs to be changed to work in an event-driven model as opposed to why, without referencing dbt-2, as that is just an example to show that it can be done, not what to copy. It's also ok to experiment with code first to see if you think you can do what you want to propose to be done.

[dreamer-00]

Mark,

Taking your advice, here is the exact breakdown of how the thread model currently operates in the BrokerageHouse, and exactly what I propose to change for the event-driven model.

1. Current Thread Model Implementation
The Master process enters a while(true) loop in CBrokerageHouse::startListener(). When m_Socket.dbt5Accept() receives a new connection, it constructs a TThreadParameter and passes it to entryWorkerThread(). This function then calls pthread_create() to spawn a detached thread running workerThread(). Inside workerThread(), the execution blocks in a do-while loop on sockDrv.dbt5Receive() waiting for driver transactions.

2. Proposed Event-Driven Multi-Process Model
To implement the event-driven model, I will remove all pthread logic (entryWorkerThread, pthread_create, and thread attribute structs) from BrokerageHouse.cpp and BrokerageHouseMain.cpp. Because this cannot be phased, the new concurrency structure will be implemented altogether as follows:

Pre-Forking: Before the listener starts, the Master will fork() a fixed pool of Worker processes. The Master and Workers will be connected via socketpair(AF_UNIX, SOCK_STREAM, 0, fd).

Master Event Loop: The Master will run an ev_default_loop. It will register an ev_io watcher on the main listener socket. Upon triggering, it will accept() the connection and immediately pass the client File Descriptor to an available Worker via the socketpair using sendmsg() with SCM_RIGHTS.

Worker Event Loop: Each Worker will run its own ev_default_loop. It will maintain an ev_io watcher on its end of the socketpair. When it receives a client File Descriptor from the Master, it will assign it to a new ev_io watcher to asynchronously handle the database routing logic currently housed in workerThread().

I have been experimenting locally with the socketpair, libev, and sendmsg() file-descriptor passing mechanics to verify this execution path. I will use this exact structural change as the basis for my final proposal.

[markwkm]

I'm going to continue to respond in a bit of an unconventional way since I feel the purpose of this issue is really more about vying for the GSoC mentorship that would educate the prospective contributor about the benchmark so they can do this work.

What you are not showing here is how the currency model represents the components of the benchmark that it implements. Understanding what the benchmark is doing is just as important as understand the nuances between concurrency models.

[dreamer-00]

Mark,
After reading through workerThread() with the benchmark in mind rather than just the concurrency mechanics, here is what I now understand about what needs to change and why.
In the current model, each thread represents a single persistent TPC-E Driver session. workerThread() initializes one DB connection, one CSendToMarket channel to the MEE, and all 12 transaction handler objects once at session start, then loops indefinitely on that socket receiving whatever transaction the Driver sends next — TRADE_ORDER, MARKET_FEED, TRADE_RESULT, and so on — for the entire benchmark run. The thread is not a request handler. It is a stateful session that lives as long as the Driver connection lives.
The architectural consequence for the event-driven re-implementation is that this session state — currently held implicitly on the thread stack — must become an explicit per-connection context struct allocated at accept() time. When an ev_io watcher fires on a client FD, the callback retrieves that context, dispatches the transaction type from pMessage->TxnType, and returns. The DB connection, MEE channel, and transaction handlers in that context persist across watcher firings exactly as they do across loop iterations today.
The multi-process model fits here because each Worker process owns its own set of these session contexts independently, without shared memory or locking. The m_LogLock mutex currently in logErrorMessage() is one concrete example of shared state that disappears entirely under process isolation.
What does not change is the session contract itself: one persistent socket, one DB connection, one MEE channel per Driver connection, for the duration of the run.

[greysome]

Hi, I agree with the use of an ev_io watcher for client connections and a per-connection context struct. Though I think the idea to "dispatch the transaction type from pMessage->TxnType" will require a large-scale refactor of the synchronous PQexec() calls within the transaction frames to be event-driven (using PQsendQuery()).

I've been thinking about how to implement it in a clean way across all frames of all transaction types. Perhaps one way is to add a state variable to the context struct, and then implement ev_io watchers for the DB connection and MEE connection, each with its own state machine?

e.g. Inside the DB connection callback, we could have a branch that goes
if state=FRAME_1_QUERY_1_RECV, then

1. set state=FRAME_1_QUERY_2_SEND
2. call PQsendQuery() for frame 1 query 2
3. return from the callback

Thoughts?

[sahitya-chandra]

I think the multi-process model @markwkm described earlier (like dbt-2's client2.c) might already handle this. Each worker process has its own event loop and DB connection, so a blocking PQexec in one worker only stalls that worker, the others keep running.

A per-frame state machine with PQsendQuery across all 12 transaction types feels like a pretty large refactor. I'm not sure it's necessary if process isolation already contains the blocking. Maybe start with blocking PQexec inside worker processes (which is what dbt-2 does) and revisit non-blocking libpq later if needed?

[dreamer-00]

Hi,
Having thought about this — I agree with sahitya-chandra's assessment. Process isolation already contains the blocking: a synchronous PQexec() inside a Worker only stalls that specific Worker's event loop, which is precisely the property the multi-process model is designed to provide. This is also consistent with how dbt-2's client2.c handles it.
A full transition to PQsendQuery() with per-frame state machines across all 12 transaction types is a meaningful improvement worth noting, but the scope would dwarf the concurrency re-implementation itself. The cleaner path is to establish the process model first with blocking libpq, then revisit non-blocking libpq as a subsequent iteration once the architecture is stable.

[markwkm]

After reading through workerThread() with the benchmark in mind rather than just the concurrency mechanics, here is what I now understand about what needs to change and why. In the current model, each thread represents a single persistent TPC-E Driver session.

I feel like you are using "Driver session" to be synonymous with the code. I'm not sure if you are making the connection to what the code is modeling in the specification so I'm not sure if what you think you need to do is accurate..