Prevent writing to > 1 stream in a transaction

[damianh]

The (original) design purpose of stream is to represent an Aggregate Root and this is the consistency boundary. Writing to >1 stream in a transaction breaks this and is not an officially supported scenario. Besides, not all persistence engines support .net transactions, let alone 'distributed' transactions (DTC).

They purpose of the transaction support was really to allow a user to interact with another piece of infrastructure transactionally, i.e. read from a queue or post to an NServiceBus endpoint.

When a user tries to update more than one stream in a single transaction, we should throw.

Some people like to run with scissors, so I am open to allowing an override to prevent the exception from being thrown that must be explicitly set during wireup. Anyone doing this does so at their own risk and without support.

[irium]

Yes, I need with such an override. Writing to > 1 stream is needed functionality for me and I agree to use it in explicit manner.

[bartelink]

@irium Perhaps you could share a little about your specific case [and the degree to which it requires transaction support to achieve a specific goal]?

[gregoryyoung]

I'd be interested in the use case normally an altered model removes such
needs

On Monday, August 25, 2014, irium notifications@github.com wrote:

Yes, I need with such an override. Writing to > 1 stream is needed
functionality for me and I agree to use it in explicit manner.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

Studying for the Turing test

[damianh]

As there are more implementations coming into the fold that don't support
System.Transactions, I'm strongly considering removing transaction support
altogether. It appears to be encouraging the wrong sort of usage too.
On 26 Aug 2014 03:07, "Greg Young" notifications@github.com wrote:

I'd be interested in the use case normally an altered model removes such
needs

On Monday, August 25, 2014, irium notifications@github.com wrote:

Yes, I need with such an override. Writing to > 1 stream is needed
functionality for me and I agree to use it in explicit manner.

—
Reply to this email directly or view it on GitHub
<
https://github.com/NEventStore/NEventStore/issues/287#issuecomment-53349436>

.

Studying for the Turing test

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

[larsw]

@irium Jonathan has written a couple of good blog posts on how to prevent using 2PC:
http://blog.jonathanoliver.com/removing-2pc-two-phase-commit/
http://blog.jonathanoliver.com/how-i-avoid-two-phase-commit/
http://blog.jonathanoliver.com/my-beef-with-msdtc-and-two-phase-commits/

--larsw

[irium]

Yes, I agree, that my use case is very specific. And I don't want to use 2PC. Opposite, I want to use only single transaction for ALL. Let's describe. My system is far from high-load, so I decided go away from all that "eventual consistency" crap. My event store, aggregates and read model all resides in single database. Also, my app is plain ASP.NET MVC hosted at provider, so I cannot use all that "workers" for handling queues, async buses etc. I decided that all command and event processing should be synchronous. And ALL changes should go in one transaction. Transaction here is started by web api handler, so one api request - one transaction. By "all changes" I mean: commiting new events to event store, updated aggregates and simultaneously update read model, so read model is always consistent.

Such transaction could easily touch many aggregates, because domain is very complex and contains many "many-to-many" relations. Yes, I've read Vaughn Vernon articles, so I've done huge work to convert them to "one-to-many" ones. But anyway many aggregates changed by single command and I wish to commit them simultaneously.

As for "altered model removes such needs". I also want to store originating commands also for auditing purposes. So I decided to have a separate "committed commands" stream. Obviously, they shoud be persisted together with the events, that are caused by the command. So anyway there will be at least 2 streams to be written together.

All in all I understand, that "there are more implementations coming into the fold that don't support
System.Transactions". And it's sad that by following all that "modern" things "a-la clouds, nosql" etc the library is going to drop some of it's functionality. I've chosen NEventStore at first because it's a general purpose event storing library that is n't limiting me to specific use cases. Let you remember, that events could by stored not only for persisting aggregates, but for many other different reasons.

So I'll be glad, that transaction support could remain in the NEventStore. Perhaps by some sort of extension or other customization mechanism, but al least to have such extensibility point. So transaction-capable persistance engines could leverage that.

[gregoryyoung]

What happens when you have more than one read model?

Also saying 'eventual consistency crap' sets off my muppet detector.

I mean: commiting new events to event store, updated aggregates and
simultaneously update read model, so read model is always consistent.

No such thing as a consistent read model unless you use pessimistic locking.

Frankly I would spend some time reading up on things as you seem very lost
and probably should not be working on a paid project the current level of
understanding.

On Tuesday, August 26, 2014, irium notifications@github.com wrote:

Yes, I agree, that my use case is very specific. And I don't want to use
2PC. Opposite, I want to use only single transaction for ALL. Let's
describe. My system is far from high-load, so I decided go away from all
that "eventual consistency" crap. My event store, aggregates and read model
all resides in single database. Also, my app is plain ASP.NET MVC hosted
at provider, so I cannot use all that "workers" for handling queues, async
buses etc. I decided that all command and event processing should be
synchronous. And ALL changes should go in one transaction. Transaction here
is started by web api handler, so one api request - one transaction. By
"all changes" I mean: commiting new events to event store, updated
aggregates and simultaneously update read model, so read model is always
consistent.

Such transaction could easily touch many aggregates, because domain is
very complex and contains many "many-to-many" relations. Yes, I've read
Vaughn Vernon articles, so I've done huge work to convert them to
"one-to-many" ones. But anyway many aggregates changed by single command
and I wish to commit them simultaneously.

As for "altered model removes such needs". I also want to store
originating commands also for auditing purposes. So I decided to have a
separate "committed commands" stream. Obviously, they shoud be persisted
together with the events, that are caused by the command. So anyway there
will be at least 2 streams to be written together.

All in all I understand, that "there are more implementations coming into
the fold that don't support
System.Transactions". And it's sad that by following all that "modern"
things "a-la clouds, nosql" etc the library is going to drop some of it's
functionality. I've chosen NEventStore at first because it's a general
purpose event storing library that is n't limiting me to specific use
cases. Let you remember, that events could by stored not only for
persisting aggregates, but for many other different reasons.

So I'll be glad, that transaction support could remain in the NEventStore.
Perhaps by some sort of extension or other customization mechanism, but al
least to have such extensibility point. So transaction-capable persistance
engines could leverage that.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

Studying for the Turing test

[irium]

What is "more than one read model" ? Each bounded context has it's own read model which includes of course many views (tables in my case).

Also saying 'eventual consistency crap' sets off my muppet detector.

By 'eventual consistency crap' I mean that eventual consistency has it's own goal and it's own corresponding cost. I don't need the goal - scalability and therefore don't want to pay the cost - eventual consistency. "Crap" here is not that something bad, but means that I don't want to use something because it's "cool" or "modern" thingy.

Each tool/framework/methodology has it's own purpose with pros and cons, so should be selected carefully depending of my concrete situation.

No such thing as a consistent read model unless you use pessimistic locking.

Seems you are very biased toward CQRS+ES. How about CQRS without ES where read model=write model too? In such case read model is always consistent with write one.

Further, I'm not sure that I need in ES as aggregate rehydration source. I'm still in making decision - maybe I'll stop at using traditional ORM-mapped to RDBMS write model (aggregate state). So event store will be used mostly for auditing purpose and as 'single source of truth' for rebuilding entire model when it needed. This will be also useful for reproducing/debugging critical issues.

Frankly I would spend some time reading up on things as you seem very lost
and probably should not be working on a paid project the current level of
understanding.

It's not very polite from you to say such things. How do you know what and how many I've read and whether or not I'm lost? If I would be lost, I could ask for advice (on stackoverflow for ex.) But I precisely know what I want and what I need in my project and all that ddd/cqrs/es technology.

But this is all offtopic. Anyway I could just fork and go on my own.
Thanks.

[gregoryyoung]

If you don't know what 'more than one read model' is you have not come very
far. Most systems end up with multiple read models. There is also no
concept of read model per context (quite often it's one read model for
multiple contexts/ services)

Do you know the arguments against storing events and domain state as you
describe? Literally hundreds of projects have failed due to this.

As for read models yes your reads are stale unless you use pessimistic
locking. If you read data and I change it is your read still valid? If you
take 10ms just over the wire from your SQL box then your data already had a
10ms sla.

On Tuesday, August 26, 2014, irium notifications@github.com wrote:

What is "more than one read model" ? Each bounded context has it's own
read model which includes of course many views (tables in my case).

Also saying 'eventual consistency crap' sets off my muppet detector.

By 'eventual consistency crap' I mean that eventual consistency has it's
own goal and it's own corresponding cost. I don't need the goal -
scalability and therefore don't want to pay the cost - eventual
consistency. "Crap" here is not that something bad, but means that I don't
want to use something because it's "cool" or "modern" thingy.

Each tool/framework/methodology has it's own purpose with pros and cons,
so should be selected carefully depending of my concrete situation.

No such thing as a consistent read model unless you use pessimistic
locking.

Seems you are very biased toward CQRS+ES. How about CQRS without ES
where read model=write model too? In such case read model is always
consistent with write one.

Further, I'm not sure that I need in ES as aggregate rehydration source.
I'm still in making decision - maybe I'll stop at using traditional
ORM-mapped to RDBMS write model (aggregate state). So event store will be
used mostly for auditing purpose and as 'single source of truth' for
rebuilding entire model when it needed. This will be also useful for
reproducing/debugging critical issues.

Frankly I would spend some time reading up on things as you seem very lost
and probably should not be working on a paid project the current level of
understanding.

It's not very polite from you to say such things. How do you know what and
how many I've read and whether or not I'm lost? If I would be lost, I could
ask for advice (on stackoverflow for ex.) But I precisely know what I want
and what I need in my project and all that ddd/cqrs/es technology.

But this is all offtopic. Anyway I could just fork and go on my own.
Thanks.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

Studying for the Turing test

[irium]

If you take 10ms just over the wire from your SQL box then your data already had a 10ms sla.

This will always be true not depending of storage or CQRS used. What I'm talking about - is transactional consistency. Remember that "read committed" transaction level? (If I read data and you change it during or after my transaction completes - than in scope of my transaction your changes simply doesn't exists. I'll see them on next read). I.e. user sees read model in state when his transaction executes.

User expects to see immediate results of his transaction. This is what I'm talking about. And this is what eventual consistency is breaking. I'm not saying that is bad. I'm saying I don't need it. More important - my customer is far from any technical competence and it will be very difficult to explain him why he cann't see just immediate results of his action.

Do you know the arguments against storing events and domain state as you describe?

As far as I know CQRS doesn't neccessarily implies using ES, isn't it? Mostly every article about CQRS suggests to start from just splitting commands from queries, yes? That's what I already did.
So I decided to go slightly further to ES mostly because of auditing needs. ES seems to me (in this project) as mostly easy way to get full auditing together with possibilities for future improvements. I have already described my needs. They don't include classical "aggregate roots' event sourcing". What I really need - is full log of all user actions/events he had done to be able to import it into other system. Imagine sort of "disconnected" scenario - in fact it is - customer need to get some "portable" version of system with limited set of data in it. Then this system is installed on notebook and used somewhere offline. After that he connects to the main system remotely and imports all changes back.
That's my "specific" needs. So I don't see "arguments against storing events and domain state as you describe" here.

Again, this is offtopic. Original issue was about transactions. If NEventStore author will drop support for it - I can just fork and go my own.

[damianh]

In event sourcing the stream is the consistency boundary. The original
purpose of the transaction support was to be able to do something like read
from a queue and append to a stream. It's a pity that the implementation
didn't enforce that scenario somehow. Even that premise is weak because
such scenarios are better handled through idempotency... Clearly the
transactional support has allowed (maybe even encouraged) people down the
wrong path, re-enforcing my desire to remove it altogether.

Irium, you can of course stay with v5 and fork if needed.

On 27 Aug 2014 07:01, "irium" notifications@github.com wrote:

If you take 10ms just over the wire from your SQL box then your data
already had a 10ms sla.

This will always be true not depending of storage or CQRS used. What I'm
talking about - is transactional consistency. Remember that "read
committed" transaction level? (If I read data and you change it during
or after my transaction completes - than in scope of my transaction
your changes simply doesn't exists. I'll see them on next read). I.e. user
sees read model in state when his transaction executes.

User expects to see immediate results of his transaction. This is what
I'm talking about. And this is what eventual consistency is breaking. I'm
not saying that is bad. I'm saying I don't need it. More important - my
customer is far from any technical competence and it will be very
difficult to explain him why he cann't see just immediate results of his
action.

Do you know the arguments against storing events and domain state as you
describe?

As far as I know CQRS doesn't neccessarily implies using ES, isn't it?
Mostly every article about CQRS suggests to start from just splitting
commands from queries, yes? That's what I already did.
So I decided to go slightly further to ES mostly because of auditing
needs. ES seems to me (in this project) as mostly easy way to get full
auditing together with possibilities for future improvements. I have
already described my needs. They don't include classical "aggregate roots'
event sourcing". What I really need - is full log of all user
actions/events he had done to be able to import it into other system.
Imagine sort of "disconnected" scenario - in fact it is - customer need to
get some "portable" version of system with limited set of data in it. Then
this system is installed on notebook and used somewhere offline. After
that he connects to the main system remotely and imports all changes back.
That's my "specific" needs. So I don't see "arguments against storing
events and domain state as you describe" here.

Again, this is offtopic. Original issue was about transactions. If
NEventStore author will drop support for it - I can just fork and go my own.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

[damianh]

Regarding eventual consistency...

First of all, it's simple to give users 'immediate' results in an
eventually consistent system. There are plenty of patterns for that. If you
use Gmail, amazon, ebay or any multitude of web applications out there, you
are using such systems. The 'eventually' part is usually measured in
milliseconds. Just because a user says 'I want to see the thing I just
added in the list of things' does not mean one big transaction.
On 27 Aug 2014 07:01, "irium" notifications@github.com wrote:

If you take 10ms just over the wire from your SQL box then your data
already had a 10ms sla.

This will always be true not depending of storage or CQRS used. What I'm
talking about - is transactional consistency. Remember that "read
committed" transaction level? (If I read data and you change it during
or after my transaction completes - than in scope of my transaction
your changes simply doesn't exists. I'll see them on next read). I.e. user
sees read model in state when his transaction executes.

User expects to see immediate results of his transaction. This is what
I'm talking about. And this is what eventual consistency is breaking. I'm
not saying that is bad. I'm saying I don't need it. More important - my
customer is far from any technical competence and it will be very
difficult to explain him why he cann't see just immediate results of his
action.

Do you know the arguments against storing events and domain state as you
describe?

As far as I know CQRS doesn't neccessarily implies using ES, isn't it?
Mostly every article about CQRS suggests to start from just splitting
commands from queries, yes? That's what I already did.
So I decided to go slightly further to ES mostly because of auditing
needs. ES seems to me (in this project) as mostly easy way to get full
auditing together with possibilities for future improvements. I have
already described my needs. They don't include classical "aggregate roots'
event sourcing". What I really need - is full log of all user
actions/events he had done to be able to import it into other system.
Imagine sort of "disconnected" scenario - in fact it is - customer need to
get some "portable" version of system with limited set of data in it. Then
this system is installed on notebook and used somewhere offline. After
that he connects to the main system remotely and imports all changes back.
That's my "specific" needs. So I don't see "arguments against storing
events and domain state as you describe" here.

Again, this is offtopic. Original issue was about transactions. If
NEventStore author will drop support for it - I can just fork and go my own.

—
Reply to this email directly or view it on GitHub
#287 (comment)
.

[bartelink]

@irium May I suggest you read from 01:00 AM this AM in https://jabbr.net/#/rooms/DDD-CQRS-ES - It'll definitely help you understand some of the concerns here.

I think it's also critical you listen to Greg - he could have sugar coated it a lot better but you'll find your ideas will eventually become more consistent with his. I have had similar ideas to yours pass through my mind and can understand you're trying to make things simple, but it's fundamentally a really bad idea.

Not sure if you read them, but be careful with confirmation bias reading posts from @jbogard 's about the ES / eventual consistency 'paradoxes' / 'pragmatic' solutions.

[irium]

@damianh Excuse me, but you didn't read what I'm written. I don't need event sourcing, but need event storing. From my point of view theese are different things. I looked for a general, mature and stable event store framework. All of that I've found in NEventStore. From all libs I've seen it seems the best choice. Event sourcing is just an "addition" that goes for free with it. (Surely, from my specific point of view). Maybe in some point of future I'll continue to full "right" event sourcing and then all your words become true.

Again. Splitting single "big" transaction into multiple small ones - one for every stream (aggregate root) - implies that I need to introduce many "process managers" that should control every step transaction is completed successfully. Yes, that's not a big deal, but consequent need to create, implement and maintain huge number and complex "compensating" actions -- seems as over-complicating payment for what I don't really need.

I've read jbogard's and many others' good blogs, articles and so on. (In fact, I'm subscribed to jbogard's great blog and following it for a long time) So I know what I'm say about. Yet another circumstance - I'm under heavy time limit pressure and afraid that implementing full ES in "right way" (with all mentioned compensating actions logic) - will take much more time than I have.

Again, reread the scenario I've described. There's no requirement of "use event sourcing". I need just full domain events log to be able to import it into another system (which in turn doesn't use ES at all).

you're trying to make things simple, but it's fundamentally a really bad idea.

Heh, "keeping things simple" - is what software designs and architectures all about, isn't it? Maybe I'm wrong and such way "will fail", but that will be my own fall and good lesson :)

From all my experience (over 20 years in sw development) - over-complicating things is much worse then "under-complicating". Complication is always can be added later when it really needed, but getting rid of unneeded things - is much harder.