Add support for multiple filters in consumer grous

[mtmk]

• Static config uses Filters (string array) only — removed single Filter property
• Elastic config uses PartitioningFilters array of {filter, partitioning_wildcards} pairs, removed separate Filter/Filters/PartitioningWildcards properties
• Full-subject partitioning uses empty partitioning_wildcards array [] instead of [-1] sentinel
• Partition filter format is {partition}.{filter} (e.g., 0.orders.*) instead of {partition}.>
• Empty partitioning_filters array defaults to > with full-subject partitioning (matches orbit.go PR Add helper to delete elastic work queue consumers #46)
• Added Go interop tests via GoHarness against orbit.go/[email protected]

• Resolves Support Multiple Subscription Filters in PCGroups #39

Cross-implementation compatibility

JSON config stored in NATS KV is interoperable with Go (pcgroups/v0.2.0) and Java:

• Static config JSON: "filters" string array (no "filter" field)
• Elastic config JSON: "partitioning_filters" array of {"filter": "...", "partitioning_wildcards": [...]} objects (no separate "filter"/"partitioning_wildcards" fields)
• Full-subject partitioning: empty partitioning_wildcards array [] — Go never had [-1], so this removes the interop gap
• Empty partitioning_filters: empty array defaults to > filter with full-subject partitioning (requires NATS 2.12+)
• Consumer subscription subjects: {partition}.{filter} format matches Go behavior
• Verified: GoHarness interop tests confirm .NET-creates/Go-consumes and Go-creates/.NET-consumes round-trips

[mtmk]

Hey @colprog, does this make sense to you? thanks

[colprog]

Hi @mtmk , Thanks for coming up with a patch so quickly. This is exactly what I wanted. If I understand correctly, using [-1] as partitioningWildcards will require nats-server 2.12+?

[mtmk]

hey @colprog

Hi @mtmk , Thanks for coming up with a patch so quickly. This is exactly what I wanted. If I understand correctly, using [-1] as partitioningWildcards will require nats-server 2.12+?

spot on! yes, server must be 2.12+ hence the test failures on 2.10 and 2.11. fixing them now. edit: tests are fixed.

[mtmk]

@jnmoyne @piotrpio heads up, this PR adds two .NET-only extensions to PCGroups that aren't in orbit.go yet:

1. Multiple filters per consumer group (Filters property alongside the existing Filter)
2. [-1] sentinel for partition-by-full-subject: a convention we introduced here that uses {{Partition}} without wildcard positions (requires NATS server 2.12+)

Would be good to get your thoughts on whether these make sense to standardize across implementations, or if you'd approach them differently in Go.

[jnmoyne]

This is great, thanks! Actually there have been a few changes to the Go implementation lately and it's due for a new release, and we should synchronize the two. Multi-filter has actually already be done in Go :) synadia-io/orbit.go#44. The one difference however is since the plan is to cut a new release (as soon at that last PR gets merged in) I decided to add Filters and remove Filter since (talking from experience here :)) it can get irritating and source of issues to keep the old single filter field around, since PCG is so new might as well break backwards compatibility but do it right with just the array. So .net should do the same so they are interoperable (there's actually also now a Java version that just got released, and that uses the new 'array of filters only' version of the config so it interoperates with Go, and with .net once you make that change.

I like the idea of -1 for partition on the whole subject, so I'll add that to Go (and Java).

Note that there are other PRs that have merged to the Go version since the original release with some small improvements that you should check out.

[colprog]

Using -1 for partition, we could support orders.> or even > as filter, Is this something we should pursue is a future release?

[jnmoyne]

I would rather have PartitioningWildcards array being empty means it partitions over the whole subject (rather than have a single -1 entry in that array. We can get all versions to the same level (all the changes to the Go version plus empty PartitioningWildcards) and then make a release in all languages (where they can interoperate together).

[mtmk]

thanks @jnmoyne

I think all the feedback is addressed:

• Removed Filter, keeping only Filters array (static) and PartitioningFilters array (elastic). should match PR Implements multiple filters for static and elastic partitioned consumer groups orbit.go#44
• Replaced [-1] sentinel with empty PartitioningWildcards array for full-subject partitioning
• Consumer subjects now use {partition}.{filter} format instead of {partition}.>

JSON config in KV should now be interoperable with Go and Java.

[mtmk]

hey @colprog thanks for getting the ball rolling on this one. Please let us know how you get on with the changes and please fell free to review the code if you have some time.

[colprog]

Thanks, The changes looks good.

[jnmoyne]

LGTM (after the changes since my last review)

[jnmoyne]

Actually, as I implemented partitioning on the whole subject in Go I made another correlated improvement that you should also implement in .net to keep interoperability: for elastic simply you can have the "partitioning_filters" array be empty (while currently it will error out if you don't specify at least one partitioning filter), and in that case (the length of the array is 0) the library should simply have a single filter for ">" (and destination "{{Partition(x)}}.>") for the consumer it creates.

[jnmoyne]

See synadia-io/orbit.go#46

[colprog]

Hi @mtmk, any news on this?

[mtmk]

apologies @colprog I have been busy with the client codebase last couple of week. I'll try to give this a priority next week.

[colprog]

Thanks monty, looking forward to the new release.