Streams marked orphan and deleted when converting standalone to cluster

[sourabhaggrawal]

Hello Team,
I'm currently running NATS JetStream in standalone mode across multiple datacenters and need to migrate these deployments to cluster mode for high availability.
I had expected that updating the configuration to cluster mode would allow me to:

• Add the cluster configuration block
• Update existing streams from R1 to R3 replicas
• Have the data replicate to the newly added nodes

However, when I restart NATS with the cluster configuration, the existing streams are immediately marked as orphaned and automatically deleted before I have any opportunity to update their replica counts. This results in complete data loss.
Questions:

• Is there a configuration option or flag to prevent automatic orphan stream cleanup during the transition?
• Is there a supported in-place migration path from standalone to clustered mode that preserves existing stream data?
• Can streams be updated from R1 to R3 replicas without recreation?

I'm looking for a solution that doesn't require manual backup and restore operations, as we have deployments across multiple datacenters where this approach would be operationally challenging.
Any guidance on how to achieve this migration while preserving existing data would be greatly appreciated.

[MauriceVanVeen]

Standalone single-server is different from clustered deployments and can't be migrated in place like that.

You'll likely need to take a backup of your streams and then restore them in the clustered setup.

[sourabhaggrawal]

:'(
Is is possible to setup single node cluster may be as a seed node, and new node can connect to it as needed ?

[wallyqs]

a bit involved but either taking backups or think you can try to leafnode connect the standalone server to a cluster and create mirrors of the streams first, then unplug the leafnode standalone server and make the streams stop being mirrors

[sourabhaggrawal]

Thank you both for your help, @MauriceVanVeen and @wallyqs.
Since I don't have control over the Datacenters, implementing this approach would require developing production-grade automation specifically for this use case. I'll explore it, but I'm leaning toward the backup/restore path as the more practical solution.
On a related note, are there any plans to make the standalone implementation extendable to a cluster setup, or would that not be technically feasible?

[MauriceVanVeen]

are there any plans to make the standalone implementation extendable to a cluster setup, or would that not be technically feasible?

This is not planned at the moment.

A clustered setup uses consensus and stores on which servers those streams/consumers are hosted separately. A single-server setup doesn't have/need that. Migrating is perhaps technically possible, but would be tricky to get right. The mirroring approach that Wally mentioned seems like the most "native" way to do this migration with minimal to no downtime.

[tilakraj94]

Hi @MauriceVanVeen,

I’d like to propose adopting orphan streams specially for R1 streams on orphans check while running in cluster mode.

Ideally, a stream created in standalone mode should behave similarly to a backup/restore in a clustered setup. At the very least, the server should attempt to bring up any resources that are still valid.

I’ve put together a small PR for this change and tested it locally. It seems to be working.

Would you be open to reviewing the PR? Since this discussion already was open, I didn’t open a new issue.

[tilakraj94]

Thank you @wallyqs for outlining these edge cases. I’ve reviewed them and also gone through the JetStream cluster logic around stream adoption, including the handling and lifecycle of the PTC marker. I’ll incorporate these guidelines into my implementation to ensure it aligns with the expected scenarios and test cases.

Could you also please confirm if any of this is already in progress? The code you shared looks fairly complete to me, so I wanted to check if there are any remaining gaps I should be aware of.

[wallyqs]

thank you @tilakraj94 no we don't have an item in the roadmap about this, I just wanted to have an idea myself around of the complexity of the changes.

[tilakraj94]

@wallyqs Sounds good, that approach works.

Once the PR is ready, I’ll raise it so the team can review and provide guidance. If everything looks good, we can proceed accordingly. 🙏

[tilakraj94]

@wallyqs #7988 could you or the team please take a look at this PR? I’ve tried to cover all the relevant test cases. Please let me know if any improvements or edge cases are needed. Thanks!

[wallyqs]

thanks @tilakraj94 for spending time on this, we'll need some time to go over the changes