Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Fix Redis reconnection logic#7482

Merged
erikjohnston merged 4 commits intodevelopfrom
erikj/fix_positions_startup
May 13, 2020
Merged

Fix Redis reconnection logic#7482
erikjohnston merged 4 commits intodevelopfrom
erikj/fix_positions_startup

Conversation

@erikjohnston
Copy link
Copy Markdown
Member

@erikjohnston erikjohnston commented May 12, 2020

This does two things:

1. Ensures we only send a REPLICATE when we (re)connect after we successfully subscribe. This ensures we are subscribed to hear responses to REPLICATE (in practice this is unlikely to be an issue as we'll almost certainly finish subscribing by the time other instances have received and responded to the REPLICATE). Actually, this is spurious.
2. Proactively send out POSITION commands (as if we had just received a REPLICATE) when we connect to Redis. This is important as other instances won't notice we've connected to issue a REPLICATE command (unlike for direct TCP connections). This is only currently an issue if master process reconnects without restarting (if it restarts then it won't have written anything and so other instances probably won't have missed anything).

@erikjohnston erikjohnston requested a review from a team May 12, 2020 15:41
@clokep
Copy link
Copy Markdown
Member

clokep commented May 12, 2020

This seems to try to fix the same thing as #7427 (specifically d78265a)?

@erikjohnston erikjohnston force-pushed the erikj/fix_positions_startup branch from dbc964d to 6202254 Compare May 12, 2020 16:06
async def on_REPLICATE(self, conn: AbstractConnection, cmd: ReplicateCommand):
self.send_positions_to_connection(conn)

def send_positions_to_connection(self, conn: AbstractConnection):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to use conn at all. Is that on purpose?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's exciting! It should only be sending stuff to the given connection via conn.send_command but instead is sending it to all connections via self.send_command. It doesn't actually matter that we send POSITION spuriously to random connections, but lets fix it

@erikjohnston erikjohnston merged commit 8ca7961 into develop May 13, 2020
@erikjohnston erikjohnston deleted the erikj/fix_positions_startup branch May 13, 2020 08:57
@richvdh richvdh requested review from richvdh and removed request for richvdh May 14, 2020 16:48
@richvdh
Copy link
Copy Markdown
Member

richvdh commented May 14, 2020

some notes for the record:

I didn't think this was necessary when I did #7427, since each process will send out a REPLICATE on connection, which will get echoed back and will lead to a POSITION. On the other hand that does feel a bit brittle/icky, so maybe this is better.

phil-flex pushed a commit to phil-flex/synapse that referenced this pull request Jun 16, 2020
Proactively send out `POSITION` commands (as if we had just received a `REPLICATE`) when we connect to Redis. This is important as other instances won't notice we've connected to issue a `REPLICATE` command (unlike for direct TCP connections). This is only currently an issue if master process reconnects without restarting (if it restarts then it won't have written anything and so other instances probably won't have missed anything).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants