Fix/rewrite jobs fqn locks#2067
Merged
Merged
Conversation
…ymlinked jobs Signed-off-by: Michael Collado <collado.mike@gmail.com>
…othing has changed Signed-off-by: Michael Collado <collado.mike@gmail.com>
wslulciuc
reviewed
Aug 10, 2022
| RETURNING uuid INTO job_uuid; | ||
| IF TG_OP = 'INSERT' OR | ||
| (TG_OP = 'UPDATE' AND OLD.symlink_target_uuid IS DISTINCT FROM NEW.symlink_target_uuid) THEN | ||
| -- update the symlink target if null. otherwise, keep the old value |
Member
There was a problem hiding this comment.
Ok, the comments here are much appreciated ❤️
wslulciuc
approved these changes
Aug 10, 2022
wslulciuc
left a comment
wslulciuc
left a comment
Member
There was a problem hiding this comment.
@collado-mike thanks for the fix, and the write up and tests that went along with the change 💯 🥇
jonathanpmoraes
referenced
this pull request
in nubank/NuMarquez
Feb 6, 2025
* Update rewrite_jobs_fqn_table function to correctly ignore existing symlinked jobs Signed-off-by: Michael Collado <collado.mike@gmail.com> * Update rewrite_jobs_fqn_table to avoid updating jobs_fqn table when nothing has changed Signed-off-by: Michael Collado <collado.mike@gmail.com> Signed-off-by: Michael Collado <collado.mike@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
rewrite_jobs_fqn_tablefunction is inadvertently updating jobs even when no metadata about the job has changed. Under load, this causes significant locking issues, as thejobs_fqntable must be locked for every job update.Solution
I updated the function to only update the table if the job is a new record or if the
symlink_target_uuidis distinct from the previous value. These graphs show the number of blocked transactions and commits over time before and after the change. After applying the change (around 20:10 on these graphs), the number of commits goes way up and the number of active/blocked transactions drops.I also added a test assertion to validate the update of the
jobs_fqntable.Checklist
CHANGELOG.mdwith details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sqldatabase schema migration according to Flyway's naming convention (if relevant)