Skip to content

Add JDBC-based source coordination store plugin#6757

Open
lawofcycles wants to merge 1 commit intoopensearch-project:mainfrom
lawofcycles:feature/jdbc-coordination-store
Open

Add JDBC-based source coordination store plugin#6757
lawofcycles wants to merge 1 commit intoopensearch-project:mainfrom
lawofcycles:feature/jdbc-coordination-store

Conversation

@lawofcycles
Copy link
Copy Markdown
Contributor

Description

Adds a new jdbc source coordination store plugin backed by a relational database.

The plugin uses standard SQL operations to implement the SourceCoordinationStore interface.

Capability Implementation
Conditional create INSERT with duplicate key check
Optimistic locking UPDATE ... WHERE version = ?
Partition acquisition Status-based query with priority ordering
TTL cleanup Scheduled deletion of expired COMPLETED partitions

Configuration example

source_coordination:
 store:
   jdbc:
     url: "jdbc:postgresql://localhost:5432/dataprepper"
     username: "dp_user"
     password: "dp_pass"

Tested with PostgreSQL and MySQL. Integration tests run against PostgreSQL by default and can be configured for MySQL via system properties. Also verified end-to-end with OpenSearch Source and Iceberg Source pipelines on both databases, including TTL-based cleanup of completed partitions.

Issues Resolved

Resolves #6740

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add JDBC-based source coordination store

1 participant