refactor(codec/unknownproto): use official descriptorpb types, do not rely on gogo code gen types. by testinginprod · Pull Request #18541 · cosmos/cosmos-sdk

testinginprod · 2023-11-22T14:10:50Z

Description

This PR removes de gogo-proto-gen dependency from unknown proto and uses the official descriptor APIs living google.golang.org/protobuf.

This is a required step for a sub-sequent PR which will enable the usage of google.Protobuf.Anys in Txs without them being registered in the interface registry.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

Summary by CodeRabbit

Refactor
- Transitioned to using official protobuf descriptorpb types for enhanced compatibility and future maintenance.
Bug Fixes
- Fixed an issue with the detection of the length of the ed25519 pubkey to prevent potential crashes.
New Features
- Implemented new utility functions to improve the handling of protobuf types and wire encoding.
Documentation
- Updated CHANGELOG to reflect the introduction of descriptorpb types and the removal of deprecated protoc-gen-gogo/descriptor usage.

…rpb types

coderabbitai · 2023-11-22T14:10:57Z

Walkthrough

The changes involve refactoring the codec/unknownproto module to replace the usage of gogo protobuf descriptor types with the official descriptorpb package types from Google's protobuf library. This includes updating import statements, function signatures, and type references throughout the code to align with the descriptorpb types, as well as adding new utility functions to handle wire type conversions and scalar type checks.

Changes

File Path	Change Summary
`codec/unknownproto/unknown_fields.go`	Replaced `gogo` descriptor imports with `descriptorpb`, updated types and function signatures, added new utility functions.
`CHANGELOG.md`	Documented the introduction of `descriptorpb` types for unknown proto handling and other changes.

Assessment against linked issues

Objective	Addressed	Explanation
Refactor `codec/unknownproto` to not rely on `gogo` descriptor internals (#18540)	✅	The changes replace `gogo` types with `descriptorpb` types, which aligns with the objective of the issue.

Related issues (Beta)

#18541: This PR seems to be part of the work mentioned in the changelog and the code changes, addressing the use of descriptorpb types for unknown proto handling.

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

If you reply to a review comment from CodeRabbit, the bot will automatically respond.
To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
Note: Review comments are made on code diffs or files, not on the PR overview.
Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

coderabbitai

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 6ca0b2d and 8ada39e.

Files selected for processing (1)

codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 8

codec/unknownproto/unknown_fields.go (8)

10-19: The import of github.com/cosmos/gogoproto/proto should be removed if it is no longer used after the refactoring to use descriptorpb types.

100-113: The comment on lines 111-113 references github.com/cosmos/gogoproto, which may no longer be relevant if the dependency on gogo protobuf has been removed. Verify if this comment needs updating or removal.

198-257: The checks array has been updated to use descriptorpb.FieldDescriptorProto_Type. Ensure that all references to checks throughout the codebase are updated to use the correct index type (int instead of protowire.Type).

260-265: The canEncodeType function signature has been updated to use int for wireType. Ensure that all calls to this function are updated accordingly.

364-370: The extractFileDescMessageDesc function is using protov2.Unmarshal which is correct as per the refactor. However, ensure that the protoBlob is correctly formatted for descriptorpb.FileDescriptorProto before unmarshalling.

381-384: The descriptorMatch struct has been updated to use descriptorpb.FieldDescriptorProto. Ensure that all usages of descriptorMatch are consistent with this change.

440-483: The findWireTypeFromFieldDescriptorProtoType function should be reviewed to ensure that it returns the correct protowire.Type for each descriptorpb.FieldDescriptorProto_Type. This is critical as the function is used to determine the expected wire type for a given field descriptor type.

486-509: The isScalar function checks if a field is a scalar type. Ensure that the function's logic is still correct after the refactor and that it aligns with the updated descriptorpb.FieldDescriptorProto_Type values.

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 8ada39e and bf53128.

Files selected for processing (1)

CHANGELOG.md (1 hunks)

Additional comments: 1

CHANGELOG.md (1)

67-67:
The changelog entry is clear and follows the repository's format for documenting changes.

coderabbitai

Review Status

Actionable comments generated: 2

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between bf53128 and 1dfa602.

Files selected for processing (1)

codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 7

codec/unknownproto/unknown_fields.go (7)

13-17: The import of github.com/cosmos/gogoproto/proto should be removed if it's no longer used after the refactor to use descriptorpb.

101-114: The comment on lines 112-113 seems to be outdated and should be updated or removed if it no longer applies to the current implementation.

199-258: The checks array is a good optimization for constant time lookups. Ensure that all wire types and descriptor types are correctly mapped and that no types are missing.

365-371: Ensure that the error from gzip.NewReader and io.ReadAll are handled properly and that the resources are closed appropriately to avoid resource leaks.

380-385: The descriptorMatch struct is used for caching, ensure that the cache is properly synchronized to prevent data races.

407-413: > Note: This review was outside of the patch, so it was mapped to the patch with the greatest overlap. Original lines [392-413]

When caching descriptor information in descprotoCache, ensure that the cache is invalidated or updated when descriptors change to prevent stale data issues.

487-510: The isScalar function correctly identifies scalar types. Ensure that this function is used consistently throughout the codebase to determine scalar types.

coderabbitai · 2023-11-22T14:55:54Z

 					Type:         reflect.ValueOf(msg).Type().String(),
 					TagNum:       tagNum,
 					GotWireType:  wireType,
-					WantWireType: protowire.Type(fieldDescProto.WireType()),
+					WantWireType: findWireTypeFromFieldDescriptorProtoType(fieldDescProto.GetType()),
 				}


The error struct errMismatchedWireType should include the expected and actual wire types using descriptorpb.FieldDescriptorProto_Type instead of protowire.Type to align with the refactor.

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 1dfa602 and 67b3b8a.

Files selected for processing (1)

CHANGELOG.md (1 hunks)

Additional comments: 1

CHANGELOG.md (1)

68-68: The changelog has been updated correctly to reflect the changes made in pull requests #18541 and #18506.

odeke-em

Thank you for this change @testinginprod. I've added some comments. Although what's the rationale behind this change? Is it to get rid of the protogen-gogo-descriptor package? If it is, I believe we forked an entire repository for which it'll still exist. Thank you

testinginprod · 2023-11-29T10:35:59Z

@odeke-em thank you for the review.

This is a preliminary step PR to make a more significant change to unknownproto, specifically unknonwproto currently does not allow google.Protobuf.Any to be passed into a TX unless they've been registered in interface registry.

This is not desirable for two reasons:

there's a use-case for having non-interface used google.Protobuf.Anys in the TX body.
we have the first use case for the x/accounts module.
this is also the default behaviour implemented by x/tx which is meant to replace codec/unknownproto: https://github.com/cosmos/cosmos-sdk/blob/main/x/tx/decode/unknown.go (note: we want to have this in the legacy baseapp, x/tx will be used by the new serverv2).

Alongside enabling support for having non-interface-registry anys in TXs, we want to also be able to decode anys which have a protov2 counterparty but not a gogoproto one. Using default descriptorpb types makes this transition a lot more effortless otherwise we'd be forced to convert from gogo-protoc-gen.Descriptor into descriptorp.Descriptor .

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 67b3b8a and e4f6e44.

Files selected for processing (1)

codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 11

codec/unknownproto/unknown_fields.go (11)

12-20: The import changes align with the summary and the refactoring goal of using descriptorpb types from the official protobuf library.

69-72: The update to use fieldTypeToProtowireType for determining WantWireType is consistent with the refactoring goal and should ensure correct wire type matching.

104-104: The update to use isScalar for checking if a field is scalar aligns with the refactoring goal and the summary.

199-266: The refactoring of checks to use descriptorpb.FieldDescriptorProto_Type is consistent with the goal of using descriptorpb types.

333-335: The update of the global variable protoFileToDesc to use descriptorpb.FileDescriptorProto is consistent with the refactoring goal.

337-342: The update of the unnestDesc function signature to use descriptorpb.DescriptorProto is consistent with the refactoring goal.

347-350: The update of the extractFileDescMessageDesc function signature to use descriptorpb types is consistent with the refactoring goal.

393-396: The update of the getDescriptorInfo function signature to use descriptorpb types is consistent with the refactoring goal.

441-442: The DefaultAnyResolver.Resolve function signature appears unchanged, which seems to be a no-op. Please verify if this was intentional or if changes were expected but not made.

445-504: The addition of the fieldTypeToProtowireType and isScalar functions is consistent with the refactoring goal and the summary.

480-481: The citation for the isScalar function is correctly included, addressing the previous review comment.

coderabbitai

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between e4f6e44 and a0e9770.

Files selected for processing (2)

CHANGELOG.md (1 hunks)
codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 14

codec/unknownproto/unknown_fields.go (14)

12-20: The import changes align with the summary, replacing github.com/cosmos/gogoproto/protoc-gen-gogo/descriptor with protov2 "google.golang.org/protobuf/proto" and "google.golang.org/protobuf/types/descriptorpb".

69-72: The use of fieldTypeToProtowireType in the errMismatchedWireType struct instantiation is consistent with the summary.

104-104: The use of isScalar in the condition within RejectUnknownFields is consistent with the summary.

199-258: The constants and maps have been correctly updated to use descriptorpb.FieldDescriptorProto.

199-258: The protoMessageForTypeName function has been updated to return an error, aligning with the summary.

261-266: The canEncodeType function has been updated to take descriptorpb.FieldDescriptorProto_Type, which is consistent with the summary.

337-342: The unnestDesc function has been updated to take descriptorpb.DescriptorProto, which is consistent with the summary.

365-372: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [347-372]

The extractFileDescMessageDesc function has been updated to return descriptorpb.FileDescriptorProto and descriptorpb.DescriptorProto, which is consistent with the summary.

383-384: The cache field in the descriptorMatch struct has been updated to use descriptorpb.FieldDescriptorProto, which is consistent with the summary.

393-396: The getDescriptorInfo function has been updated to return map[int32]*descriptorpb.FieldDescriptorProto and *descriptorpb.DescriptorProto, which is consistent with the summary.

333-334: The protoFileToDesc map has been updated to use descriptorpb.FileDescriptorProto, which is consistent with the summary.

441-442: The Resolve method in DefaultAnyResolver has been updated to return proto.Message and an error, which is consistent with the summary.

445-477: The new function fieldTypeToProtowireType() has been added, aligning with the summary.

479-505: The new function isScalar() has been added, aligning with the summary. However, the comment indicates that the code was copied from gogo/protobuf, which matches the existing comment from a previous review. This should be verified to ensure proper attribution is given.

odeke-em

Thank you @testinginprod, LGTM with some comments.

testinginprod · 2023-11-29T11:43:52Z

thank you @odeke-em, reviews have been addressed.

odeke-em · 2023-11-29T11:44:25Z

Awesome, thank you and LGTM++

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between a0e9770 and 80211c4.

Files selected for processing (1)

codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 14

codec/unknownproto/unknown_fields.go (14)

13-17: The import changes are consistent with the summary and reflect the transition to official descriptorpb types.

69-72: The replacement of protowire.Type(fieldDescProto.WireType()) with toProtowireType(fieldDescProto.GetType()) is consistent with the summary.

104-104: The replacement of fieldDescProto.IsScalar() with isScalar(fieldDescProto) is consistent with the summary.

202-258: The constants and maps have been correctly updated to use descriptorpb.FieldDescriptorProto.

263-266: The function signature for canEncodeType has been correctly updated to use descriptorpb.FieldDescriptorProto_Type.

337-342: The function signature for unnestDesc has been correctly updated to use descriptorpb.DescriptorProto.

365-372: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [347-372]

The function extractFileDescMessageDesc has been correctly updated to return descriptorpb types.

382-384: The type descriptorMatch has been correctly updated to use descriptorpb types.

407-413: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [392-413]

The function getDescriptorInfo has been correctly updated to return descriptorpb types.

333-333: The map protoFileToDesc has been correctly updated to use descriptorpb.FileDescriptorProto.

445-480: The function toProtowireType has been correctly added to the codebase.

485-507: The function isScalar has been correctly added to the codebase.

484-484: The citation for the isScalar function has been added as requested in a previous review.

459-459: The code has appropriate spacing between sections, which should aid in readability as suggested in a previous review.

odeke-em · 2023-11-29T12:09:05Z

@testinginprod please update with the latest from main.

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 80211c4 and bd6d723.

Files selected for processing (1)

codec/unknownproto/unknown_fields.go (10 hunks)

Additional comments: 12

codec/unknownproto/unknown_fields.go (12)

13-17: The import changes are correctly reflected in the code, aligning with the summary.

69-74: The update to use toProtowireType in the RejectUnknownFields function is correctly implemented.

105-115: The update to use isScalar in the RejectUnknownFields function is correctly implemented.

200-267: The constants and maps related to field types have been updated correctly, reflecting the changes from descriptor.FieldDescriptorProto_Type to descriptorpb.FieldDescriptorProto_Type.

264-267: The function canEncodeType has been correctly updated to take descType descriptorpb.FieldDescriptorProto_Type.

334-345: The map protoFileToDesc has been correctly updated to use descriptorpb.FileDescriptorProto.

338-343: The function unnestDesc has been correctly updated to take mdescs []*descriptorpb.DescriptorProto.

366-373: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [348-373]

The function extractFileDescMessageDesc has been correctly updated to return *descriptorpb.FileDescriptorProto and *descriptorpb.DescriptorProto.

383-385: The type descriptorMatch has been correctly updated to use *descriptorpb.FieldDescriptorProto and *descriptorpb.DescriptorProto.

394-397: The function getDescriptorInfo has been correctly updated to return map[int32]*descriptorpb.FieldDescriptorProto and *descriptorpb.DescriptorProto.

447-480: The function toProtowireType has been correctly added to convert descriptorpb.FieldDescriptorProto_Type to protowire.Type.

486-508: The function isScalar has been correctly added to determine whether a field is a scalar type.

julienrbrt

lgtm!

… rely on gogo code gen types. (#18541) Co-authored-by: unknown unknown <unknown@unknown> (cherry picked from commit ae7288b) # Conflicts: # CHANGELOG.md

… rely on gogo code gen types. (backport #18541) (#18593) Co-authored-by: testinginprod <98415576+testinginprod@users.noreply.github.com>

make unknown proto not use gogo code gen types, it now uses descripto…

8ada39e

…rpb types

testinginprod requested a review from a team November 22, 2023 14:10

coderabbitai Bot reviewed Nov 22, 2023

View reviewed changes

chore: CHANGELOG.md

bf53128

coderabbitai Bot reviewed Nov 22, 2023

View reviewed changes

testinginprod added the backport/v0.50.x label Nov 22, 2023

lint

1dfa602

coderabbitai Bot reviewed Nov 22, 2023

View reviewed changes

tac0turtle assigned kocubinski and alexanderbez Nov 26, 2023

Merge branch 'main' into tip/baseapp/use_x_tx

67b3b8a

coderabbitai Bot reviewed Nov 27, 2023

View reviewed changes

odeke-em suggested changes Nov 28, 2023

View reviewed changes

Comment thread codec/unknownproto/unknown_fields.go

Comment thread codec/unknownproto/unknown_fields.go Outdated

Comment thread CHANGELOG.md Outdated

unknown unknown added 2 commits November 29, 2023 11:46

address PR reviews

e4f6e44

nicer panic debugging

b663898

coderabbitai Bot reviewed Nov 29, 2023

View reviewed changes

more descriptive CHANGELOG.md

a0e9770

testinginprod requested a review from odeke-em November 29, 2023 10:50

coderabbitai Bot reviewed Nov 29, 2023

View reviewed changes

Comment thread CHANGELOG.md

odeke-em approved these changes Nov 29, 2023

View reviewed changes

Comment thread codec/unknownproto/unknown_fields.go Outdated

Comment thread codec/unknownproto/unknown_fields.go Outdated

Comment thread codec/unknownproto/unknown_fields.go

address PR reviews 2

80211c4

coderabbitai Bot reviewed Nov 29, 2023

View reviewed changes

odeke-em reviewed Nov 29, 2023

View reviewed changes

Comment thread codec/unknownproto/unknown_fields.go

add spacing in default case

bd6d723

coderabbitai Bot reviewed Nov 29, 2023

View reviewed changes

julienrbrt approved these changes Nov 29, 2023

View reviewed changes

testinginprod added this pull request to the merge queue Nov 29, 2023

Merged via the queue into main with commit ae7288b Nov 29, 2023

testinginprod deleted the tip/baseapp/use_x_tx branch November 29, 2023 15:36

mergify Bot mentioned this pull request Nov 29, 2023

refactor(codec/unknownproto): use official descriptorpb types, do not rely on gogo code gen types. (backport #18541) #18593

Merged

testinginprod mentioned this pull request Dec 19, 2023

Epic: x/accounts #17786

Closed

44 tasks

faddat mentioned this pull request Nov 8, 2024

faddat/many times #22476

Closed

12 tasks

Conversation

testinginprod commented Nov 22, 2023 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Author Checklist

Reviewers Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Assessment against linked issues

Related issues (Beta)

Chat with CodeRabbit Bot (@coderabbitai)

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Nov 22, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

odeke-em left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

testinginprod commented Nov 29, 2023

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

odeke-em left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

testinginprod commented Nov 29, 2023

Uh oh!

odeke-em commented Nov 29, 2023

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

odeke-em commented Nov 29, 2023

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

julienrbrt left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

testinginprod commented Nov 22, 2023 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Nov 22, 2023 •

edited

Loading

Chat with CodeRabbit Bot (`@coderabbitai`)

CodeRabbit Configration File (`.coderabbit.yaml`)