proto3 and unknown fields

[joshuarubin]

I know that unknown fields have been removed from proto3, but I am trying to get an explanation about why this change was made and if there is any way to replicate that behavior in proto3.

Thanks so much.

referred from golang/protobuf#25

[dhendry]

I too am wondering about this. I am looking into migrating what is essentially a messaging system to gRPC (where proto3 seems to be recommended). In my case, clients send messages (text plus rendering information) to each other via a server where the server needs to understand the text and certain parts of the rendering info. I want to allow client developers to experiment with new features (pre-release) without having to deploy server code for every change.

Essentially, its a case where I want a shared proto definition between the client(s) and server, but dont want to require the server proto definition to be the latest to process requests.

[solicomo]

I'd like to hear about the explanation, too.

The behavior of proto2 makes sense to me.

[jeremyong]

I have a lot of concerns about silently deleting data upon deserialization, to the point that even though we have internally been using proto3 for several months, I am considering changing things back to proto2. This change would be a lot easier to stomach if there was a message option to allow serialization and deserialization of unknown fields instead of discarding them.

[jeremyong]

Being unable to add unknown fields that persist is also unacceptable for us. Reading the code, it's pretty clear the decision to omit unknown fields happens at compile time rather than at runtime (based on the generated code), so it seems proto3 is a no-go. Personally, I very much liked most of the changes to the new version except this one. Changing the default behavior alone might have been ok, especially given that the new behavior is well-documented, but doing so without a way to restore old behavior seems like a misstep. Supporting a plugin that reverts that behavior seems too expensive relative to the cost of just using proto2 with restrictions (optional only, etc).

[dhendry]

Still no answers to this? This is a fundamental issue which is seriously hindering our the adoption of protobuf in many areas.

[jeremyong]

+1 proto2 is a permanent fixture for us. Changing default behavior is one thing but changing it in a way that doesn't let the user even control it is a strict loss in my opinion. What I foresee moving forward is a huge fragmentation in the client ecosystem. Maintaining support for both proto2 and proto3 semantics is too much to chew for most developers, and I'm already seeing some client libraries do this awkward dance where they have some proto2 properties and some proto3 properties. The easiest example of this causing a problem in history is the move from Python2 to Python3. One possible solution might be a file level option that informs the protobuf compiler not to strip unknown fields.

[liujisi]

The proto3 spec doesn't forbid preserving unknown fields. Instead, it allows implementation to choose whether to preserve unknowns. The current C++/Java chose to drop the unknowns though. We are currently looking the issue and will keep this thread posted.

[jeremyong]

Thanks @pherl for providing the update. FWIW, I think it is worth considering how the behavior might be standardized, for the same reason people argue against undefined behavior in C or C++. Undefined behavior (if present) should really be due to a lack of foresight if it exists, but for something like this, we might as well come up with an actual solution since we're already aware of the problem.

[joshuarubin]

Thanks for keeping this issue alive. I'd just like to add that we are interested in support for Go, but that might need to be addressed in golang/protobuf.

[jeremyong]

@pherl Any progress on this front?

[gfecher]

+1 for preserving unknown fields.

I accept that you can not trivially maintain compatibility with the JSON format (at least as long as you want to marshal fields with their names), but I think a lot of shops would be happy to pay this price for not having to release their low-level infrastructure in lock step with their newest clients.

In fact Kenton seems to wonder himself (https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html): Apparently, version 3 of Protocol Buffers, aka “proto3”, removes this feature. I honestly don’t know what they’re thinking. This feature has been absolutely essential in many of Google’s internal systems.

In my opinion the right approach would be to make this an option of the proto compiler on compiling the proto: this way everybody can decide for themselves whether the benefits outweigh the downsides.

For now I have overridden the PreserveUnknownFields function in both cpp_helpers.h and java_helpers.h in the compiler code to always return true and this seems to work, but I would appreciate it if someone from google could confirm.

[xfxyjwf]

Some updates: we tried to gather data to prove "unknown fields are essential for Google systems", but the result is not so convincing (the experiment is done in a Google sub-system, not the whole of Google).

For those of you who are interested in adding back unknown fields in proto3, could you describe your use case in more details and explain why unknown fields is required (e.g., can the same use case be supported using some other proto3 features)? We need to prove unknown fields are needed in some common use cases in order to add it back.

[jeremyong]

Here is a use case I developed internally that makes heavy usage of unknown
fields:

In addition to the message itself, we often annotate the message before
sending it over the wire with metadata indicating if a field was deleted or
not, if it was set to a default field, etc. Internally, we use a diff-ing
scheme to create a protobuf message "diff" which handles maps, fields, and
messages (recursively applied). The application of the diff itself is
associative, so many diffs can accumulate into one, and this makes for a
fairly elegant scheme for updating state for a particular message across
many clients that may or may not be online.

Generalizing this use case, any protobuf message that is derived from the
reflection API must necessarily leverage the unknown field set, since by
definition, we cannot know the shape of the message a priori. Think of this
as a "higher order message" whereas messages that are schema defined are
first order messages.

On Sun, Jun 12, 2016 at 11:30 AM, Feng Xiao notifications@github.com
wrote:

Some updates: we tried to gather data to prove "unknown fields are
essential for Google systems", but the result is not so convincing (the
experiment is done in a Google sub-system, not the whole of Google).

For those of you who are interested in adding back unknown fields in
proto3, could you describe your use case in more details and explain why
unknown fields is required (e.g., can the same use case be supported using
some other proto3 features)? We need to prove unknown fields are needed in
some common use cases in order to add it back.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#272 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAPRJdU9zmn_iHC60rz014oYrvt0n_zQks5qLFBXgaJpZM4D8C3u
.

Jeremy Ong
PlexChat CTO
650.400.6453

[gfecher]

Hi,

We have a use case with a mixture of data validation/data transformation and storage.
Our infrastructure component understands certain bits of the schema that it validates/changes, but it is oblivious to the rest of the payload. It does store it, however, and clients running on the new schema expect the newly introduced fields to be returned intact.

In general any component could benefit from preserving unknown fields where only a partial understanding of the message is needed, especially where the bits the component does care about does not change often, but the rest of the schema does. I can think of routing, storage, certain types of data transformation, etc.

I would be interested in knowing how you managed to solve these use cases (which I'm sure you have internally at google) without preserving unknown fields.