Missing value/null support for scalar value types in proto 3

[lostindark]

From the protobuf wire format we can tell whether a specific field exists or not. And in protobuf 2 generated code, all fields have a "HasXXX" method to tell whether the field exists or not in code. However in proto 3, we lost that ability. E.g. now we can't tell if a int32 field is missing, or has a value of 0 (default for int32 type).

In many scenario, we need the ability to differentiate missing vs default value (basically nullable support for scalar types).

Feng suggest workarounds:

1. Use a wrapper message, such as google.protobuf.Int32Value. In proto3, message fields still have has-bits.
2. Use an oneof. For example:
   message Test1 {
   oneof a_oneof {
   int32 a = 1;
   }
   }
   then you can check test.getAOneofCase().

However, this requires change the message definition, which is a no go if you need to keep the message compatible while upgrade from proto2 to proto 3.

We need to add back the support to detect whether a scalar type field exist or not.

More discussion here: https://groups.google.com/forum/#!topic/protobuf/6eJKPXXoJ88

[ngbrown]

While proto3 mentions using proto2 Message Types

It's possible to import proto2 message types and use them in your proto3 messages, and vice versa. However, proto2 enums cannot be used in proto3 syntax.

It might not be good application design to expect the .proto files and binary format to stay the same after switching to proto3.

Even though it appears that using the wrappers.proto is the way proto3 is addressing nullable fields, and the c#/.net library has convenient nullable formatting for them, they will end up consuming an extra byte per value because of the additional message layer.

[xfxyjwf]

@lostindark, workaround (2) using an oneof is a wire compatible change.

[lostindark]

@lostindark, workaround (2) using an oneof is a wire compatible change.

The issue is it doesn't produce nice code, and it needs box/unbox when use (hurt performance).

[xfxyjwf]

@lostindark what if we optimize that case (one primitive field in an oneof) to eliminate the box/unbox cost?

[lostindark]

@xfxyjwf That seems a good solution with minimum change required. However, I do wish I don't need to name the oneof, like below:
message Test1 { oneof { int32 a = 1; } }

[xfxyjwf]

I am thinking of a dedicated syntax for that. Something like:

message Foo {
  nullable int32 value = 1;
}

which is basicaly a syntax sugar for:

message Foo {
  oneof value_oneof {
    int32 value = 1;
  }
}

I'll bring this up with the team and see how others think about this.

[skorokhod]

@xfxyjwf

message Foo
{
int x = 1;
oneof v1 { int32 value = 1;}
oneof v2 { string value = 1;}
}

gets error on v2: "value" is already defined in "Foo".
Is it possible to have several independent oneofs in one message?
Moreover it's not clear to me which tag is assigned for v1 and v2 fields?

[xfxyjwf]

@skorokhod

Oneof fields are just like regular fields and are in the parent message's naming scope. You need to define your message as:

message Foo {
  int x = 1;
  oneof v1 { int32 value1 = 2; }
  oneof v2 { int32 value2 = 3; }
}

All these fields need different names and field numbers.

[ngbrown]

The C#/.NET library already optimizes for the wrappers.proto case and doesn't box/unbox those nullable values. Looping in @jskeet to ask if it can/should optimize both cases?

Even in my own application which used protobuf2, I relied on the HasXXX functions for field presence. This was used to broadcast updates, only filling in the fields that had changed. Not present didn't mean that it was the default value, but that it hadn't changed from the previous value.

I think though, this was probably viewed as a mistake in the original design/spec since it conflicts with the eliding of fields with default values. Going forward, I don't see many realistic ways to say that v2 and v3 are binary compatible for applications/messages that relied on this behavior.

[jskeet]

@ngbrown: I don't want to start special-casing the situation where there's a oneof with only a single field in it, no. That feels like complexity for little benefit. The way proto3 has been designed to work is that if you want a primitive field with presence, you use the wrapper type. I wouldn't want to subvert that.

[christian-storm]

In struggling (#1655) with this 'bug' I realized that I don't understand the rational behind having default values at all when you can't set the default value or make the field required, i.e., trust that you should use the default value if the field isn't set on the wire. Why not do away with defaults all together? If a field is set put it on the wire even it is a zero length string or a zero. Otherwise one can assume it is null.

This problem is killing me because I have a situation where a int32 field may or may not be set and one of the valid data values is zero. As it stands I have to use oneof or do something hackish like +1 to all values zero or greater before serialization and -1 after parsing the message.

In the current incarnation enums are particularly problematic. Given that "every enum definition must contain a constant that maps to zero as its first element" and fields are inherently optional it is impossible to use it in an application. Was the zeroth element explicitly set before serialization? Was it not set? There should be a big warning- don't use the zeroth element in an enum. You won't know if it has been set or not. It is best to set the first element to something that will never be selected, e.g., NOT_SET = 0;

[christian-storm]

Ran across this in python_message.py and thought it may be pertinent. Just trying to help...

def _AddPropertiesForNonRepeatedScalarField(field, cls):
  """Adds a public property for a nonrepeated, scalar protocol message field.
  Clients can use this property to get and directly set the value of the field.
  Note that when the client sets the value of a field by using this property,
  all necessary "has" bits are set as a side-effect, and we also perform
  type-checking.
 <snip>
  def getter(self):
    # TODO(protobuf-team): This may be broken since there may not be
    # default_value.  Combine with has_default_value somehow.
    return self._fields.get(field, default_value)

[lostindark]

@jskeet The problem with wrapper type is it is not wire format compatible with existing proto 2 message. Assume you want to upgrade to proto 3 for some services in a large project, what is the option here if the project already leverage HasXXX feature in proto 2? Upgrade to a different wire format? The cost might be too high.
I know proto 3 is not 100% compatible to proto 2. But I think it shouldn't break important functionalities that people may frequently depend on. Maybe in proto 3 design the choice was made to get ride of this support for simplicity. However, this break an important feature and makes the upgrade from proto 2 painful.