Skip to content

A36: xDS-Enabled Servers #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ejona86 merged 25 commits into from
Feb 26, 2021
Merged

A36: xDS-Enabled Servers #214

Changes from 5 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b825f33
A36: xDS-Enabled Servers
ejona86 Nov 20, 2020
6eee828
Mostly editorial changes and small tweaks
ejona86 Dec 11, 2020
56ddd35
Language tweaks
ejona86 Dec 22, 2020
dabf9aa
Rework how listener_filters are handled and the impact to Matchers
ejona86 Dec 23, 2020
9b883c5
Swap to server_listener_resource_name_template
ejona86 Dec 23, 2020
a933f33
Revamp waiting-for-xds-config
ejona86 Jan 15, 2021
6fc8390
Explicitly define how to shut down existing connections
ejona86 Jan 15, 2021
c110c45
Grammar nit
ejona86 Jan 15, 2021
a40b6ed
Update the Go section.
easwars Jan 20, 2021
009c014
Comma
ejona86 Jan 20, 2021
0ed8d03
Flesh out implementation plan and mention C++/wrapped languages are s…
ejona86 Jan 20, 2021
fe7cee7
Add discussion link
ejona86 Jan 20, 2021
9bc49f4
Clarify; don't say C will be a separate gRFC; XdsServer will manage t…
ejona86 Jan 22, 2021
743a55a
FilterChainMatch handling is not ordered
ejona86 Feb 4, 2021
55d7a1b
Add link for determining most specific match
ejona86 Feb 4, 2021
6bae4a6
Avoid saying/implying no matcher matches individually.
ejona86 Feb 4, 2021
3d1459a
Use "missing or unknown" instead of "unavailable"
ejona86 Feb 4, 2021
6eff58a
Require filter handling and validation
ejona86 Feb 25, 2021
9b301cc
Make explicit that xds v3 is optional
ejona86 Feb 25, 2021
35286d5
s/addition/additional/;s/second time/more than once/
ejona86 Feb 25, 2021
5c4d52d
If no chain matches, close connection
ejona86 Feb 25, 2021
c8bc62f
s/is missing or unknown/does not exist or has not yet been received/
ejona86 Feb 25, 2021
d0098ea
Validate use_original_dst != true
ejona86 Feb 25, 2021
2ef6bc2
Mark as ready for implementation; sync last updated date
ejona86 Feb 26, 2021
e60b6df
Replace broken link, since this is being merged before A29
ejona86 Feb 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 29 additions & 28 deletions A36-xds-for-servers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,24 +68,25 @@ XdsServerCredentials.

To serve RPCs, the XdsServer must have its xDS configuration, provided via a
Listener resource and potentialy other related resources. When its xDS
configuration is unavailable the server must be in a "not serving" mode. This is
ideally implemented by not `listen()`ing on the port. If that is impractical an
implementation may be `listen()`ing on the port, but it must also `accept()` and
immediately `close()` connections, making sure to not send any data to the
client (e.g., no TLS ServerHello nor HTTP/2 SETTINGS). With either behavior,
client connection attempts will quickly fail and clients would not perform
further attempts without a backoff. Load balancing policies like `pick_first`
would naturally attempt connections to any remaining addresses to quickly find
an operational backend. However, the `accept()`+`listen()` approach will be
improperly detected as server liveness for TCP heath checking.
configuration is missing or unknown the server must be in a "not serving" mode.
Comment thread
ejona86 marked this conversation as resolved.
Outdated
This is ideally implemented by not `listen()`ing on the port. If that is
impractical an implementation may be `listen()`ing on the port, but it must also
`accept()` and immediately `close()` connections, making sure to not send any
data to the client (e.g., no TLS ServerHello nor HTTP/2 SETTINGS). With either
behavior, client connection attempts will quickly fail and clients would not
perform further attempts without a backoff. Load balancing policies like
`pick_first` would naturally attempt connections to any remaining addresses to
quickly find an operational backend. However, the `accept()`+`listen()` approach
will be improperly detected as server liveness for TCP heath checking.

If the xDS bootstrap is missing or invalid, implementations would ideally fail
XdsServer startup, but it is also acceptable to consider it a lack of xDS
configuration and enter a permanent "not serving" mode.

If the server is unable to open serving ports, XdsServer startup may fail or it
may enter "not serving" mode. If entering "not serving" mode, opening the port
must be retried automatically (e.g., retry every minute).
If the server is unable to open serving ports (e.g., because the port is already
in use by another process), XdsServer startup may fail or it may enter "not
serving" mode. If entering "not serving" mode, opening the port must be retried
automatically (e.g., retry every minute).

Communication failures do not impact the XdsServer's xDS configuration; the
XdsServer should continue using the most recent configuration until connectivity

@sanjaypujare sanjaypujare Feb 22, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that true also for the very first time config is needed during start() (in Java) ? What's the expected behavior when the listener resource is missing at startup time? Should the server's start() fail with the server going into permanent failure state? Or should it go into "Not serving" state and loop until the listener resource is available?

@ejona86 ejona86 Feb 25, 2021

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically saying "if an I/O error occurs, continue doing what you were doing before." And that applies to startup, because at startup we are "not serving." No, the server's start() should not fail. That is covered by the "XdsServer's start must not fail due to transient xDS issues, like missing xDS configuration from the xDS server" language below. We absolutely must avoid a permanent failure state triggered by transient I/O failures. We'd stay "not serving" until something improves.

Expand Down Expand Up @@ -137,19 +138,18 @@ The `GRPC_XDS_BOOTSTRAP` file will be enhanced to have a new field:
XdsServer will use the normal `XdsClient` to communicate with the xDS server.
There is no default value for `server_listener_resource_name_template` so if it
Comment thread
ejona86 marked this conversation as resolved.
is not present in the bootstrap then server creation or start will fail or the
XdsServer will become "not serving". The XdsServer will pass the listening
address (commonly using a wildcard IP address, like `::` or `0.0.0.0`) to a new
XdsClient API to start a watch for a `envoy.config.listener.v3.Listener`
resource. XdsClient will perform the `%s` replacement if the token is present
and watch the corresponding listener resource. No special character handling of
XdsServer will become "not serving". XdsServer will perform the `%s` replacement
in the template (if the token is present) to produce a listener name. The
Comment thread
easwars marked this conversation as resolved.
XdsServer will start an XdsClient watch on the listener name for a
`envoy.config.listener.v3.Listener` resource. No special character handling of
Comment thread
ejona86 marked this conversation as resolved.
the template or its replacement is performed. For example, with an address of
`[::]:80` and a template of `grpc/server?xds.resource.listening_address=%s`, the
resource name would be `grpc/server?xds.resource.listening_address=[::]:80`.

To be useful, the xDS-returned Listener must have an
[`address`][Listener.address] that matches the listening address provided. The
Listener's `address` would be a TCP `SocketAddress` with matching `address` and
`port_value`. The XdsClient must be "not serving" if the address does not match.
`port_value`. The XdsServer must be "not serving" if the address does not match.
Comment thread
easwars marked this conversation as resolved.

The xDS client must NACK the Listener resource if `Listener.listener_filters` is
Comment thread
ejona86 marked this conversation as resolved.
non-empty.
Expand All @@ -159,14 +159,15 @@ non-empty.
Although `FilterChain.filters`s will not be used as part of this gRFC, the
Comment thread
ejona86 marked this conversation as resolved.
`FilterChain` contains data that may be used like TLS configuration in
`transport_socket`. When looking for a FilterChain, the standard matching logic
must be used. Each `filter_chain_match` of the repeated
[`filter_chains`][Listener.filter_chains] should be checked for the specific
connection and if none match, the `default_filter_chain` must be used. The
following is a snippet of [all current fields of
must be used. The [most specific `filter_chain_match`][matching logic] of the
repeated [`filter_chains`][Listener.filter_chains] should be found for the
specific connection and if one is not found, the `default_filter_chain` must be
used. The following is a snippet of [all current fields of
Comment thread
ejona86 marked this conversation as resolved.
Outdated
FilterChainMatch][FilterChainMatch], and if they will be handled specially:

[Listener.address]: https://github.com/envoyproxy/envoy/blob/11dee4e9c37c244f9de4bc339fdc8695e5de2c5d/api/envoy/config/listener/v3/listener.proto#L108
[Listener.filter_chains]: https://github.com/envoyproxy/envoy/blob/11dee4e9c37c244f9de4bc339fdc8695e5de2c5d/api/envoy/config/listener/v3/listener.proto#L117
[matching logic]: https://github.com/envoyproxy/envoy/blob/928a62b7a12c4d87ce215a7c4ebd376f69c2e080/api/envoy/config/listener/v3/listener_components.proto#L44
[FilterChainMatch]: https://github.com/envoyproxy/envoy/blob/928a62b7a12c4d87ce215a7c4ebd376f69c2e080/api/envoy/config/listener/v3/listener_components.proto#L85

```
Expand Down Expand Up @@ -197,8 +198,7 @@ on `use_original_dst`. It also applies to `server_names`, `transport_protocol`,

The overall "wrapping" server API has many language-specific ramifications. We
show each individual language's approach. For C++ and wrapped languages we just
show a rough sketch of how they would be done and expect a followup gRFC for
more implementation details.
show a rough sketch of how they would be done.
Comment thread
ejona86 marked this conversation as resolved.

XdsClients may be shared without impacting this design. If shared, any mentions
of "creating" or "shutting down" an XdsClient would simply mean "acquire a
Expand Down Expand Up @@ -390,9 +390,10 @@ waiting on the C core implementation.

* C++/wrapped languages. The surface APIs should be relatively easy compared to
the C core changes. The C core changes are being investigated by @markdroth
and the plan is to cover them in a separate gRFC. So this gRFC just tries to
show that the API design would work for C++/wrapped languages but there is no
associated implementation work.
and the plan is to flesh them out later when we have a better idea of the
appropriate structure. So the gRFC just tries to show that the API design
would work for C++/wrapped languages but there is currently no associated
implementation work.

* Java. Implementation work primarily by @sanjaypujare, along with gRFC A29.
Added classes are in the `grpc-xds` artifact and `io.grpc.xds` package.
Expand Down
⚡