Experimental support for incremental delivery (@defer/@stream) (#6827)

Also provides improved compatibility with the graphql-over-http spec by
supporting `content-type: application/graphql-response+json` if requested via
`accept` header, and adds `; charset=utf-8` to content-types.

Executing `@defer` and `@stream` directives requires you to install pre-release
of `graphql@17` in your server. This PR does not update `package.json` to
install that prerelease (nor does it even broaden peer deps, which would be
inadequate as many of its dependencies have peer deps that won't include v17).
However, it does add a new CI step that installs a particular v17 pre-release
and runs the test suite and smoke test (including running some
otherwise-disabled tests that exercise the execution of these directives). We
also add a new `__testing_incrementalExecutionResults` option that lets us test
transport-level behavior without installing the prerelease.

This change reworks `GraphQLResponse` and `HTTPGraphQLResponse` to allow
responses to be single- or multi-part. `GraphQLResponse` had previously (in v4)
moved most of its fields onto `result`; we now instead of `body` with two
`kind`s determining the structure of the rest of the response.
`HTTPGraphQLResponse` (new in v4) had tried to anticipate this change, but now
the structure is a bit different.

A few other changes were made for compatibility with `graphql@17` such as
removing some uses of the non-options multi-argument GraphQLError constructor.

Add two new plugin APIs: `didEncounterSubsequentErrors` and
`willSendSubsequentPayload`.

Updates to plugins:
- Usage reporting waits until all payloads are ready before it's done with
  a given operation.
- The response cache does not cache incremental responses (although that would
  likely be quite helpful).
- No cache-control HTTP headers are written with incremental responses (since we
  don't know all the fields that will be executed yet).
- Inline traces are not added to incremental delivery responses (though it might
  make sense to add them to the last payload or something).

Fixes #6671.

Author: glasser

.changeset/chilled-cows-drum.md

@@ -0,0 +1,7 @@
+---
+'@apollo/server-integration-testsuite': patch
+'@apollo/server-plugin-response-cache': patch
+'@apollo/server': patch
+---
+
+Experimental support for incremental delivery (`@defer`/`@stream`) when combined with a prerelease of `graphql-js`.

.changeset/thirty-donkeys-sing.md

@@ -0,0 +1,8 @@
+---
+'@apollo/server-integration-testsuite': patch
+'@apollo/server-plugin-response-cache': patch
+'@apollo/server': patch
+---
+
+Support application/graphql-response+json content-type if requested via Accept header, as per graphql-over-http spec.
+Include `charset=utf-8` in content-type headers.

.circleci/config.yml

@@ -69,6 +69,22 @@ jobs:
       - setup-node
       - run: npm run test:smoke
 
+  Full incremental delivery tests with graphql-js 17 canary:
+    docker:
+    - image: cimg/base:stable
+    environment:
+      INCREMENTAL_DELIVERY_TESTS_ENABLED: t
+    steps:
+      - setup-node:
+          node-version: "18"
+      # Install a prerelease of graphql-js 17 with incremental delivery support.
+      # --legacy-peer-deps because nothing expects v17 yet.
+      # --no-engine-strict because Node v18 doesn't match the engines fields
+      # on some old stuff.
+      - run: npm i --legacy-peer-deps --no-engine-strict graphql@17.0.0-alpha.1.canary.pr.3361.04ab27334641e170ce0e05bc927b972991953882
+      - run: npm run test:ci
+      - run: npm run test:smoke
+
   Prettier:
     docker:
     - image: cimg/base:stable
@@ -143,4 +159,5 @@ workflows:
       - ESLint
       - Spell check
       - Smoke test built package
+      - Full incremental delivery tests with graphql-js 17 canary
       - Changesets

cspell-dict.txt

@@ -78,6 +78,7 @@ goofql
 graphiql
 graphqlcodegenerator
 GraphQLJSON
+gzipped
 hackily
 herokuapp
 Hofmann

docs/source/api/apollo-server.mdx

@@ -589,7 +589,7 @@ In some circumstances, Apollo Server calls `stop` automatically when the process
 The async `executeOperation` method is used primarily for [testing GraphQL operations](../testing/testing/#testing-using-executeoperation) through Apollo Server's request pipeline _without_ sending an HTTP request.
 
 ```js
-const result = await server.executeOperation({
+const response = await server.executeOperation({
   query: 'query SayHelloWorld($name: String) { hello(name: $name) }',
   variables: { name: 'world' },
 });
@@ -601,6 +601,12 @@ The `executeOperation` method takes two arguments:
   - Supported fields are listed in the table below.
 - The second optional argument is used as the operation's [context value](../data/resolvers/#the-context-argument). Note, this argument is only optional if your server _doesn't_ expect a context value (i.e., your server uses the default context because you didn't explicitly provide another one).
 
+The `response` object returned from `executeOperation` is a `GraphQLResponse`, which has `body` and `http` fields.
+
+Apollo Server 4 supports incremental delivery directives such as `@defer` and `@stream` (when combined with an appropriate version of `graphql-js`), and so the structure of `response.body` can represent either a single result or multiple results. `response.body.kind` is either `'single'` or `'incremental'`. If it is `'single'`, then incremental delivery has not been used, and `response.body.singleResult` is an object with `data`, `errors`, and `extensions` fields. If it is `'incremental'`, then `response.body.initialResult` is the initial result of the operation, and `response.body.subsequentResults` is an async iterator that will yield subsequent results. (The precise structure of `initialResult` and `subsequentResults` is defined by `graphql-js` and may change between the current pre-release of `graphql-js` v17 and its final release; if you write code that processes these values before `graphql-js` v17 has been released you may have to adapt it when the API is finalized.)
+
+The `http` field contains an optional numeric `status` code and a `headers` map specifying any HTTP status code and headers that should be set.
+
 Below are the available fields for the first argument of `executeOperation`:
 
 ### Fields

docs/source/data/errors.mdx

@@ -660,7 +660,8 @@ const setHttpPlugin = {
     return {
       async willSendResponse({ response }) {
         response.http.headers.set('custom-header', 'hello');
-        if (response?.result?.errors?.[0]?.extensions?.code === 'TEAPOT') {
+        if (response.body.kind === 'single' &&
+            response.body.singleResult.errors?.[0]?.extensions?.code === 'TEAPOT') {
           response.http.status = 418;
         }
       },

docs/source/integrations/building-integrations.md

@@ -20,7 +20,7 @@ requests and responses between a web framework's native format to the format use
 
 > For more examples, see these Apollo Server 4 [integrations demos for Fastify and Lambda](https://github.com/apollographql/server-v4-integration-demos/tree/main/packages).
 
-If you are building a serverless integration, we **strongly recommend** prepending your function name with the word `start` (e.g., `startServerAndCreateLambdaHandler(server)`). This naming convention helps maintain Apollo Server's standard that every server uses a function or method whose name contains the word `start` (such as `startStandaloneServer(server)`. 
+If you are building a serverless integration, we **strongly recommend** prepending your function name with the word `start` (e.g., `startServerAndCreateLambdaHandler(server)`). This naming convention helps maintain Apollo Server's standard that every server uses a function or method whose name contains the word `start` (such as `startStandaloneServer(server)`.
 
 ### Main function signature
 
@@ -225,30 +225,40 @@ interface HTTPGraphQLHead {
   headers: Map<string, string>;
 }
 
-type HTTPGraphQLResponse = HTTPGraphQLHead &
-  (
-    | {
-        completeBody: string;
-        bodyChunks: null;
-      }
-    | {
-        completeBody: null;
-        bodyChunks: AsyncIterableIterator<HTTPGraphQLResponseChunk>;
-      }
-  );
+type HTTPGraphQLResponseBody =
+  | { kind: 'complete'; string: string }
+  | { kind: 'chunked'; asyncIterator: AsyncIterableIterator<string> };
+
+
+type HTTPGraphQLResponse = HTTPGraphQLHead & {
+  body: HTTPGraphQLResponseBody;
+};
 ```
 
-The Express implementation uses the `res` object to update the response
-with the appropriate status code and headers, and finally sends the body:
+Note that a body can either be "complete" (a complete response that can be sent immediately with a `content-length` header), or "chunked", in which case the integration should read from the async iterator and send each chunk one at a time. This typically will use `transfer-encoding: chunked`, though your web framework may handle that for you automatically. If your web environment does not support streaming responses (as in some serverless function environments like AWS Lambda), you can return an error response if a chunked body is received.
+
+The Express implementation uses the `res` object to update the response with the appropriate status code and headers, and finally sends the body. Note that in Express, `res.send` will send a complete body (including calculating the `content-length` header), and `res.write` will use `transfer-encoding: chunked`. Express does not have a built-in "flush" method, but the popular `compression` middleware (which supports `accept-encoding: gzip` and similar headers) adds a `flush` method to the response; since response compression typically buffers output until a certain block size it hit, you should ensure that your integration works with your web framework's response compression feature.
 
 ```ts
 for (const [key, value] of httpGraphQLResponse.headers) {
   res.setHeader(key, value);
 }
 res.statusCode = httpGraphQLResponse.status || 200;
-res.send(httpGraphQLResponse.completeBody);
+
+if (httpGraphQLResponse.body.kind === 'complete') {
+  res.send(httpGraphQLResponse.body.string);
+  return;
+}
+
+for await (const chunk of httpGraphQLResponse.body.asyncIterator) {
+  res.write(chunk);
+  if (typeof (res as any).flush === 'function') {
+    (res as any).flush();
+  }
+}
+res.end();
 ```
 
 ## Additional resources
 
-The [`@apollo/server-integration-testsuite`](https://www.npmjs.com/package/@apollo/server-integration-testsuite) provides a set of Jest tests for authors looking to test their Apollo Server integrations. 
+The [`@apollo/server-integration-testsuite`](https://www.npmjs.com/package/@apollo/server-integration-testsuite) provides a set of Jest tests for authors looking to test their Apollo Server integrations.

docs/source/integrations/plugins-event-reference.mdx

@@ -370,7 +370,7 @@ executionDidStart?(
 ): Promise<GraphQLRequestExecutionListener | void>;
 ```
 
-`executionDidStart` may return an object with one or both of the methods `executionDidEnd` and `willResolveField`.  `executionDidEnd` is treated like an end hook: it is called after execution with any errors that occurred. `willResolveField` is documented in the next section.
+`executionDidStart` may return an object with one or both of the methods `executionDidEnd` and `willResolveField`.  `executionDidEnd` is treated like an end hook: it is called after execution with any errors that occurred. (If the operation uses [incremental delivery](../workflow/requests#incremental-delivery-experimental) directives such as `@defer`, `executionDidEnd` is called when the fields required to fill the *initial* payload have finished executing; you can use `willSendSubsequentPayload` to hook into the end of execution for each subsequent payload.) `willResolveField` is documented in the next section.
 
 ### `willResolveField`
 
@@ -424,7 +424,9 @@ const server = new ApolloServer({
 ### `didEncounterErrors`
 
 The `didEncounterErrors` event fires when Apollo Server encounters errors while
-parsing, validating, or executing a GraphQL operation.
+parsing, validating, or executing a GraphQL operation. The errors are available on `requestContext.errors`.
+
+(If the operation uses [incremental delivery](../workflow/requests#incremental-delivery-experimental) directives such as `@defer`, `didEncounterErrors` is only called when errors that will be sent in the *initial* payload are encountered; you can use `didEncounterSubsequentErrors` to find out if more errors are found later.)
 
 ```ts
 didEncounterErrors?(
@@ -434,16 +436,42 @@ didEncounterErrors?(
 ): Promise<void>;
 ```
 
+### `didEncounterSubsequentErrors`
+
+The `didEncounterSubsequentErrors` event only fires for operations that use [incremental delivery](../workflow/requests#incremental-delivery-experimental) directives such as `@defer`. This hook is called when any execution errors are encountered *after* the initial payload is sent; `didEncounterErrors` is *not* called in this case. The errors in question are provided as the second argument to the hook (*not* as `requestContext.errors`, which will continue to be the list of errors sent in the initial payload).
+
+```ts
+didEncounterSubsequentErrors?(
+  requestContext: GraphQLRequestContextDidEncounterSubsequentErrors<TContext>,
+  errors: ReadonlyArray<GraphQLError>,
+): Promise<void>;
+```
+
+
+
 ### `willSendResponse`
 
 The `willSendResponse` event fires whenever Apollo Server is about to send a response
 for a GraphQL operation. This event fires (and Apollo Server sends a response) even
 if the GraphQL operation encounters one or more errors.
 
+(If the operation uses [incremental delivery](../workflow/requests#incremental-delivery-experimental) directives such as `@defer`, `willSendResponse` is called before the *initial* payload is sent; you can use `willSendSubsequentPayload` to find out when more payloads will be sent.)
+
 ```ts
 willSendResponse?(
   requestContext: WithRequired<
     GraphQLRequestContext<TContext>, 'source' | 'queryHash'
   >,
 ): Promise<void>;
 ```
+
+### `willSendSubsequentPayload`
+
+The `willSendSubsequentPayload` event only fires for operations that use [incremental delivery](../workflow/requests#incremental-delivery-experimental) directives such as `@defer`. This hook is called before each payload after the initial one is sent, similarly to `willSendResponse`. The payload in question is provided as the second argument to the hook (*not* on `requestContext`). If this is the last payload, `payload.hasNext` will be false. Note that the precise format of `payload` is determined by the `graphql-js` project, and incremental delivery support has not yet (as of September 2022) been released in an official release of `graphql-js`. When the official release (expected to be `graphql@17`) is released, the format of this argument may potentially change; in this case, Apollo Server may change the precise details of this hook in a backward-incompatible way in a minor release of Apollo Server. (For now, this hook can only be called if you install a pre-release of `graphql@17`.)
+
+```ts
+willSendSubsequentPayload?(
+  requestContext: GraphQLRequestContextWillSendSubsequentPayload<TContext>,
+  payload: GraphQLExperimentalFormattedSubsequentIncrementalExecutionResult,
+): Promise<void>;
+```

docs/source/migration.mdx

@@ -298,7 +298,7 @@ In Apollo Server 3, the `apollo-server-core` package exports built-in plugins, l
 
 In Apollo Server 4, these built-in plugins are part of the main `@apollo/server` package, which also imports the `ApolloServer` class. The  `@apollo/server` package exports these built-in plugins with deep exports. This means you use deep imports for each built-in plugin, enabling you to evaluate only the plugin you use in your app and making it easier for bundlers to eliminate unused code.
 
-There's one exception: the `ApolloServerPluginLandingPageGraphQLPlayground` plugin is now in its own package `@apollo/server-plugin-landing-page-graphql-playground`, which you can install separately. 
+There's one exception: the `ApolloServerPluginLandingPageGraphQLPlayground` plugin is now in its own package `@apollo/server-plugin-landing-page-graphql-playground`, which you can install separately.
 
 This plugin installs the [unmaintained](https://github.com/graphql/graphql-playground/issues/1143) GraphQL Playground project as a landing page and is provided for compatibility with Apollo Server 2. This package will **not** be supported after Apollo Server 4 is released. We strongly recommend you switch to Apollo Server's 4's [default landing page](./api/plugin/landing-pages), which installs the actively maintained Apollo Sandbox.
 
@@ -727,11 +727,12 @@ new ApolloServer<MyContext>({
         return {
           async willSendResponse(requestContext) {
             const { response } = requestContext;
-            // Augment response with an extension, as long
-            // as the operation actually executed.
-            if ('data' in response.result) {
-              response.result.extensions = {
-                ...(response.result.extensions),
+            // Augment response with an extension, as long as the operation
+            // actually executed. (The `kind` check allows you to handle
+            // incremental delivery responses specially.)
+            if (response.body.kind === 'single' && 'data' in response.body.singleResult) {
+              response.body.singleResult.extensions = {
+                ...response.body.singleResult.extensions,
                 hello: 'world',
               };
             }
@@ -1112,6 +1113,8 @@ In Apollo Server 3, you can indirectly specify an operation's context value by p
 
 In Apollo Server 4, the `executeOperation` method optionally receives a context value directly, bypassing your `context` function. If you want to test the behavior of your `context` function, we recommend running actual HTTP requests against your server.
 
+Additionally, the [structure of the returned `GraphQLResponse` has changed](#graphqlresponse), as described below.
+
 So a test for Apollo Server 3 that looks like this:
 
 <MultiCodeBlock>
@@ -1127,7 +1130,7 @@ const server = new ApolloServer({
   context: async ({ req }) => ({ name: req.headers.name }),
 });
 
-const { result } = await server.executeOperation({
+const result = await server.executeOperation({
   query: 'query helloContext { hello }',
 }, {
   // A half-hearted attempt at making something vaguely like an express.Request,
@@ -1158,13 +1161,17 @@ const server = new ApolloServer<MyContext>({
   },
 });
 
-const { result } = await server.executeOperation({
+const { body } = await server.executeOperation({
   query: 'query helloContext { hello }',
 }, {
   name: 'world',
 });
 
-expect(result.data?.hello).toBe('Hello world!'); // -> true
+// Note the use of Node's assert rather than Jest's expect; if using
+// TypeScript, `assert` will appropriately narrow the type of `body`
+// and `expect` will not.
+assert(body.kind === 'single');
+expect(body.singleResult.data?.hello).toBe('Hello world!'); // -> true
 ```
 
 </MultiCodeBlock>
@@ -1425,23 +1432,16 @@ Specifically, the `http` field is now an `HTTPGraphQLRequest` type instead of a
 
 Apollo Server 4 refactors the [`GraphQLResponse` object](https://github.com/apollographql/apollo-server/blob/version-4/packages/server/src/externalTypes/graphql.ts#L25), which is available to plugins as `requestContext.response` and is the type `server.executeOperation` returns.
 
-The `data`, `errors`, and `extensions` fields are now nested within an object returned by the `result` field:
+In Apollo Server 3, the  `data`, `errors`, and `extensions` fields existed at the top level, right beside `http`.
 
-```ts disableCopy
-export interface GraphQLResponse {
-  // The below result field contains an object with the
-  //  data, errors, and extensions fields
-  result: FormattedExecutionResult;
-  http: HTTPGraphQLHead;
-}
-```
+Because Apollo Server 4 supports incremental delivery directives such as `@defer` and `@stream` (when combined with an appropriate version of `graphql-js`), the structure of the response can now represent either a single result or multiple results, so these fields no longer exist at the top level of `GraphQLResponse`.
+
+Instead, there is a `body` field at the top level of `GraphQLResponse`. `response.body.kind` is either `'single'` or `'incremental'`. If it is `'single'`, then incremental delivery has not been used, and `response.body.singleResult` is an object with `data`, `errors`, and `extensions` fields. If it is `'incremental'`, then `response.body.initialResult` is the initial result of the operation, and `response.body.subsequentResults` is an async iterator that will yield subsequent results. (The precise structure of `initialResult` and `subsequentResults` is defined by `graphql-js` and may change between the current pre-release of `graphql-js` v17 and its final release; if you write code that processes these values before `graphql-js` v17 has been released you may have to adapt it when the API is finalized.)
 
 Additionally, the  `data` and `extensions` fields are both type `Record<string, unknown>`, rather than `Record<string, any>`.
 
 The value of `http.headers` is now a `Map` rather than a Fetch API `Headers` object. All keys in this map must be lower-case; if you insert any header name with capital letters, it will throw.
 
-> We plan to implement experimental support for incremental delivery (`@defer`/`@stream`) before the v4.0.0 release and expect this to change the structure of `GraphQLResponse` further.
-
 ### `plugins` constructor argument does not take factory functions
 
 In Apollo Server 3, each element of the `plugins` array provided to `new ApolloServer` could either be an `ApolloServerPlugin` (ie, an object with fields like `requestDidStart`) or a zero-argument "factory" function returning an `ApolloServerPlugin`.
@@ -1525,7 +1525,7 @@ Apollo Server supports [batching HTTP requests](./workflow/requests/#batching),
 
 In Apollo Server 4, you must explicitly enable this feature by passing `allowBatchedHttpRequests: true` to the `ApolloServer` constructor.
 
-Not all GraphQL clients support HTTP batching, and batched requests will not support incremental delivery when Apollo Server implements that feature. HTTP batching can help performance by sharing a `context` object across operations, but it can make it harder to understand the amount of work any given request does.
+Not all GraphQL clients support HTTP batching, and batched requests do not support incremental delivery. HTTP batching can help performance by sharing a `context` object across operations, but it can make it harder to understand the amount of work any given request does.
 
 
 ### Default cache is bounded