Apollo Server cannot fetch operation manifest, permission denied

[jhalborg]

Intended outcome:
Following the quickstart from the docs will allow the server to fetch operations from the registry

Actual outcome:
I get the following error on server startup:

Apollo Server could not begin serving requests immediately because the operation manifest could not be fetched.  Attempts will continue to fetch the manifest, but all requests will be forbidden until the manifest is fetched. Could not fetch manifest <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymouscaller does not have storage.objects.get access to engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/e95537ddd915785365bbb8a35003ab00d9613cf51b72b628a52e71672b56b41d8e4a2624cb4803b214ab032ccfcd793c97afa46ddba5cf366cb80a9ed5499580.</Details></Error>

in spite of having correctly pass the Engine API key. I've double and triple checked the key, and it works correctly with other services (i.e. tracing).

const apolloServer = new ApolloServer({
  ...
  engine: {
    apiKey: appConfig.ApolloEngineKey,
    schemaTag: appConfig.NodeEnvironment,
    privateVariables: true,
  },
  subscriptions: false,
  plugins: [
    // tslint:disable-next-line:no-require-imports
    require('apollo-server-plugin-operation-registry')({
      forbidUnregisteredOperations: true,
    }),
  ],
});

How to reproduce the issue:
Follow the quickstart, see the error

Versions

"apollo-server": "2.2.6",
"apollo-server-express": "2.2.6",
"apollo-server-plugin-operation-registry": "0.0.1-alpha.6",
"apollo":"2.4.4"

[trevor-scheer]

Hey @jhalborg, thanks for submitting an issue! I was actually running through this workflow yesterday and ran into the same problem. For now, there's a small catch when you first introduce the plugin to your server. You have to publish the schema one more time with service:push.

See the docs here for more details. Sorry about the trouble! Either way, let me know if this solves the problem for you.
https://www.apollographql.com/docs/platform/operation-registry.html#Troubleshooting

[jhalborg]

Hey @trevor-scheer - Thanks for getting back to me, and sorry for the late reply. I'll give it a go ASAP, but it's been moved down our priorities list until #994 is addressed as well, because until then we can't really use the operation registry anyway, as we rely heavily on tag use

[jhalborg]

@trevor-scheer - Finally got time to circle around to this just now, and this issue seems to indeed have been resolved by pushing the service again. However, now I'm blocked by this issue instead 😣 😄

[jhalborg]

My mistake - I still get the same error after downgrading to "apollo-server-plugin-operation-registry": "0.1.0-alpha.1", as a workaround to the issue mentioned above. Output when starting server:

[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Initializing operation registry plugin.
[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Operation registry is configured for 'brandheroes'.  The schema hash is 91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.
[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Initializing operation registry agent...
[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Starting operation registry agent...
🚀 Server ready at http://localhost:4000/graphql
[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Checking for manifest changes at https://storage.googleapis.com/engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json
Apollo Server could not begin serving requests immediately because the operation manifest could not be fetched.  Attempts will continue to fetch the manifest, but all requests will be forbidden until the manifest is fetched. Could not fetch manifest <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json.</Details></Error>
[DEBUG] Tue Apr 16 2019 14:17:01 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Checking for manifest changes at https://storage.googleapis.com/engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json
Could not fetch manifest <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json.</Details></Error>

Our server instantiation:

const apolloServer = new ApolloServer({
  schema,
  formatError: errorFormatter,
  resolvers: { Currency: OTCurrency },
  context: async (req: any): Promise<Context> => {
    ...
  },
  validationRules: [depthLimit(queryDepthLimit)],
  tracing: true,
  cacheControl: true,
  playground: appConfig.NodeEnvironment !== PossibleNodeEnvs.production,
  introspection: true,
  engine: {
    apiKey: appConfig.ApolloEngineKey,
    schemaTag: appConfig.NodeEnvironment === PossibleNodeEnvs.stage ? appConfig.NodeEnvironment : undefined,
    privateVariables: true,
  },
  subscriptions: false,
  plugins: [
    // tslint:disable-next-line:no-require-imports
    require('apollo-server-plugin-operation-registry')({
      // forbidUnregisteredOperations: false, // TODO: Set to true once client queries are set up correctly
      // De-structure the object to get the HTTP `headers` and the GraphQL
      // request `context`.  Additional validation is possible, but this
      // function must be synchronous.  For more details, see the note below.
      forbidUnregisteredOperations({
        context, // Destructure the shared request `context`.
        request: {
          http: { headers }, // Destructure the `headers` class.
        },
      }: {
        context: Context;
        request: {
          http: { headers: Headers };
        };
      }) {
        // If a magic header is in place, allow any unregistered operation.
        if (headers.get('x-Registry-Bypass') === 'SomePasswordToOverrideManifest') {
          return false;
        }

        // Enforce operation safe-listing on all other users.
        return true;
      },
      debug: true,
    }),
  ],
});

[abernix]

@jhalborg Could you try re-publishing your schema with the latest version of the apollo CLI? (e.g. apollo service:push --endpoint=http://server/graphql) and then try using the latest version of the apollo-server-plugin-operation-registry? (Currently, latest is 0.1.0-alpha.2)

[abernix]

Correction: The latest is apollo-server-plugin-operation-registry@0.1.0-alpha.3.

[jhalborg]

@abernix - Same result.

yarn apollo -v:
apollo/2.8.3 darwin-x64 node-v11.6.0

...

$ apollo service:push --key=$ENGINE_API_KEY --endpoint="[our endpoint]"
  ✔ Loading Apollo Project
  ✔ Uploading service to Engine


No change in schema from previous version

id      schema       tag
──────  ───────────  ───────
91a849  schemaname  current


✨  Done in 2.96s.

After that, yarn start still provides output

[nodemon] starting `ts-node src/index.ts`
Welcome to app version 1.1.50
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Initializing operation registry plugin.
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Operation registry is configured for 'brandheroes'.  The schema hash is 91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Initializing operation registry agent...
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Starting operation registry agent...
🚀 Server ready at http://localhost:4000/graphql
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Checking for manifest changes at https://storage.googleapis.com/engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json
Apollo Server could not begin serving requests immediately because the operation manifest could not be fetched.  Attempts will continue to fetch the manifest, but all requests will be forbidden until the manifest is fetched. Could not fetch manifest <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json.</Details></Error>
[DEBUG] Wed Apr 24 2019 16:14:16 GMT+0200 (Central European Summer Time) apollo-server:apollo-server-plugin-operation-registry Checking for manifest changes at https://storage.googleapis.com/engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json
Could not fetch manifest <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to engine-op-manifest-storage-prod/30624306dc472e38a3c9c6bf22e20f77d1a3ef441357be6d7e07af37aca03f9e36d25303367f3edd449a57addf4996bd45f19c7b2127d0d60837dadf3b0ffe82/91a849237e71431a84261e750a8e4779e5ac23aeb7123371a8ec905bf0e8aa6ed7f96b2a40e291f29daa436ca6ac7d4861f03e7cbfbfbf20f16eb7a7d2b975ce.v2.json.</Details></Error>

I noticed another version, 0.1.0-alpha.4 and tried with that as well, but no change.

[jhalborg]

Ping @abernix :)

Do you, Apollo, deem that the Registry is ready for production use? I've met a handful of issues just when trying to get started following the tutorials, and the plugin is still in alpha - but it's marketed as ready in the docs and in talks as far as I can tell

[abernix]

@jhalborg Sorry for the problems and delays in response! (It may be easier to debug this through Intercom support within Apollo Engine. 😄)

That said, we do have many using it in production and we do plan on graduating it to a non-alpha release soon.

Your output from apollo service:push above looks good. Currently, as I'm peeking into your account, it appears operations were last client:pushed with a pre-2.7.0 version of the apollo CLI. Could you also share the output of running both apollo -v and apollo client:push (with any appropriate include flags if you don't have an apollo.config.js setup, per the documentation) with the new version of the Apollo CLI as well?

[trevor-scheer]

@jhalborg is this still an ongoing issue for you? Our new iteration of the operation manifest should resolve this issue altogether. You can find our migration docs here: https://www.apollographql.com/docs/platform/operation-registry/#migrating-from-01-alpha4-to-020-alpha1

I'm going to close this for now, but feel free to reopen if the issue persists after upgrading. Thanks again!

[sanket-work]

@trevor-scheer I am still facing this issue.