Encoding issue when calling importImage with changes

[Clovel]

Issue

It seems that there is an encoding issue when calling importImage with the changes parameter.

When passing changes as a string array to the function, I get the following error :

(HTTP code 400) unexpected - ["EXPOSE is not a valid change command

If I call encodingURIComponent on all the elements of the changes array, I get the following error :

(HTTP code 400) unexpected - ["EXPOSE%2022%2Ftcp","EXPOSE%2080%2Ftcp", ....... entrypoint%22%5D"] is not a valid change command

It is interesting to see that the docs specify that elements of the changes array should be URI encoded : https://docs.docker.com/reference/api/engine/version/v1.43/#tag/Image/operation/ImageCommit

I use to have a working "import" function before using this one, but it would missinterpret the brackets of ENTRYPOINT & CMD changes. Here is it's implementation :

export type ImportDockerImageOptions = {
  repo: string;
  tag: string;
  platform?: string;
} & {
  changes?: string[];
} &
(
  {
    fromSrc: string;
    message?: string;
  } |
  {
    fromImage: string;
  }
);

export const importDockerImage = (
  docker: Dockerode,
  file: string | NodeJS.ReadableStream,
  options: ImportDockerImageOptions,
): Promise<NodeJS.ReadableStream> => {
  const queryString = qs
    .encode(options)
    .replace(/%5B/g, '[')
    .replace(/%5D/g, ']');

  // console.log(`[DEBUG] importDockerImage query string : ${queryString}`);

  const optsf: DockerModem.DialOptions = {
    path: `/images/create?${queryString}`,
    method: 'POST',
    options: {},
    file: file,
    abortSignal: undefined,
    isStream: true,
    statusCodes: {
      200: true,
      404: `repository ${options.repo} does not exist or no read access`,
      500: 'server error',
    },
  };

  return new Promise((resolve, reject) => {
    docker.modem.dial(
      optsf,
      (err, data) => {
        if(err) {
          return reject(err);
        }
        resolve(data as NodeJS.ReadableStream);
      });
  });
};

In the beginning of the function you can see that I process brackets in a special way to make it work. Otherwise I get errors from the Docker API. My ENTRYPOINT bug lied there.

I was hoping that using the "official" importImage function would fix my issue AND make my code more standard.

TODO

• Fix the encoding issue
• Document the behaviour of the function with regards to the changes options
• Declare a type for the options parameter of the function
• Update tests

[Clovel]

I get a new error with this implementation. £

I tried the standard way to make URL query strings : the URLSearchParams class.

export const importDockerImage = (
  docker: Dockerode,
  file: string | NodeJS.ReadableStream,
  options: ImportDockerImageOptions,
): Promise<NodeJS.ReadableStream> => {
  const queryString = new URLSearchParams(options).toString();

  console.log(`[DEBUG] importDockerImage query string : ${queryString}`);

  const optsf: DockerModem.DialOptions = {
    path: `/images/create?${queryString}`,
    method: 'POST',
    // options: {
    //   options,
    // },
    file: file,
    abortSignal: undefined,
    isStream: true,
    statusCodes: {
      200: true,
      400: `invalid request, check your options : ${queryString}`,
      404: `repository ${options.repo} does not exist or no read access`,
      500: 'server error',
    },
  };

  return new Promise((resolve, reject) => {
    docker.modem.dial(
      optsf,
      (err, data) => {
        if(err) {
          return reject(err);
        }
        resolve(data as NodeJS.ReadableStream);
      });
  });
};

Now my error is :

invalid URL escape "%5"

[Clovel]

It seems that the URI encoding for brackets is causing the issue. Replacing %5D and %5B by the bracket signs is accepted by the Docker Modem. But this causes other issues.

I tried a different approch to serializing the changes by encoding elements one by one and adding the changes fields manually like so :

export const importDockerImage = (
  docker: Dockerode,
  file: string | NodeJS.ReadableStream,
  options: ImportDockerImageOptions,
): Promise<NodeJS.ReadableStream> => {
  const { changes, ...restOptions } = options;

  let queryString = qs
    .stringify(restOptions);

  let processedChanges: string | undefined = undefined;
  if(
    changes !== undefined &&
    changes.length > 0
  ) {
    processedChanges = changes
      .map(
        (change) => {
          return `changes=${encodeURIComponent(change)}`
            .replace(/%5D/g, ']')
            .replace(/%5B/g, '[');
        }
      )
      .join('&');
  }

  queryString = `${queryString}${processedChanges !== undefined && processedChanges.length > 0 ? `&${processedChanges}` : ''}`;

  console.log(`[DEBUG] <importDockerImage> Query string :`, queryString);

  const dialOptions: DockerModem.DialOptions = {
    // path: `/images/create?`,
    path: `/images/create?${queryString}`,
    method: 'POST',
    // options: processedOptions,
    options: {},
    file: file,
    abortSignal: undefined,
    isStream: true,
    statusCodes: {
      200: true,
      400: `invalid request, check your options : ${queryString}`,
      // 400: `invalid request, check your options : ${JSON.stringify(processedOptions)}`,
      404: `repository ${options.repo} does not exist or no read access`,
      500: 'server error',
    },
  };

  ...
};

The log of the query string prints out :

[DEBUG] <importDockerImage> Query string : fromSrc=-&repo=redacted&tag=backup-1736343271874&changes=EXPOSE%2022%2Ftcp&changes=EXPOSE%2080%2Ftcp&changes=USER%20redacted&changes=WORKDIR%20%2Fapp&changes=ENTRYPOINT%20[%22docker-php-entrypoint%22]

This implementaition yeilds the exact same result as the first one : Import works but ENTRYPOINT is broken :

Inspect image gives :

{
  ...

          "Entrypoint": [
            "/bin/sh",
            "-c",
            "[\"docker-php-entrypoint\""
        ],
  ...
}

As you can see the 3rd element is broken as it doesn't have an enclosing bracket.

Input is

const input = [
  'EXPOSE 22/tcp',
  'EXPOSE 80/tcp',
  'USER redacted',
  'WORKDIR /app',
  'ENTRYPOINT ["docker-php-entrypoint"]'
]

It is noteworthy that brackets around the ENTRYPOINT command is important, as it doesn't behave like ENTRYPOINT cmdWithNoBrackets.sh, see here : https://docs.docker.com/reference/dockerfile/#understand-how-cmd-and-entrypoint-interact

[Clovel]

I foudn the bug ! It lies within the docker-modem code. Their Modem.prototype.buildQuerystring function doesn't serialize arrays as expected.

I found a fix but editing the function in my node_modules like so :

Modem.prototype.buildQuerystring = function (opts) {
  var clone = {};

  // serialize map and array values as JSON strings, else querystring truncates.
  // 't' and 'extrahosts' can be arrays but need special treatment so that they're
  // passed as multiple qs parameters instead of JSON values.

  console.log(`[DEBUG] <Modem.prototype.buildQuerystring> opts :`, opts);

  Object.keys(opts)
  .map(
    (key, i) => {
      if (
        opts[key] &&
        typeof opts[key] === 'object' &&
        !Array.isArray(opts[key]) &&
        !['t', 'extrahosts'].includes(key)
      ) {
        console.log(`[DEBUG] <Modem.prototype.buildQuerystring> Serializing object key ${key}, value`, opts[key]);

        clone[key] = JSON.stringify(opts[key]);
        console.log(`[DEBUG] <Modem.prototype.buildQuerystring> Serialized object key ${key}, value`, opts[key], 'to', clone[key]);
      } else if (Array.isArray(opts[key])) {
        clone[key] = opts[key]
          .map(
            (val) => {
              if(val && typeof val === 'object') {
                console.log(`[DEBUG] <Modem.prototype.buildQuerystring> Serializing object in array key ${key}, value`, val);

                return JSON.stringify(val);
              } else {
                console.log(`[DEBUG] <Modem.prototype.buildQuerystring> Serializing non-object in array key ${key}, value`, val);

                return val;
              }
            }
          );
      } else {
        console.log(`[DEBUG] <Modem.prototype.buildQuerystring> Serializing non-object key ${key}, value`, opts[key]);

        clone[key] = opts[key];
      }
    }
  );

  const queryStr = querystring.stringify(clone);

  console.log(`[DEBUG] <Modem.prototype.buildQuerystring> queryStr :`, queryStr);

  return queryStr;
};

The comment seems to indicate that this behavior is voluntary, so I will play around with the repo to find the best fix that induces the least changes/bugs.

I'll open an issue over @ docker-modem.

[apocas]

Merged and published.