Skip to content

ignoreUnknownValues: true is silently ignored for INSERT operations #74

@gugahoi

Description

@gugahoi

What happened?

When inserting rows into BigQuery using the Node.js client with ignoreUnknownValues: true, the emulator (ghcr.io/recidiviz/bigquery-emulator:0.6.6-recidiviz.3.5.2) rejects rows that contain fields not present in the table schema, returning a PartialFailureError. The expected behavior (and what the BigQuery API does) is that ignoreUnknownValues: true causes extra fields to be silently dropped, and the row is inserted successfully with only the schema-defined fields retained.

Here is the definition in the BigQuery client codebase: https://github.com/googleapis/google-cloud-node/blob/cb5c2d8b012e1023ddc5299bd6f3ac26f5b071e1/handwritten/bigquery/src/types.d.ts#L5412

Actual result

Insert fails with a PartialFailureError indicating the unknown fields are invalid.

What did you expect to happen?

Row is inserted successfully; unknown fields are ignored.

How can we reproduce it (as minimally and precisely as possible)?

  1. Create a table with a defined schema (e.g., fields id, amount, postedAt)
  2. Insert a row that includes additional fields not in the schema (e.g., category, accountId)
  3. Pass ignoreUnknownValues: true in the insert options
import { BigQuery } from '@google-cloud/bigquery';
const bigquery = new BigQuery({
  projectId: 'test',
  apiEndpoint: 'http://localhost:9050', // recidiviz emulator
});
const schema = {
  fields: [
    { mode: 'REQUIRED', name: 'id',     type: 'STRING' },
    { mode: 'REQUIRED', name: 'amount', type: 'FLOAT64' },
  ],
};
await bigquery.dataset('my_dataset').table('my_table').insert(
  [
    {
      id: 'tx_1',
      amount: 42.0,
      extraField: 'this field is not in the schema', // <-- unknown field
    },
  ],
  { ignoreUnknownValues: true, schema },
);
// Expected: insert succeeds, extraField is silently dropped
// Actual:   throws PartialFailureError — row rejected due to unknown field

Anything else we need to know?

To workaround this, explicitly pick only schema-defined fields before inserting, so no unknown fields are ever
sent to the emulator.

Also worth noting that the original goccy emulator respected this field.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions