What happened?
When inserting rows into BigQuery using the Node.js client with ignoreUnknownValues: true, the emulator (ghcr.io/recidiviz/bigquery-emulator:0.6.6-recidiviz.3.5.2) rejects rows that contain fields not present in the table schema, returning a PartialFailureError. The expected behavior (and what the BigQuery API does) is that ignoreUnknownValues: true causes extra fields to be silently dropped, and the row is inserted successfully with only the schema-defined fields retained.
Here is the definition in the BigQuery client codebase: https://github.com/googleapis/google-cloud-node/blob/cb5c2d8b012e1023ddc5299bd6f3ac26f5b071e1/handwritten/bigquery/src/types.d.ts#L5412
Actual result
Insert fails with a PartialFailureError indicating the unknown fields are invalid.
What did you expect to happen?
Row is inserted successfully; unknown fields are ignored.
How can we reproduce it (as minimally and precisely as possible)?
- Create a table with a defined schema (e.g., fields
id, amount, postedAt)
- Insert a row that includes additional fields not in the schema (e.g.,
category, accountId)
- Pass
ignoreUnknownValues: true in the insert options
import { BigQuery } from '@google-cloud/bigquery';
const bigquery = new BigQuery({
projectId: 'test',
apiEndpoint: 'http://localhost:9050', // recidiviz emulator
});
const schema = {
fields: [
{ mode: 'REQUIRED', name: 'id', type: 'STRING' },
{ mode: 'REQUIRED', name: 'amount', type: 'FLOAT64' },
],
};
await bigquery.dataset('my_dataset').table('my_table').insert(
[
{
id: 'tx_1',
amount: 42.0,
extraField: 'this field is not in the schema', // <-- unknown field
},
],
{ ignoreUnknownValues: true, schema },
);
// Expected: insert succeeds, extraField is silently dropped
// Actual: throws PartialFailureError — row rejected due to unknown field
Anything else we need to know?
To workaround this, explicitly pick only schema-defined fields before inserting, so no unknown fields are ever
sent to the emulator.
Also worth noting that the original goccy emulator respected this field.
What happened?
When inserting rows into BigQuery using the Node.js client with
ignoreUnknownValues: true, the emulator (ghcr.io/recidiviz/bigquery-emulator:0.6.6-recidiviz.3.5.2) rejects rows that contain fields not present in the table schema, returning aPartialFailureError. The expected behavior (and what the BigQuery API does) is thatignoreUnknownValues: truecauses extra fields to be silently dropped, and the row is inserted successfully with only the schema-defined fields retained.Here is the definition in the BigQuery client codebase: https://github.com/googleapis/google-cloud-node/blob/cb5c2d8b012e1023ddc5299bd6f3ac26f5b071e1/handwritten/bigquery/src/types.d.ts#L5412
Actual result
Insert fails with a
PartialFailureErrorindicating the unknown fields are invalid.What did you expect to happen?
Row is inserted successfully; unknown fields are ignored.
How can we reproduce it (as minimally and precisely as possible)?
id,amount,postedAt)category,accountId)ignoreUnknownValues: truein the insert optionsAnything else we need to know?
To workaround this, explicitly pick only schema-defined fields before inserting, so no unknown fields are ever
sent to the emulator.
Also worth noting that the original
goccyemulator respected this field.