Skip to content

RELEASE_NOTES_2026.03.1.md

Latest commit

 

History

History
361 lines (253 loc) · 21.9 KB

RELEASE_NOTES_2026.03.1.md

File metadata and controls

361 lines (253 loc) · 21.9 KB

Arc 2026.03.1 Release Notes

Bug Fixes

Null Handling in Line Protocol Ingestion (#202)

Fixed a bug where missing fields in line protocol ingestion were stored as 0 instead of NULL. When ingesting lines with different field sets to the same measurement (e.g., line 1 has field1, line 2 has field2), the missing fields now correctly produce NULL values in the Parquet output instead of zero values.

Root cause: The type conversion pipeline (convertColumnsToTyped) was discarding null information when converting []interface{} columns to typed arrays, and the Parquet writer was not passing validity bitmaps to Arrow's AppendValues.

Fix: Introduced TypedColumnBatch to carry validity bitmaps alongside typed column data throughout the ingestion pipeline (merge, sort, slice, write). Arrow now receives proper validity bitmaps so Parquet files correctly distinguish between "value is 0" and "value is absent."

Reported by @bjarneksat — thank you!

Stale Cache After Compaction Causes 404 Errors (#204)

Fixed a bug where queries would intermittently fail with HTTP 404 errors after compaction deleted old parquet files from S3. DuckDB's cache_httpfs extension was caching glob results (directory listings) that still referenced deleted files, causing queries to attempt reading non-existent objects.

Root cause: After compaction merged and deleted old parquet files, no cache invalidation was performed. DuckDB's cache_httpfs glob cache, parquet_metadata_cache, and Arc's partition pruner caches all retained stale references until their TTLs expired (~1 hour).

Fix: Added post-compaction cache invalidation that clears all relevant caches (DuckDB cache_httpfs, parquet_metadata_cache, partition pruner, and SQL transform cache) immediately after each successful compaction job completes.

Distributed Cache Invalidation for Enterprise Clustering (#204, #206)

Extended the post-compaction cache fix (#204) to support enterprise clustering, where compaction runs on a dedicated Compactor node separate from Reader/Writer nodes. After each successful compaction job, the Compactor now broadcasts cache invalidation to all healthy cluster peers via POST /api/v1/internal/cache/invalidate.

How it works: Fire-and-forget goroutines with a 5-second timeout broadcast to each peer. If a reader is temporarily unreachable, its cache expires naturally via TTL. The next compaction cycle implicitly retries. Local node caches are invalidated before the broadcast.

Generic Query Error Messages (#207)

All query endpoints (/api/v1/query, /api/v1/query/arrow, /api/v1/query/:measurement) now return the actual DuckDB error message instead of a generic "Query execution failed". Users can now see actionable errors like Parser Error: syntax error at or near "SELEC" directly in API responses and Grafana, making SQL debugging significantly easier.

time_bucket and date_trunc Return Per-Second Rows Instead of Proper Buckets (#212)

Fixed a bug where time_bucket() and date_trunc() GROUP BY queries returned one row per unique second instead of proper time buckets. A 7-day hourly query returned 604,801 rows / 16.9MB instead of 169 rows / 5KB, causing Grafana dashboards to timeout on ranges > 24h.

Root cause: Arc's query rewriter (rewriteTimeBucket, rewriteDateTrunc) converts time-bucketing SQL to epoch-based arithmetic for performance. The rewritten SQL used DuckDB's / operator for division, which performs float division on integers (unlike PostgreSQL). This meant (epoch(time)::BIGINT / 3600) * 3600 returned the original value — no bucketing happened.

Fix: Changed / to // (DuckDB's integer division operator) in all three rewrite locations. (epoch(time)::BIGINT // 3600) * 3600 now correctly truncates to hour boundaries.

WAL Recovery After Flush Failure Replays Already-Flushed Data (#218)

Fixed a bug where WAL recovery after a storage flush failure (S3/Azure timeout) replayed all rotated WAL files — including files whose data was already successfully flushed to parquet. This caused data duplication on object storage, inflated query results, and increased storage costs.

Root cause: The hasFlushFailure recovery branch in the periodic WAL maintenance goroutine called RecoverWithOptions() without first purging old WAL files. The normal maintenance branch already called PurgeOlderThan(safeAge), but the failure branch skipped this step, causing every WAL file on disk to be replayed and re-written with new parquet filenames (since generateStoragePath() uses time.Now()).

Fix: Added PurgeOlderThan(safeAge) before RecoverWithOptions() in the failure branch, mirroring the normal maintenance path. This limits the replay window from all WAL history to safeAge (minimum 30s), eliminating duplication for data flushed before the failure window.

Self-Adjusting Flush Timer (#142)

The periodic flush goroutine used a fixed-period ticker (max_buffer_age_ms / 2), meaning buffers created just after a tick waited up to ~1.5x the configured age before flushing. Replaced with a self-adjusting time.Timer that fires exactly when the oldest buffer is due to expire. A newBufferCh signal channel recomputes the timer on every new buffer creation. Worst-case flush delay drops from ~1.5x to ~1.0x of max_buffer_age_ms.

MQTT CleanSession Configurable — Default Changed to False (#239)

CleanSession was hardcoded to true, which told the MQTT broker to discard any unacknowledged messages when Arc disconnected. With QoS=1 (the default), this silently dropped messages sent during reconnection windows — making at-least-once delivery behave as at-most-once across reconnects.

CleanSession is now configurable per subscription via the API (clean_session field) and defaults to false. Existing subscriptions are unaffected — the migration sets the column default to 0 (false). Users who intentionally want ephemeral sessions can set clean_session: true when creating or updating a subscription.

Delete API: Partial Failure Reporting (#235)

The delete endpoint now correctly reports partial failures. Previously, the response always returned success: true even when some files failed to rewrite. Now returns success: false with HTTP 207 (Multi-Status) when files fail, includes a failed_files list of filenames that could not be processed, and populates the error field with a summary. Successfully processed files are still reported in files_processed.

Replication Observability: Prometheus Metrics and Sequence Gap Detection (#237)

Added two Prometheus metrics for replication monitoring: arc_replication_entries_dropped_total tracks entries dropped due to full replication buffers (both shard and WAL replication), and arc_replication_sequence_gaps_total tracks sequence gaps detected on replication receivers. Receivers now log a warning with gap details when non-consecutive sequences are received, giving operators visibility into which data windows may be incomplete on replicas.

Orphaned Hot File Cleanup After Tiering Migration (#236)

Added a reconciliation pass to the tiering migration cycle that detects and removes orphaned files from hot storage. When migration copies a file to cold but fails to delete it from hot, the hot copy was left orphaned — wasting storage and potentially confusing compaction. The reconciliation runs automatically after each migration cycle: it queries metadata for all cold-tier files, checks if they still exist in hot storage, and deletes any orphans.

Compaction Manifest Cleanup Leaves Orphaned Files (#240)

Fixed three related bugs in the compaction manifest system that could leave orphaned input files alongside compacted output files, causing queries to return duplicated data.

  1. Stale manifest deletion skipped input file cleanup — Manifests older than 7 days were deleted without first removing the input files they tracked. If the compacted output file existed, both input files and output file remained in storage, doubling query results for that partition.

  2. Manifest deleted despite failed input file deletion — During manifest recovery, if some input files failed to delete (e.g., transient storage error), the manifest was deleted anyway. The failed-to-delete input files became permanently orphaned with no tracking for retry.

  3. Manifest filtering skipped on error — When GetFilesInManifests() returned an error, compaction candidates were returned unfiltered, risking re-compaction of files already being processed by another job.

Fix: Stale manifests now follow the same recovery path as normal manifests (verify output, delete inputs, then delete manifest). Manifest deletion is deferred until all input files are successfully removed. On filtering errors, the partition is skipped rather than processed without safety checks.

Unified cache_httpfs TTLs and Scaled Cache Sizes (#214)

DuckDB's cache_httpfs glob, metadata, and file handle caches are now properly tuned. Metadata and file handle TTLs match s3_cache_ttl_seconds (these reference immutable parquet files). Glob TTL is fixed at 10 seconds — directory listings change during compaction, and S3 LIST overhead is negligible. Cache sizes now scale proportionally with s3_cache_size (glob: 5% of block count, metadata/file handles: 10%), with floors at DuckDB defaults for small deployments. No new config settings.

Code Quality

Comprehensive code review across 11 Arc components. Two review passes covering Scheduler, Clustering, Backup, MQTT, Querying, Compaction, Line Protocol, MsgPack, CSV/Parquet Import, and TLE.

Security: Missing Database Name Validation on Write Endpoints

Added isValidDatabaseName() validation to Line Protocol write handlers (/write, /api/v2/write, /api/v1/write/line-protocol), CSV import (/api/v1/import/csv), Parquet import (/api/v1/import/parquet), and MsgPack write (/api/v1/write/msgpack). These endpoints accepted user-supplied database names from query parameters and headers without validation, which could allow path traversal in storage operations. TLE and LP bulk import endpoints already validated correctly.

Security: Backup Restore File Permissions

Backup restore (restoreConfig, restoreSQLite) now writes files with 0600 permissions instead of 0644. The config file (arc.toml) and SQLite database (containing auth tokens and audit logs) were previously world-readable after restore.

Fix: Scheduler Goroutine Leak on Stop

CQScheduler.Stop() now waits for all in-flight query goroutines to complete via sync.WaitGroup before returning. Previously, Stop() cancelled the context but did not wait, leaking goroutines that held DuckDB connections.

Fix: MQTT Subscription TOCTOU Race

StartSubscription() now inserts a nil placeholder into the subscribers map under lock before starting the subscriber, preventing concurrent callers from starting duplicate subscribers for the same subscription ID. All map readers (Shutdown, Stop, Pause, Delete, Restart, GetStats) handle nil entries.

Fix: Cluster Router Unbounded Map Growth

The activeConns map in the cluster router now prunes entries when a node's connection count drops to zero. Previously, entries were never removed, causing memory growth proportional to the total number of unique nodes seen over the lifetime of the process.

Fix: Line Protocol Precision Parameter Now Honored

The precision query parameter on /write (v1) and /api/v2/write (v2) endpoints was silently ignored — timestamps were always treated as nanoseconds. The parameter is now validated (ns, us, ms, s) and passed to ParseBatchWithPrecision(), matching InfluxDB's behavior. Invalid precision values return HTTP 400.

Performance: Pooled Gzip Decompression for Imports

CSV, Parquet, and TLE bulk import endpoints now use the same pooled klauspost gzip reader as the streaming Line Protocol and MsgPack handlers, instead of allocating a new stdlib compress/gzip reader per request. This reduces GC pressure on import-heavy workloads and improves decompression throughput by 3–5x.

Performance: Single-Pass Line Protocol Unescape

The LP parser's unescape() function was replaced with a single-pass byte scanner (from three sequential strings.ReplaceAll calls). A fast path returns immediately when no backslash escapes are present, avoiding allocation entirely. Combined with pre-allocated record slices in ParseBatch/ParseBatchWithPrecision, this reduces allocations in the ingestion hot path.

Performance: Single-Pass SQL Safety Regex

The 7 separate compiled regexes in the SQL safety validator were combined into a single alternation pattern, reducing query validation from 7 regex passes to 1. The repeated strings.ToLower calls in the query path were also consolidated to compute once after SQL pre-processing.

Cleanup: Import Handler Deduplication

The CSV and Parquet import handlers, which shared ~80% identical code (validation, RBAC, file upload, temp file management, import execution, response formatting), were consolidated into a shared handleFileImport() function. Each handler is now a thin wrapper that supplies format-specific options.

Cleanup: Line Protocol Parser Deduplication

The duplicate splitLine() and splitOnComma() functions (~65 lines of identical escape/quote-aware delimiter logic) were extracted into a shared splitOnDelimiter() function. The unused parseTimestamp() method was removed (timestamp parsing is handled inline in parseLineWithPrecision()).

Cleanup: MQTT Repository

Replaced custom contains()/containsImpl() helper functions with strings.Contains() from the standard library.

Infrastructure

Go 1.26 Upgrade

Upgraded from Go 1.25.6 to Go 1.26. Key runtime improvements:

  • Green Tea GC: 10–40% reduction in GC overhead (enabled by default)
  • 30% faster cgo calls: Benefits every DuckDB query and SQLite operation
  • 2x faster io.ReadAll: Improves S3/Azure storage reads and HTTP response parsing
  • Stack allocation for slice backing stores: Compiler can stack-allocate more slices, reducing heap pressure in hot paths

New Features

Backup & Restore API

Arc now includes a full backup and restore system via REST API. Backups capture parquet data files, SQLite metadata (auth, audit, MQTT config), and the arc.toml configuration file — with async operations, real-time progress tracking, and selective restore.

API endpoints (admin-only):

Method Endpoint Description
POST /api/v1/backup Trigger a full backup (async)
GET /api/v1/backup List all available backups
GET /api/v1/backup/status Progress of active operation
GET /api/v1/backup/:id Get backup manifest
DELETE /api/v1/backup/:id Delete a backup
POST /api/v1/backup/restore Restore from a backup (async)

Create a backup:

curl -X POST "http://localhost:8000/api/v1/backup" \
  -H "Authorization: Bearer $TOKEN"

# Response: 202 Accepted
# {"message": "Backup started", "status": "running"}

Poll progress:

curl "http://localhost:8000/api/v1/backup/status" \
  -H "Authorization: Bearer $TOKEN"

# {"operation": "backup", "backup_id": "backup-20260211-143022-a1b2c3d4",
#  "status": "running", "total_files": 1200, "processed_files": 450,
#  "total_bytes": 5368709120, "processed_bytes": 2147483648}

Restore from backup:

curl -X POST "http://localhost:8000/api/v1/backup/restore" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backup_id": "backup-20260211-143022-a1b2c3d4",
    "restore_data": true,
    "restore_metadata": true,
    "restore_config": false,
    "confirm": true
  }'

Key features:

  • Async operations — backup and restore run in background goroutines (2-hour timeout). Clients poll /status for progress
  • What gets backed up — parquet data files, SQLite database (with WAL checkpoint for consistency), and arc.toml config
  • Selective restore — independently choose to restore data, metadata, and/or config
  • Pre-restore safety — existing SQLite and config files are copied with .before-restore suffix before overwriting
  • Destructive restore protection — restore requires explicit confirm: true in the request body
  • Serialized operations — only one backup or restore can run at a time
  • All storage backends — works with local filesystem, S3, and Azure Blob Storage

Configuration:

[backup]
enabled = true                  # default: true
local_path = "./data/backups"   # default: ./data/backups

Backup structure:

{backup_id}/
  manifest.json        # metadata: databases, measurements, file counts, sizes
  data/                # parquet files preserving partition layout
  metadata/arc.db      # SQLite database snapshot
  config/arc.toml      # configuration file

Line Protocol Bulk Import

New endpoint POST /api/v1/import/lp for bulk importing InfluxDB Line Protocol files. Enables one-command migration from InfluxDB to Arc by uploading .lp or .txt files (plain or gzip-compressed).

Data flows through Arc's high-performance columnar ingest pipeline (ArrowBuffer → ArrowWriter → Parquet → storage) — the same path used by streaming LP ingestion — so bulk imports benefit from the same throughput and sort optimization.

Endpoint:

curl -X POST "http://localhost:8000/api/v1/import/lp" \
  -H "X-Arc-Database: mydb" \
  -F "file=@export.lp"

Query parameters:

Parameter Default Description
measurement (all) Filter to a single measurement from the LP file
precision ns Timestamp precision: ns, us, ms, s

Features:

  • Gzip support — automatically detects and decompresses .gz files (magic byte detection)
  • Precision-aware timestamps — lossless conversion from any precision to Arc's internal microsecond format
  • Multi-measurement — a single LP file can contain multiple measurements; all are imported in one request
  • RBAC-aware — checks write permissions for every measurement in the file
  • 500 MB size limit — enforced on both compressed and uncompressed data

Example with precision:

# Import LP file with second-precision timestamps
curl -X POST "http://localhost:8000/api/v1/import/lp?precision=s" \
  -H "X-Arc-Database: mydb" \
  -F "file=@export_seconds.lp"

Response:

{
  "status": "ok",
  "result": {
    "database": "mydb",
    "measurements": ["cpu", "mem"],
    "rows_imported": 150000,
    "precision": "ns",
    "duration_ms": 342
  }
}

TLE (Two-Line Element) Ingestion & Import

Native support for ingesting satellite orbital data in the standard TLE format used by Space-Track.org, CelesTrak, and ground station pipelines. TLE data is parsed into a configurable measurement (default: satellite_tle) with orbital elements as fields and satellite identifiers as tags.

Two endpoints are provided:

Streaming ingestion — POST /api/v1/write/tle — for continuous TLE feeds, cron jobs, and real-time updates:

curl -X POST "http://localhost:8000/api/v1/write/tle" \
  -H "X-Arc-Database: satellites" \
  --data-binary @stations.tle
# → 204 No Content

Bulk import — POST /api/v1/import/tle — for historical backfill from Space-Track.org exports or CelesTrak catalog dumps:

curl -X POST "http://localhost:8000/api/v1/import/tle" \
  -H "X-Arc-Database: satellites" \
  -F "file=@catalog.tle"
{
  "status": "ok",
  "result": {
    "database": "satellites",
    "measurement": "satellite_tle",
    "satellite_count": 28000,
    "rows_imported": 28000,
    "duration_ms": 1250
  }
}

Headers:

Header Default Description
X-Arc-Database default Target database
X-Arc-Measurement satellite_tle Target measurement name

Custom measurement example:

curl -X POST "http://localhost:8000/api/v1/write/tle" \
  -H "X-Arc-Database: satellites" \
  -H "X-Arc-Measurement: iss_orbital_elements" \
  --data-binary @iss.tle

Schema (default measurement satellite_tle):

Column Type Description
norad_id tag NORAD catalog number
object_name tag Satellite name
classification tag U (unclassified), C, S
international_designator tag Launch year + piece
orbit_type tag LEO, MEO, GEO, HEO
inclination_deg field Orbital inclination
raan_deg field Right ascension of ascending node
eccentricity field Orbital eccentricity
arg_perigee_deg field Argument of perigee
mean_anomaly_deg field Mean anomaly
mean_motion_rev_day field Revolutions per day
bstar field BSTAR drag coefficient
semi_major_axis_km field Derived: semi-major axis
period_min field Derived: orbital period
apogee_km field Derived: apogee altitude
perigee_km field Derived: perigee altitude

Features:

  • Pure Go parser — no external dependencies
  • Supports both 3-line (with name) and 2-line (no name) TLE formats, including mixed-format files
  • Gzip-compressed payloads auto-detected
  • Checksum validation with graceful skip on bad entries (warnings collected, not fatal)
  • Derived orbital metrics computed automatically (semi-major axis, period, apogee, perigee, orbit classification)
  • RBAC-aware, cluster routing enabled
  • 500 MB size limit on bulk imports

Performance: TLE ingestion uses a typed columnar fast path that bypasses the []interface{} intermediary and convertColumnsToTyped pass used by generic ingestion. The parser operates directly on []byte input with contiguous record allocation and single-pass typed column construction, achieving ~3.5M records/sec on commodity hardware.

Example query:

SELECT object_name, orbit_type, period_min, perigee_km, apogee_km
FROM satellite_tle
WHERE orbit_type = 'LEO'
ORDER BY period_min
⚡