MemCP Monthly Engineering Update (Feb 9 – Mar 8, 2026)

This month delivered major practical gains: lower memory usage, faster analytics, better reliability under load, and a much stronger admin experience.

Highlights First (Non-Technical)

1) Lower Memory and Storage Cost

Many real-world values (like phone numbers, UUIDs, IDs, and short codes) now use a denser internal format.

For these values, storage can drop by around 50%. That means lower RAM pressure, less disk usage, and better performance because more data fits in cache.

Gain: lower infrastructure cost + faster scans.

2) Faster KPI Dashboards

MemCP now keeps expensive subquery results cached and refreshes them incrementally when rows change.

Instead of recomputing everything after every update, only affected parts are refreshed.

Gain: much better read performance in mixed read/write workloads.

This kind of subquery-cache-plus-incremental-update behavior is still unusual in mainstream databases.

3) Better Admin Dashboard

The dashboard now gives clearer health/status views, better query insights, and more direct management controls.

Gain: faster debugging, easier administration, and less operational risk.

4) Faster Time-Based Analytics (for example GROUP BY YEAR(...))

We improved how MemCP executes grouped analytics on computed time buckets.

Queries like yearly rollups stay efficient instead of degrading into heavy full rescans.

Gain: faster dashboards, reporting, and BI-style aggregate queries.

In practice, this is a class of query where MemCP now performs unusually well.

What You Feel Immediately

• More speed: important read paths stay fast even during concurrent updates.
• Lower footprint: denser encoding means less memory and storage.
• More stability: strong concurrency fixes reduce lock/panic edge cases.
• Easier operations: a more capable dashboard shortens incident/debug cycles.

Big List of Changes

Storage, Concurrency, and Correctness

• Added panic-safe trigger lock handoff in UPDATE/DELETE paths.
• Removed obsolete row-lock experiment code after cleanup.
• Added shard write-owner markers to prevent non-transaction self-deadlocks.
• Avoided nested shard RLock when transaction already owns write lock.
• Resolved stale UPDATE recids before expression evaluation and relaxed lock windows around trigger sections.
• Hardened concurrent update paths with stale recid follow-up logic.
• Fixed multiple ProcessUniqueCollision panic/double-unlock failure modes.
• Refined uniquelockHeld handling to preserve FK/upsert behavior during panic-safe cleanup.
• Fixed boundary equality semantics for index scans.
• Respected lower/upper inclusive flags in index scan execution.
• Added eager index rebuild support after shard rebuild.
• Improved shard load ordering and WAL auto-increment logging.
• Added defensive blob column access to prevent .blobs scan crashes.
• Fixed serializer/backward-compat handling for StorageString, StorageSCMER, StorageSparse, StorageDecimal, and StorageEnum.
• Introduced version-byte storage architecture for safer format evolution.
• Added index threshold setting to skip index creation on tiny shards.
• Added single-shard fast path in shard iteration to avoid goroutine overhead on trivial scans.

Query Planner, Analyzer, and SQL Semantics

• Added index boundary extraction for IN (...) lists and LIKE 'prefix%' patterns.
• Improved boundary extraction for inclusiveness merge, reversed comparisons, and IS NULL.
• Added basic OR support in boundary extraction via conservative range widening.
• Treated lambda sort columns as computed index boundaries.
• Appended sort columns to boundaries in all-equality filter patterns.
• Added computed index columns for optimizer-resolved expressions.
• Restored scalar subquery deduplication and then hardened it via per-row hash caching.
• Unified canonical temp/group/key naming in planner and queryplan for cache stability.
• Improved keytable init code generation to evaluate variables correctly.
• Fixed optimizer constant-folding for nested list values; added dedicated tests.
• Added parser support for SELECT DISTINCT inside subqueries.
• Added parser support for SQL user identifiers, DROP USER IF EXISTS, charset clauses, and FLUSH syntax.
• Fixed CASE parser ambiguities and validity checks (searched/simple CASE ordering and WHEN-THEN requirements).
• Added broader date/math SQL function coverage and MySQL-style date parts.

GROUP BY, Computed Columns, and Invalidation

• Added/extended incremental aggregate cache updates via $increment proxy.
• Re-enabled incremental triggers for additive computed columns.
• Disabled incremental triggers for internal dot columns where correctness risk is higher.
• Enforced exists-filter semantics for non-global grouped plans.
• Reused canonical exists column names and forced full invalidation where needed.
• Added row-specific cache invalidation via $invalidate virtual columns.
• Added filtered sparse computation for compute proxies.
• Avoided recomputation in ComputeColumn while proxy remains valid.
• Generalized exists column behavior to normal keytables.
• Added keytable cleanup triggers and stronger lifecycle invalidation hooks.
• Dropped cached group tables on base-table lifecycle events.

String/SCM Runtime and Low-Level Performance

• Added format-aware StorageString compression.
• Implemented dense nibble-packed tagCString representation.
• Optimized tagCString equality with nibble comparison and no temporary allocations.
• Added tagBString with base64 detection/equality fixes.
• Added Scmer.AppendString for zero-allocation numeric stringification.
• Moved Scmer tag extraction to low byte for faster dispatch.
• Added targeted helper paths (nibbleRangeEqual) for fast-string equality scenarios.
• Added parallel runtime helpers (parallelN, producen mut variants).

Dashboard and Developer UX

• Added EXPLAIN width setting and multiline rendering via <pre> for readability.
• Switched query plan rendering from compact serialization to pretty-printed format.
• Improved shard detail with per-shard deltas, deletion counters, state, sizes, and index details.
• Added trigger visibility and trigger delete actions in table detail pages.
• Improved users/settings UX and added user management page elements.
• Added TRUNCATE handling in dashboard workflows (drop-all-table behavior on DB truncate).
• Fixed dashboard engine selector case handling and nil partition filtering.
• Changed default Scheme console query from (show) to (help).
• Made help output string-based to improve embedding in UI/console.

Testing, CI, Tooling, and Operations

• Added GitHub Actions CI workflow and duplicate-run guard behavior.
• Stabilized SQL test runner startup/restart readiness handling.
• Added worker parallelism options for suite execution.
• Added per-suite duration reporting in test output.
• Improved setup-step failure reporting and fail-fast behavior in tests.
• Hardened UTF-8 SQL handling and isolated known flaky suites.
• Added/expanded suites for incremental aggregates, computed indexes, string compression, date functions, CASE expressions, concurrent contention, and serialization.
• Updated pre-commit launcher to --no-repl and simplified launcher mechanics.
• Improved Dockerfile behavior and startup defaults.
• Updated dependency pinning and Go directive for environment compatibility.

Security and Safety

• Fixed admin privilege escalation caused by repeated migration execution on restart.
• Added guardrails to prevent accidental user base-table drops during cascade paths.

Why These Highlights Matter

• Nibble encoding improves both storage efficiency and runtime cache behavior in one move.
• Incremental subquery caching shifts expensive repeated work off the hot path during mixed read/write traffic.
• Dashboard upgrades shorten the debug loop from “inspect logs and scripts” to direct operational visibility.
• Efficient GROUP BY YEAR(x) is a concrete example of expression-aware planning paying off in real analytics patterns.

Next Focus Areas

• Continue reducing lock hold times in mutation paths while preserving strict safety.
• Expand planner braking and late materialization strategies for ORDER BY + LIMIT pipelines.
• Further normalize computed-index planning and extend coverage for mixed predicate classes.
• Push more runtime operations toward allocation-free hot paths.

More

Visit: https://github.com/launix-de/memcp