{"id":8143,"date":"2026-03-08T19:47:22","date_gmt":"2026-03-08T18:47:22","guid":{"rendered":"https:\/\/launix.de\/launix\/?p=8143"},"modified":"2026-03-10T11:16:56","modified_gmt":"2026-03-10T10:16:56","slug":"memcp-monthly-engineering-update-feb-9-mar-8-2026","status":"publish","type":"post","link":"https:\/\/launix.de\/launix\/memcp-monthly-engineering-update-feb-9-mar-8-2026\/","title":{"rendered":"MemCP Monthly Engineering Update (Feb 9 &#8211; Mar 8, 2026)"},"content":{"rendered":"\n<p>This month delivered major practical gains: lower memory usage, faster analytics, better reliability under load, and a much stronger admin experience.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Highlights First (Non-Technical)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1) Lower Memory and Storage Cost<\/h3>\n\n\n\n<p>Many real-world values (like phone numbers, UUIDs, IDs, and short codes) now use a denser internal format.<\/p>\n\n\n\n<p>For these values, storage can drop by around <strong>50%<\/strong>. That means lower RAM pressure, less disk usage, and better performance because more data fits in cache.<\/p>\n\n\n\n<p>Gain: <strong>lower infrastructure cost + faster scans<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2) Faster KPI Dashboards<\/h3>\n\n\n\n<p>MemCP now keeps expensive subquery results cached and refreshes them incrementally when rows change.<\/p>\n\n\n\n<p>Instead of recomputing everything after every update, only affected parts are refreshed.<\/p>\n\n\n\n<p>Gain: <strong>much better read performance in mixed read\/write workloads<\/strong>.<\/p>\n\n\n\n<p>This kind of subquery-cache-plus-incremental-update behavior is still unusual in mainstream databases.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3) Better Admin Dashboard<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"514\" src=\"https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-1024x514.png\" alt=\"\" class=\"wp-image-8145\" srcset=\"https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-1024x514.png 1024w, https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-300x151.png 300w, https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-768x385.png 768w, https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-1536x771.png 1536w, https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1-18x9.png 18w, https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/grafik-1.png 1600w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>The dashboard now gives clearer health\/status views, better query insights, and more direct management controls.<\/p>\n\n\n\n<p>Gain: <strong>faster debugging, easier administration, and less operational risk<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4) Faster Time-Based Analytics (for example <code>GROUP BY YEAR(...)<\/code>)<\/h3>\n\n\n\n<p>We improved how MemCP executes grouped analytics on computed time buckets.<\/p>\n\n\n\n<p>Queries like yearly rollups stay efficient instead of degrading into heavy full rescans.<\/p>\n\n\n\n<p>Gain: <strong>faster dashboards, reporting, and BI-style aggregate queries<\/strong>.<\/p>\n\n\n\n<p>In practice, this is a class of query where MemCP now performs unusually well.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What You Feel Immediately<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>More speed:<\/strong> important read paths stay fast even during concurrent updates.<\/li>\n\n\n\n<li><strong>Lower footprint:<\/strong> denser encoding means less memory and storage.<\/li>\n\n\n\n<li><strong>More stability:<\/strong> strong concurrency fixes reduce lock\/panic edge cases.<\/li>\n\n\n\n<li><strong>Easier operations:<\/strong> a more capable dashboard shortens incident\/debug cycles.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Big List of Changes<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Storage, Concurrency, and Correctness<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added panic-safe trigger lock handoff in UPDATE\/DELETE paths.<\/li>\n\n\n\n<li>Removed obsolete row-lock experiment code after cleanup.<\/li>\n\n\n\n<li>Added shard write-owner markers to prevent non-transaction self-deadlocks.<\/li>\n\n\n\n<li>Avoided nested shard <code>RLock<\/code> when transaction already owns write lock.<\/li>\n\n\n\n<li>Resolved stale UPDATE recids before expression evaluation and relaxed lock windows around trigger sections.<\/li>\n\n\n\n<li>Hardened concurrent update paths with stale recid follow-up logic.<\/li>\n\n\n\n<li>Fixed multiple <code>ProcessUniqueCollision<\/code> panic\/double-unlock failure modes.<\/li>\n\n\n\n<li>Refined <code>uniquelockHeld<\/code> handling to preserve FK\/upsert behavior during panic-safe cleanup.<\/li>\n\n\n\n<li>Fixed boundary equality semantics for index scans.<\/li>\n\n\n\n<li>Respected lower\/upper inclusive flags in index scan execution.<\/li>\n\n\n\n<li>Added eager index rebuild support after shard rebuild.<\/li>\n\n\n\n<li>Improved shard load ordering and WAL auto-increment logging.<\/li>\n\n\n\n<li>Added defensive blob column access to prevent <code>.blobs<\/code> scan crashes.<\/li>\n\n\n\n<li>Fixed serializer\/backward-compat handling for <code>StorageString<\/code>, <code>StorageSCMER<\/code>, <code>StorageSparse<\/code>, <code>StorageDecimal<\/code>, and <code>StorageEnum<\/code>.<\/li>\n\n\n\n<li>Introduced version-byte storage architecture for safer format evolution.<\/li>\n\n\n\n<li>Added index threshold setting to skip index creation on tiny shards.<\/li>\n\n\n\n<li>Added single-shard fast path in shard iteration to avoid goroutine overhead on trivial scans.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Query Planner, Analyzer, and SQL Semantics<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added index boundary extraction for <code>IN (...)<\/code> lists and <code>LIKE 'prefix%'<\/code> patterns.<\/li>\n\n\n\n<li>Improved boundary extraction for inclusiveness merge, reversed comparisons, and <code>IS NULL<\/code>.<\/li>\n\n\n\n<li>Added basic <code>OR<\/code> support in boundary extraction via conservative range widening.<\/li>\n\n\n\n<li>Treated lambda sort columns as computed index boundaries.<\/li>\n\n\n\n<li>Appended sort columns to boundaries in all-equality filter patterns.<\/li>\n\n\n\n<li>Added computed index columns for optimizer-resolved expressions.<\/li>\n\n\n\n<li>Restored scalar subquery deduplication and then hardened it via per-row hash caching.<\/li>\n\n\n\n<li>Unified canonical temp\/group\/key naming in planner and queryplan for cache stability.<\/li>\n\n\n\n<li>Improved keytable init code generation to evaluate variables correctly.<\/li>\n\n\n\n<li>Fixed optimizer constant-folding for nested list values; added dedicated tests.<\/li>\n\n\n\n<li>Added parser support for <code>SELECT DISTINCT<\/code> inside subqueries.<\/li>\n\n\n\n<li>Added parser support for SQL user identifiers, <code>DROP USER IF EXISTS<\/code>, charset clauses, and <code>FLUSH<\/code> syntax.<\/li>\n\n\n\n<li>Fixed CASE parser ambiguities and validity checks (searched\/simple CASE ordering and WHEN-THEN requirements).<\/li>\n\n\n\n<li>Added broader date\/math SQL function coverage and MySQL-style date parts.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">GROUP BY, Computed Columns, and Invalidation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added\/extended incremental aggregate cache updates via <code>$increment<\/code> proxy.<\/li>\n\n\n\n<li>Re-enabled incremental triggers for additive computed columns.<\/li>\n\n\n\n<li>Disabled incremental triggers for internal dot columns where correctness risk is higher.<\/li>\n\n\n\n<li>Enforced exists-filter semantics for non-global grouped plans.<\/li>\n\n\n\n<li>Reused canonical exists column names and forced full invalidation where needed.<\/li>\n\n\n\n<li>Added row-specific cache invalidation via <code>$invalidate<\/code> virtual columns.<\/li>\n\n\n\n<li>Added filtered sparse computation for compute proxies.<\/li>\n\n\n\n<li>Avoided recomputation in <code>ComputeColumn<\/code> while proxy remains valid.<\/li>\n\n\n\n<li>Generalized exists column behavior to normal keytables.<\/li>\n\n\n\n<li>Added keytable cleanup triggers and stronger lifecycle invalidation hooks.<\/li>\n\n\n\n<li>Dropped cached group tables on base-table lifecycle events.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">String\/SCM Runtime and Low-Level Performance<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added format-aware <code>StorageString<\/code> compression.<\/li>\n\n\n\n<li>Implemented dense nibble-packed <code>tagCString<\/code> representation.<\/li>\n\n\n\n<li>Optimized <code>tagCString<\/code> equality with nibble comparison and no temporary allocations.<\/li>\n\n\n\n<li>Added <code>tagBString<\/code> with base64 detection\/equality fixes.<\/li>\n\n\n\n<li>Added <code>Scmer.AppendString<\/code> for zero-allocation numeric stringification.<\/li>\n\n\n\n<li>Moved Scmer tag extraction to low byte for faster dispatch.<\/li>\n\n\n\n<li>Added targeted helper paths (<code>nibbleRangeEqual<\/code>) for fast-string equality scenarios.<\/li>\n\n\n\n<li>Added parallel runtime helpers (<code>parallelN<\/code>, <code>producen<\/code> mut variants).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Dashboard and Developer UX<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added EXPLAIN width setting and multiline rendering via <code>&lt;pre><\/code> for readability.<\/li>\n\n\n\n<li>Switched query plan rendering from compact serialization to pretty-printed format.<\/li>\n\n\n\n<li>Improved shard detail with per-shard deltas, deletion counters, state, sizes, and index details.<\/li>\n\n\n\n<li>Added trigger visibility and trigger delete actions in table detail pages.<\/li>\n\n\n\n<li>Improved users\/settings UX and added user management page elements.<\/li>\n\n\n\n<li>Added TRUNCATE handling in dashboard workflows (drop-all-table behavior on DB truncate).<\/li>\n\n\n\n<li>Fixed dashboard engine selector case handling and nil partition filtering.<\/li>\n\n\n\n<li>Changed default Scheme console query from <code>(show)<\/code> to <code>(help)<\/code>.<\/li>\n\n\n\n<li>Made help output string-based to improve embedding in UI\/console.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Testing, CI, Tooling, and Operations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Added GitHub Actions CI workflow and duplicate-run guard behavior.<\/li>\n\n\n\n<li>Stabilized SQL test runner startup\/restart readiness handling.<\/li>\n\n\n\n<li>Added worker parallelism options for suite execution.<\/li>\n\n\n\n<li>Added per-suite duration reporting in test output.<\/li>\n\n\n\n<li>Improved setup-step failure reporting and fail-fast behavior in tests.<\/li>\n\n\n\n<li>Hardened UTF-8 SQL handling and isolated known flaky suites.<\/li>\n\n\n\n<li>Added\/expanded suites for incremental aggregates, computed indexes, string compression, date functions, CASE expressions, concurrent contention, and serialization.<\/li>\n\n\n\n<li>Updated pre-commit launcher to <code>--no-repl<\/code> and simplified launcher mechanics.<\/li>\n\n\n\n<li>Improved Dockerfile behavior and startup defaults.<\/li>\n\n\n\n<li>Updated dependency pinning and Go directive for environment compatibility.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security and Safety<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fixed admin privilege escalation caused by repeated migration execution on restart.<\/li>\n\n\n\n<li>Added guardrails to prevent accidental user base-table drops during cascade paths.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Why These Highlights Matter<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Nibble encoding<\/strong> improves both storage efficiency and runtime cache behavior in one move.<\/li>\n\n\n\n<li><strong>Incremental subquery caching<\/strong> shifts expensive repeated work off the hot path during mixed read\/write traffic.<\/li>\n\n\n\n<li><strong>Dashboard upgrades<\/strong> shorten the debug loop from \u201cinspect logs and scripts\u201d to direct operational visibility.<\/li>\n\n\n\n<li><strong>Efficient <code>GROUP BY YEAR(x)<\/code><\/strong> is a concrete example of expression-aware planning paying off in real analytics patterns.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Next Focus Areas<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Continue reducing lock hold times in mutation paths while preserving strict safety.<\/li>\n\n\n\n<li>Expand planner braking and late materialization strategies for ORDER BY + LIMIT pipelines.<\/li>\n\n\n\n<li>Further normalize computed-index planning and extend coverage for mixed predicate classes.<\/li>\n\n\n\n<li>Push more runtime operations toward allocation-free hot paths.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">More<\/h2>\n\n\n\n<p>Visit: <a href=\"https:\/\/github.com\/launix-de\/memcp\">https:\/\/github.com\/launix-de\/memcp<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This month delivered major practical gains: lower memory usage, faster analytics, better reliability under load, and a much stronger admin experience.<\/p>\n","protected":false},"author":2,"featured_media":8144,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","_uag_custom_page_level_css":"","footnotes":""},"categories":[129,128],"tags":[],"class_list":["post-8143","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-memcp","category-programming","single-item"],"featured_image_urls_v2":{"full":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"thumbnail":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-150x150.png",150,150,true],"medium":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-300x200.png",300,200,true],"medium_large":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-768x512.png",751,501,true],"large":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-1024x683.png",751,501,true],"1536x1536":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"2048x2048":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"trp-custom-language-flag":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-18x12.png",18,12,true],"xs-thumb":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-64x64.png",64,64,true],"appku-shop-single":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-620x500.png",620,500,true]},"post_excerpt_stackable_v2":"<p>This month delivered major practical gains: lower memory usage, faster analytics, better reliability under load, and a much stronger admin experience. Highlights First (Non-Technical) 1) Lower Memory and Storage Cost Many real-world values (like phone numbers, UUIDs, IDs, and short codes) now use a denser internal format. For these values, storage can drop by around 50%. That means lower RAM pressure, less disk usage, and better performance because more data fits in cache. Gain: lower infrastructure cost + faster scans. 2) Faster KPI Dashboards MemCP now keeps expensive subquery results cached and refreshes them incrementally when rows change. Instead of&hellip;<\/p>\n","category_list_v2":"<a href=\"https:\/\/launix.de\/launix\/category\/memcp\/\" rel=\"category tag\">MemCP<\/a>, <a href=\"https:\/\/launix.de\/launix\/category\/programming\/\" rel=\"category tag\">Programming<\/a>","author_info_v2":{"name":"Carl-Philip H\u00e4nsch","url":"https:\/\/launix.de\/launix\/author\/carli\/"},"comments_num_v2":"0 comments","uagb_featured_image_src":{"full":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"thumbnail":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-150x150.png",150,150,true],"medium":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-300x200.png",300,200,true],"medium_large":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-768x512.png",751,501,true],"large":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-1024x683.png",751,501,true],"1536x1536":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"2048x2048":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6.png",1536,1024,false],"trp-custom-language-flag":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-18x12.png",18,12,true],"xs-thumb":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-64x64.png",64,64,true],"appku-shop-single":["https:\/\/launix.de\/launix\/wp-content\/uploads\/2026\/03\/c58b4cc6-bc2c-46d5-9351-4c0c776edba6-620x500.png",620,500,true]},"uagb_author_info":{"display_name":"Carl-Philip H\u00e4nsch","author_link":"https:\/\/launix.de\/launix\/author\/carli\/"},"uagb_comment_info":0,"uagb_excerpt":"This month delivered major practical gains: lower memory usage, faster analytics, better reliability under load, and a much stronger admin experience.","_links":{"self":[{"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/posts\/8143","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/comments?post=8143"}],"version-history":[{"count":2,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/posts\/8143\/revisions"}],"predecessor-version":[{"id":8148,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/posts\/8143\/revisions\/8148"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/media\/8144"}],"wp:attachment":[{"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/media?parent=8143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/categories?post=8143"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/launix.de\/launix\/wp-json\/wp\/v2\/tags?post=8143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}