Announcement: MemCP gets adaptible consistency layer

MemCP is now capable of serializing and deserializing data from RAM.

Thereby MemCP goes new ways regarding database persistency.

In MemCP, every table can be adjusted as wished for one of these three persistency strategies:

• In-Memory Only: This is useful for session storage, observer handles or other temporary handles
• Block-Wise Persistency: Only full shards are persisted. This enables breathtaking IO throughput. Use case: Time Series, Real Time Data, Sensor Data
• Transaction-Wise Persistency: Shards (block-wise) as well as transactions (in a transaction log) are persisted. This is the default for most (slow) databases and guarantees crash-safe transactional persistency

Here’s the comparison:

	In-Memory	Block-Wise	Transaction-Wise
IO load	0	10%	100%
usage	Sessions Temporary Handles	Sensor Data Big Tables	Accounting Data Customer Data
Failure pattern	All data is lost in power outage	At most the recent changes of the last 15 minutes are lost	No data loss in power outage

Compared to MySQL, MemCP uses far less storage space. The ratio on disk is ~1:5 (which is exactly the compression ratio of the compression algorithms of MemCP). On RAM, MemCP performs even better since traditional hard disk based databases only copy the HDD contents into a RAM cache.

Examples for the outstanding compression ratio:

A 109 MB MySQL database of hundrets of thousands of datasets of orders, articles and construction plans (no PDF documents, those are in a separate storage) of a MES system compressed to 19 MB in MemCP

55 MB Online-Shop data (orders, lineitems, categories, customers) compressed to 8.8 MB in MemCP

With a compression ratio of 1:5, the IO throughput for Serialization/Deserialization is also 5-fold. MemCP does not do format conversion. It just dumps a hugely sized memory page into a file. This makes it so fast. There are also plans to do a mmap’ed implementation to further compact crash recovery times.

Saying crash recovery: Thanks to golangs exception handling, the database has never crashed fully yet. Every segfault will cause the current goroutine to throw an exception to either the query handler or the prompt, so the survivability of the process is extremely high. A crash will only effect the currently executed query, but not the other users.

Another feature of golang that benefits us is go’s garbage collector. MariaDB/MySQL has memory leaks. These leaks steadily increase RAM usage over time when using triggers in update queries. The process consumes more RAM than the internal set limits of MariaDB/MySQL. To not run out of hardware resources, we need to restart our servers frequently (on our server, a restart is required every 15 minutes to not exceed the operating system’s graceful limits). On MemCP, there is no such problem. Unused memory references are detected by the garbage collector and cleaned up properly. This also helps with correct snapshotting of transactional data. The in-memory database does not need any extra logic to clean up old versions of the data from old isolated long-running transactions.