# etmem **Repository Path**: bcsecurity/etmem ## Basic Information - **Project Name**: etmem - **Description**: the memory vertical expansion technology that can achieve the purpose of memory capacity expansion and memory cost reduction. - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 58 - **Created**: 2024-10-29 - **Last Updated**: 2025-02-13 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # etmem ## Introduction The development of CPU computing power - particularly lower costs of ARM cores - makes memory cost and capacity become the core frustration that restricts business costs and performance. Therefore, the most pressing issue is how to save memory cost and how to expand memory capacity. etmem is a tiered memory extension technology that uses DRAM+memory compression/high-performance storage media to form tiered memory storage. Memory data is tiered, and cold data is migrated from memory media to high-performance storage media to release memory space and reduce memory costs. ## Compilation Tutorial 1. Download the etmem source code. ``` $ git clone https://gitee.com/openeuler/etmem.git ``` 2. Compile and run the dependencies. The compilation and running of etmem depend on the libboundscheck component. 3. Compile. ``` $ cd etmem $ mkdir build $ cd build $ cmake .. $ make ``` ## Note ### Running Dependency As one of memory extension tools, etmem depends on the support of kernel features to recognize the access information and it could write the memory to the swap area actively in order to achieve the requirements of vertical extension of memory. Etmem needs to insert the module "etmem_scan" and "etmem_swap" when it is running. ``` modprobe etmem_scan modprobe etmem_swap ``` Openeuler 21.03、21.09、20.03 LTS SP2 and 20.03 LTS SP3 support the function of memory extension completely so that we could use these kernels directly. ### Authority Limitation Running etmem needs root authority so we must obey the guidance strictly during the process to avoid the system suffered from security risk because root account has the highest authority. ### Usage Limitation * The client and server of etmem should be deployed on the same server since it not support to communicate cross servers. * Etmem could only scan the target progress which satisfy the length of names less equal than 15 bytes. * Expending memory with AEP devices, it depends on the AEP devices that could be recognized by system and init them as numa nodes. Field of "vm_flags" must be set to "ht" in configure file. * Private commands in engine only point to those tasks within it, e.g. cslide supply showhostpages and showtaskpages. * The fd in function "eng_mgt_func" musn't set 0xff or 0xfe for achievement of the code in the third party. * Support to add different third party librarys in single project distinguished by eng_name in configure file. * Prohibit to scan the same process concurrently. Couldn't use /proc/xxx/idle_pages and /proc/xxx/swap_pages files when etmem_scan.ko and etmem_swap.ko are not loaded. ## Instructions ### Starting Etmemd #### How to Use Run the etmemd process in binary mode. For example: ``` $ etmemd -l 0 -s etmemd_socket ``` #### Help Information Options: ``` -l|\-\-log-level Log level -s|\-\-socket Name of the socket to be listened to -h|\-\-help Show this message -m|\-\-mode-systemctl Mode used to start (systemctl) ``` #### Command-line Options | Option | Description | Mandatory | With Parameter or Not | Value Range | Example Description| | --------------- | ---------------------------------- | -------- | ---------- | --------------------- | ------------------------------------------------------------ | | -l or \-\-log-level | etmemd log level | No | Yes | 0 to 3 | `0`: debug level. `1`: info level. `2`: warning level. `3`: error level. Only logs of the level that is higher than or equal to the configured level are recorded in the `/var/log/message` file.| | -s or \-\-socket | Name of socket to be listened to by etmemd, which is used to interact with the client. | Yes| Yes| A string of fewer than 107 characters| Specify the name of socket to be listened to. | | -h or \-\-help | Help information| No| No| N/A| If this option is specified, the command execution exits after the command output is printed.| | -m or \-\-mode-systemctl| When etmemd is started as a service, this option can be used in the command to support startup in fork mode.| No| No| N/A| N/A| ### etmem configuration file Before running the etmem process, the administrator needs to plan the processes that require memory extension, configure the process information in the etmem configuration file, and configure the memory scan cycles and times, and cold and hot memory thresholds. The example configuration file is stored in `conf/example_conf.yaml` of the root directory of the source code in the source code package. You are advised to store the example configuration file in the `/etc/etmem/` directory. The following is an example: ``` /etc/etmem/cslide_conf.yaml /etc/etmem/slide_conf.yaml /etc/etmem/thirdparty_conf.yaml ``` Contents are as following: ``` [project] name=test loop=1 interval=1 sleep=1 #Example of the slide engine [engine] name=slide project=test [task] project=test engine=slide name=background_slide type=name value=mysql T=1 max_threads=1 #Example of the cslide engine [engine] name=cslide project=test node_pair=2,0;3,1 hot_threshold=1 node_mig_quota=1024 node_hot_reserve=1024 [task] project=test engine=cslide name=background_cslide type=pid name=23456 vm_flags=ht anon_only=no ign_host=no #Example of the thirdparty engine [engine] name=thirdparty project=test eng_name=my_engine libname=/usr/lib/etmem_fetch/my_engine.so ops_name=my_engine_ops engine_private_key=engine_private_value [task] project=test engine=my_engine name=backgroud_third type=pid value=12345 task_private_key=task_private_value ``` Fields in the configuration file are described as follows. | Configuration Item| Description| Mandatory | With Parameter or Not | Value Range| Example Description| |-----------|---------------------|------|-------|------------|-----------------------------------------------------------------| | [project] | Start flag of the common configuration section of a project| No| No| N/A| Start flag of the `project` configuration item, indicating that the following configuration items, before another *[xxx]* or to the end of the file, belong to the project section| | name | Name of a project| Yes| Yes| A string of fewer than 64 characters| Specify the project to‧which‧the‧engine‧and‧task‧are‧to‧be‧mounted. | | loop | Number of memory scan cycles| Yes| Yes| 1 to 10 | loop=3 // Scan for three times.| | interval | Interval for scanning the memory| Yes| Yes| 1 to 1200 | interval=5 // The scanning interval is 5s.| | sleep | Interval between large cycles of each memory scan and operation| Yes| Yes| 1 to 1200 | sleep=10 // The interval between two large cycles is 10s.| | sysmem_threshold | Configuration item of slide engine, stands for the threshold of system swap memory | No | Yes | 0 to 100 | | swapcache_high_wmark | Configuration item of slide engine, the proportion of system memory that swacache could occupy, high_wmark | No | Yes | 1 to 100 | | swapcache_low_wmark | Configuration item of slide engine, the proportion of system memory that swacache could occupy, low_wmark | No | Yes | 1 to swapcache_high_wmark | | [engine] | Start flag of the common configuration section of an engine| No| No| N/A| Start flag of the `engine` configuration item, indicating that the following configuration items, before another *[xxx]* or to the end of the file, belong to the engine section| | project | Project to which the engine belongs| Yes| Yes| A string of fewer than 64 characters| If a project named `test` already exists, you can enter `project=test`.| | engine | Name of the engine| Yes| Yes| slide/cslide/thirdparty | Specify the `slide`, `cslide`, or `thirdparty` policy that is used.| | node_pair | Configuration item of the `cslide` engine, which specifies the node pair of the AEP and DRAM in the system | Mandatory when `engine` is set to `cslide`| Yes| Node IDs of the AEP and DRAM are configured in pairs and separated by commas (,). Node pairs are separated by semicolons (;).| node_pair=2,0;3,1 | | hot_threshold | Configuration item of the `cslide` engine, which specifies the threshold of the hot and cold memory| Mandatory when `engine` is set to `cslide`| Yes| Integer (≥ 0)| hot_threshold=3 // Memory that is accessed fewer than 3 times is identified as cold memory.| |node_mig_quota|Configuration item of the `cslide` engine, which specifies the maximum unidirectional traffic during each migration between the DRAM and AEP|Mandatory when `engine` is set to `cslide`|Yes|Integer (≥ 0)|node_mig_quota=1024 //T he unit is MB. A maximum of 1,024 MB data can be migrated from the AEP to the DRAM or from the DRAM to the AEP at a time.| |node_hot_reserve|Configuration item of the `cslide` engine, which specifies the size of the reserved space for the hot memory in the DRAM|Mandatory when `engine` is set to `cslide`|Yes|Integer (≥ 0)|node_hot_reserve=1024 // The unit is MB. When the hot memory of all VMs is greater than the value of this configuration item, the hot memory is migrated to the AEP.| |eng_name|Configuration item of the `thirdparty` engine, which specifies the engine name and is used for task mounting|Mandatory when `engine` is set to `thirdparty`|Yes|A string of fewer than 64 characters|eng_name=my_engine // When a task is mounted to the thirdparty engine, you can enter `engine=my_engine` in the task.| |libname|Configuration item of the `thirdparty` engine, which specifies the address of the dynamic library of the third-party policy. The address is an absolute address.|Mandatory when `engine` is set to `thirdparty`|Yes|A string of fewer than 64 characters|libname=/user/lib/etmem_fetch/code_test/my_engine.so| |ops_name|Configuration item of the `thirdparty` engine, which specifies the name of the operator in the dynamic library of the third-party policy|Mandatory when `engine` is set to `thirdparty`|Yes|A string of fewer than 64 characters|ops_name=my_engine_ops // Name of the structure of the third-party policy implementation interface| |engine_private_key|(Optional) Configuration item of the `thirdparty` engine. This configuration item is reserved for the third-party policy to parse private parameters.|No|No|Configured based on the private parameters of the third-party policy|Set this configuration item based on the private engine parameters of the third-party policy.| | [task] | Start flag of the common configuration section of a task| No| No| N/A| Start flag of the `task configuration item`, indicating that the following configuration items, before another *[xxx]* or to the end of the file, belong to the task section| | project | Project to which the task is mounted| Yes| Yes| A string of fewer than 64 characters| If a project named `test` already exists, you can enter `project=test`.| | engine | Engine to which the task is mounted| Yes| Yes| A string of fewer than 64 characters| Specify the name of the engine to which the task is mounted. | | name | Name of the task| Yes| Yes| A string of fewer than 64 characters| name=background1 // The task name is `background1`.| | type | Method of identifying the target process| Yes| Yes| pid/name | `pid` indicates that the process is identified based on the process ID, and `name` indicates that the process is identified based on the process name.| | value | Specific fields identified by the target process| Yes| Yes| Actual process ID/name| This configuration item is used together with the `type` configuration item to specify the ID or name of the target process. Ensure that the configuration is correct and unique.| | T | Configuration item of `task` when `engine` is set `slide`. It specifies the threshold of the hot and cold memory.| Mandatory when `engine` is set to `slide`| Yes| 0 to `loop` x 3 | T=3 // The memory that is accessed fewer than three times is identified as cold memory.| | max_threads | Configuration item of `task` when `engine` is set `slide`. It specifies the maximum number of threads in the internal thread pool of etmemd. Each thread processes a memory scan+operation task of a process or subprocess.| No| Yes| 1 to 2 x Number of cores + 1. The default value is `1`.| This configuration item controls the number of internal processing threads of etmemd. When the target process has multiple subprocesses, the larger the value of this configuration item, the more the concurrent executions, but the more the occupied resources.| | vm_flags | Configuration item of `task` when `engine` is set `cslide`. It specifies the flag of the VMA to be scanned. If this configuration item is not configured, the scan is not distinguished.| Mandatory when `engine` is set to `cslide`| Yes| Currently, only `ht` is supported.| vm_flags=ht // Scan the VMA memory whose flag is `ht` (huge page).| | anon_only | Configuration item of `task` when `engine` is set `cslide`. It specifies whether to scan only anonymous pages.| No| Yes| yes/no | anon_only=no // If this configuration item is set to `yes`, only anonymous pages are scanned. If this configuration item is set to `no`, non-anonymous pages are also scanned.| | ign_host | Configuration item of `task` when `engine` is set `cslide`. It specifies whether to ignore the page table scan information on the host.| No| Yes| yes/no | ign_host=no // `yes`: Ignore. `no`: Do not ignore.| | task_private_key | (Optional) Configuration item of `task` when `engine` is set `thirdparty`. This configuration item is reserved for the task of the third-party policy to parse private parameters.| No| No| Configured based on the private parameters of the third-party policy | Set this configuration item based on the private task parameters of the third-party policy.| | swap_threshold | Configuration item of `slide engine` means the process swap memory threshold | No | Yes | The abs value that processes could use | swap_threshold=10g // Processes would not trigger the swap if them posses less than 10g memory. In current version, only support the value unit as g/G. Cooperate with sysmem_threshold, it filter the process in whitelist when system memory below to this threshold. | swap_flag | Configuration item of `slide engine` means if enable specific process swap memory | No | Yes | yes/no | swap_flag=yes//enable specific memory swap in process | ### Creating and Deleting Etmem Project/Engine/Task Objects #### Scenario 1. The administrator creates an etmem project, engine, or task. (A project can contain multiple etmem engines, and an engine can contain multiple tasks.) 2. The administrator deletes an existing etmem project, engine, or task. (Before a project is deleted, all tasks in the project automatically stop.) #### How to Use Run etmem in binary mode. Specify the second parameter `obj` to create or delete a project, engine, or task. The project, engine, or task is identified and distinguished based on the content configured in the configuration file. The prerequisite is that the etmem configuration file is correctly configured and the etmemd process is started. Add an object. ``` etmem obj add -f /etc/example_config.yaml -s etmemd_socket ``` Delete an object. ``` etmem obj del -f /etc/example_config.yaml -s etmemd_socket ``` Print help information. ``` etmem obj help ``` #### Help Information Usage: ``` etmem obj add [options] etmem obj del [options] etmem obj help ``` Options: ``` -f|\-\-file Add configuration file -s|\-\-socket Socket name to connect ``` Notes: The configuration file is required. #### Command-line Options | Option| Description| Mandatory | With Parameter or Not| Example Description| | ------------ | ------------------------------------------------------------ | -------- | ---------- | -------------------------------------------------------- | | -f or \-\-file | Configuration file of an object| Mandatory for the `add` and `del` subcommands| Yes| Specify the path of the file.| | -s or \-\-socket | Name of the socket for communicating with the etmemd server. The value must be the same as that specified when the etmemd process is started.| Mandatory for the `add` and `del` subcommands| Yes| This option is mandatory. When there are multiple etmemd processes, the administrator selects an etmemd process to communicate with.| ### Starting, Stopping, and Querying an Etmem Task #### Scenario After adding a project by running the `etmem obj add` command, you can start or stop the etmem project before running the `etmem obj del` command to delete the project. 1. The administrator starts the added project. 2. The administrator stops a project that has been started. When the administrator runs the `obj del` command to delete a project, the project automatically stops if it has been started. #### How to Use For a project that has been successfully added, you can run the `etmem project` command to start or stop the project. Example commands are as follows: Start a project. ``` etmem project start -n test -s etmemd_socket ``` Stop a project. ``` etmem project stop -n test -s etmemd_socket ``` Query a project. ``` etmem project show -n test -s etmemd_socket ``` Print help information. ``` etmem project help ``` #### Help Information Usage: ``` etmem project start [options] etmem project stop [options] etmem project show [options] etmem project help ``` Options: ``` -n|\-\-name Add project name -s|\-\-socket Socket name to connect ``` Notes: 1. The project name and socket name are required when you execute the `add` or `del` option. 2. The socket name are required when you execute the `show` option. #### Command-line Options | Option| Description| Mandatory | With Parameter or Not| Example Description| | ------------ | ------------------------------------------------------------ | -------- | ---------- | -------------------------------------------------------- | | -n or \-\-name | Project name| Mandatory for the `start`, `stop`, and `show` subcommands| Yes| The project name corresponds to the configuration file name.| | -s or \-\-socket | Name of the socket for communicating with the etmemd server. The value must be the same as that specified when the etmemd process is started.| Mandatory for the `start`, `stop`, and `show` subcommands| Yes| This option is mandatory. When there are multiple etmemd processes, the administrator selects an etmemd process to communicate with.| ### Automatically Starting Etmem with System #### Scenario etmemd allows you to configure the systemd configuration file and start the systemd service in fork mode. #### How to Use Compile the service configuration file to start etmemd. Use the `-m` option to specify the mode. For example: ``` etmemd -l 0 -s etmemd_socket -m ``` #### Help Information Options: ``` -l|\-\-log-level Log level -s|\-\-socket Name of the socket to be listened to -m|\-\-mode-systemctl Mode used to start (systemctl) -h|\-\-help Show this message ``` #### Command-line Options | Option| Description | Mandatory | With Parameter or Not| Value Range| Example Description| |----------------|------------|------|-------|------|-----------| | -l or \-\-log-level | etmemd log level| No| Yes| 0 to 3| `0`: debug level. `1`: info level. `2`: warning level. `3`: error level. Only logs of the level that is higher than or equal to the configured level are recorded in the `/var/log/message` file.| | -s or \-\-socket |Name of socket to be listened to by etmemd, which is used to interact with the client.| Yes| Yes| A string of fewer than 107 characters| Specify the name of socket to be listened to. | |-m or \-\-mode-systemctl | When etmemd is started as a service, this option must be specified in the command.| No| No| N/A| N/A| | -h or \-\-help | Help information| No|No|N/A|If this option is specified, the command execution exits after the command output is printed.| ### Supporting Third-party Memory Extension Policies #### Scenario etmem allows you to register third-party memory extension policies and provides the dynamic library of the scanning module. When etmem is running, the third-party policy elimination algorithm is used to eliminate the memory. You can use the dynamic library of the scanning module provided by etmem and implement the interfaces in the structure required for connecting to etmem. #### How to Use To use a third-party extended elimination policy, perform the following steps: 1. Invoke the scanning interface provided by the scanning module as required. 2. Implement each interface based on the function template provided in the etmem header file and encapsulate the interfaces into structures. 3. Compile the dynamic library of the third-party extended elimination policy. 4. Specify engines of the `thirdparty` type in the configuration file as required. 5. Enter the dynamic library name and interface structure name in the `task` field in the configuration file as required. Other operations are similar to those of other etmem engines. Interface structure templates: ``` struct engine_ops { /* Parse the private parameters of the engine. If there are private parameters, implement this interface; otherwise, set it to NULL. */ int (*fill_eng_params)(GKeyFile *config, struct engine *eng); /* Clear the private parameters of the engine. If there are private parameters, implement this interface; otherwise, set it to NULL. */ void (*clear_eng_params)(struct engine *eng); /* Parse the private parameters of the task. If there are private parameters, implement this interface; otherwise, set it to NULL. */ int (*fill_task_params)(GKeyFile *config, struct task *task); /* Parse the private parameters of the task. If there are private parameters, implement this interface; otherwise, set it to NULL. */ void (*clear_task_params)(struct task *tk); /* Interface for starting a task */ int (*start_task)(struct engine *eng, struct task *tk); /* Interface for stopping a task */ void (*stop_task)(struct engine *eng, struct task *tk); /* Fill in the private parameters related to the PID. */ int (*alloc_pid_params)(struct engine *eng, struct task_pid **tk_pid); /* Destroy the private parameters related to the PID. */ void (*free_pid_params)(struct engine *eng, struct task_pid **tk_pid); /* Private commands required by third-party policies. If no private command is required, set it to NULL. */ int (*eng_mgt_func)(struct engine *eng, struct task *tk, char *cmd, int fd); }; ``` The following is an example of the configuration file. For details, see the configuration file description. ``` #thirdparty [engine] name=thirdparty project=test eng_name=my_engine libname=/user/lib/etmem_fetch/code_test/my_engine.so ops_name=my_engine_ops engine_private_key=engine_private_value [task] project=test engine=my_engine name=background1 type=pid value=1798245 task_private_key=task_private_value ``` **Notes**: You must use the dynamic library of the scanning module provided by etmem and implement the interfaces in the structure required for connecting to etmem. The `fd` field in the `eng_mgt_func` interface cannot be set to `0xff` or `0xfe`. Multiple third-party policy dynamic libraries can be added to a project. They are differentiated by `eng_name` in the configuration file. ### Memory Extension Using the AEP #### Scenario Use the etmem component package to enable the channel for tiered memory extension to the AEP. The huge page of the VM is scanned on the node, and the cslide engine is used to eliminate the policy and migrate the cold memory to the AEP. #### How to Use Use the cslide engine to extend the memory. The following is an example. For details about the parameters, see the configuration file description. ``` #cslide [engine] name=cslide project=test node_pair=2,0;3,1 hot_threshold=1 node_mig_quota=1024 node_hot_reserve=1024 [task] project=test engine=cslide name=background1 type=pid value=1823197 vm_flags=ht anon_only=no ign_host=no ``` **Note**: Do not scan the same process concurrently. In addition, the cslide policy supports the following private commands: - showtaskpages - showhostpages You can run the commands to view the page access information related to the task and the system huge page usage on the host of the VM. The following are example commands: ``` etmem engine showtaskpages <-t task_name> -n proj_name -e cslide -s etmemd_socket etmem engine showhostpages -n proj_name -e cslide -s etmemd_socket ``` **Note**: `showtaskpages` and `showhostpages` support only the `cslide` engine. #### Command-line Options | Option| Description | Mandatory | With Parameter or Not| Example Description| |----|------|------|-------|------| |-n or \-\-proj_name| Project name| Yes| Yes| Specify the name of the existing project to be executed.| |-s or \-\-socket| Name of the socket for communicating with the etmemd server. The value must be the same as that specified when the etmemd process is started.| Yes| Yes| This option is mandatory. When there are multiple etmemd processes, the administrator selects an etmemd process to communicate with.| |-e or \-\-engine| Name of the engine to be executed| Yes| Yes| Specify the name of an existing engine to be executed.| |-t or \-\-task_name| Name of the task to be executed| No| Yes| Specify the name of an existing task to be executed.| ## Contribution 1. Fork this repository. 2. Create the Feat_xxx branch. 3. Commit code. 4. Create a pull request (PR).