DCache Distributed Storage System | Creation and Use of List Cache Module

DCache Distributed Storage System | Creation and Use of List Cache Module

In the previous DCache series of articles, we introduced the use of DCache and its KV and KK-Row cache modules. This article will continue to introduce how to use the list type cache module in DCache-List cache module.

  • Introduction to List module
  • Create List Cache Module
  • Obtain the DCache interface file
  • Create a cache service proxy
  • Call List Cache Module Service
    • List module read and write operations
    • Instance
  • Other List Cache Module Service Interface
  • summary

DCache is a distributed NoSQL storage system developed based on the TARS framework. It supports multiple data structures, including key-value(key-value pairs), k-k-row(multi-key-value), list(list), set(collection), zset(ordered collection), etc., satisfying Various business needs.

listNamely linked list, often used for sorting messages, such as sorting chat messages on QQ and WeChat. Commonly used are singly linked lists and doubly linked lists, which are composed of several linked list nodes, as shown in the figure below.

In a singly linked list, each node stores the data of the node and the address of the next node; each node in the doubly linked list additionally contains the address of the previous node.

The List module in DCache is implemented based on a doubly linked list. Each memory data unit Block contains two pointers, which point to the previous Block and the next Block respectively. DCache also provides an operation listin a certain section of the API, you can directly query, delete, replace, and cut listin a certain period of elements. The following figure shows the structure of the doubly linked list and the principle of replacement operation.

Similarly, with other modules like, we complete the following steps can be used in the service listcache service

  1. Create List Cache Module
  2. Obtain the DCache interface file
  3. Create a cache service proxy
  4. Call List Cache Module Service

This article will continue on TestDemohow to create a cache module introduces List, and how to call the service TARS service to cache data.

The examples used in this article can be viewed in the GitHub repository DCacheDemo (link at the end of the article) .

In the previous article, we have already introduced the creation of cache modules. The creation process of each type of cache module is similar. This part will not repeat the similar process. It is named here TestDemoList, cache 类型selectList(MKVCache)

As we mentioned, DCache is based TARS development, the use of the TARS and services, just as by .tarsto call the corresponding cache service interface interface file. The difference is, DCache interface file is fixed, we just copy DCache/src/TarsCommunder CacheShare.tars , ProxyShare.tars and DCache/src/ProxyProxy.tars down to under their own project directory. (Link is attached at the end of the article)

For example, the project file structure after the Demo gets the DCache interface file in this article is as follows

DCacheDemo
├── CacheShare.tars
├── ProxyShare.tars
├── Proxy.tars
├── config.conf
├── main.cpp
└── makefile

Previous article we mentioned, will automatically create a routing and proxy After you create an application, and through TestDemohow to create a cache service proxy to invoke the service.

We continue to use TestDemo, add a new module name ModuleTestDemoList, the value of the module name we created earlier TestDemoList, used to call the module through the proxy later, as follows

By TestDemoproxy object module name and agency services TestDemoList, we will be able to call interface List cache module created in front of. This section through a simple example, describes listthe use of an interface module portion of the cache type. For information about other interfaces, see the Proxy Interface Guide (link is attached at the end of the article).

The interface call process is consistent with the TARS service interface call process. If you are still not sure about the calling method and process of TARS service, you can read the article TARS RPC Communication Framework|Provide a variety of remote calling methods to understand the calling method of TARS service.

In the following example, three tool functions will be used, defined as follows

So next, let's take a look at how to use DCache's List cache module.

List module read and write operations

The List module is the list cache module. Here are write interface pushListand read interfaces getList, similar to other interface usage.

Insert data into the list

An interface pushListfor the head or the end of the list to insert data, defined as follows

int pushList(const PushListReq &req)

Wherein the structure PushListis defined as follows

struct PushListReq
{
    1 require string moduleName;//module name
    2 require string mainKey;//main key
    3 require vector<InsertKeyValue> data;//data to be inserted
    4 require bool atHead = true;//true means insert into the head of the list, false means insert into the tail
};

The usage example is as follows

void testPushList(const string &mainKey, const vector<map<string, string>> &data, DCache::ProxyPrx prx)
{
   //Construct request
    DCache::PushListReq req;
    req.moduleName = ModuleTestDemoList;
    req.mainKey = mainKey;
    req.atHead = false;
    
    for(auto item: data)
{
        DCache::InsertKeyValue insertValue;
        insertValue.mainKey = mainKey;
        insertValue.expireTime2.= 60* 60* 24;
                map<string, string>::const_iterator it = item.begin();
        while(it != item.end())
        {
           //Build UpdateValue
            insertValue.mpValue[it->first] = genUpdateValue(DCache::SET, it->second);
            ++it;
        }
        req.data.push_back(insertValue);
    }
    prx->pushList(req);
}

Get list data

An interface getListfor the data and the designated primary key index list of queries is defined as follows

int getList(constGetListReq&req, GetListRsp&rsp)

Wherein the configuration request message GetListReqand a return message structure GetListRspand nested structure Entryis defined as follows

struct GetListReq
{
    1 require string moduleName;//module name
    2 require string mainKey;//main key
    3 require string field;//The set of fields to be queried, multiple fields are separated by',' such as "a,b", "*" means all
    4 require long index;//index
    5 require string idcSpecified = "";//idc area
};
struct GetListRsp
{
    1 require Entry entry;//query result
};
    struct Entry
{
    1 require map<string, string> data;
};

The usage example is as follows

void testGetList(const string &mainKey, constlong &index, DCache::ProxyPrx prx)
{
//Construct request
    DCache::GetListReq req;
    req.moduleName = ModuleTestDemoList;
    req.mainKey = mainKey;
    req.field = "*";
    req.index = index;
    
    DCache::GetListRsp rsp;
    prx->getList(req, rsp);
    
   //print return value
    printMapData(rsp.entry.data);
}

Instance

Let's actually run the above usage example. The complete usage example can be obtained in the GitHub repository DCacheDemo (link at the end of the article) .

We testListtwo read-write interface List mentioned test section, we the primary key testlist inserting a value test, as follows

void testList(DCache::ProxyPrx prx)
{
    cout << START << "testList"<< endl;
    
    string mainKey = "test";
    vector<map<string, string>> data;
    map<string, string> item;
    item["VALUE"] = "test";
    data.push_back(item);
        testPushList(mainKey, data, prx);
    long index = 0;
    testGetList(mainKey, index, prx);

    cout << END<< "testList"<< endl;
}

Next, in the mainexecution of the function

int main(int argc, char*argv[])
{
        ...
            auto prx = comm->stringToProxy<DCache::ProxyPrx>(DCacheTestDemoObj);
           //Call the DCache cache service
            testList(prx);
        ...
}

In addition to obtaining a list of the data interface getListand read key interfaces pushList, DCache also provides a rich user interface List, including bulk insert ( insertMKVBatch), delete ( delMKV), update ( updateMKV), etc., as follows

//Get a certain data specified on the list
int getList(GetListReq req, out GetListRsp rsp);
//Get the data of the specified range on the list
int getRangeList(GetRangeListReq req, out BatchEntry rsp);
//Insert data to the head or tail of the list
int pushList(PushListReq req);
//Pop up the data at the head or tail of the list
int popList(PopListReq req, out PopListRsp rsp);
//Replace the data in the list
int replaceList(ReplaceListReq req);
//Cut the list, only keep the specified interval, delete the data outside the interval
int trimList(TrimListReq req);
//Delete one or more data from the head or tail of the list
int remList(RemListReq req);

Use interfaces previously described getListand pushListis similar, the specific parameters and structure parameters on the interface can Proxy Interface Reference Guide.

This paper introduces the DCache in listprinciple and procedure cache module, while the use of part of the interface is described in detail by specific examples to help readers understand and be able to quickly get started with listthe cache module.

Links in the article:

DCacheDemo:

https://github.com/ETZhangSX/DCacheDemo

CacheShare.tars:

https://github.com/Tencent/DCache/blob/master/src/TarsComm/CacheShare.tars

ProxyShare.tars:

https://github.com/Tencent/DCache/blob/master/src/TarsComm/ProxyShare.tars

Proxy.tars:

https://github.com/Tencent/DCache/blob/master/src/Proxy/Proxy.tars

Proxy interface guide:

https://github.com/Tencent/DCache/blob/master/docs/proxy_api_guide.md

Reference: https://cloud.tencent.com/developer/article/1801835 DCache distributed storage system|The creation and use of the List cache module-Cloud + Community-Tencent Cloud