redis6.0.5之redisObject阅读笔记--redis对象(redisObject)第一部分
/* A redis object, that is a type able to hold a string / list / set */ redis对象,是用来保存 字符串,列表,集合的一种类型。 个人理解就是对这些不同的类型做一个统一的封装,这样对外面调用比较友好 /* The actual Redis Object */ #define OBJ_STRING 0 /* String object. */ #define OBJ_LIST 1 /* List object. */ #define OBJ_SET 2 /* Set object. */ #define OBJ_ZSET 3 /* Sorted set object. */ #define OBJ_HASH 4 /* Hash object. */ /* The "module" object type is a special one that signals that the object * is one directly managed by a Redis module. In this case the value points * to a moduleValue struct, which contains the object value (which is only * handled by the module itself) and the RedisModuleType struct which lists * function pointers in order to serialize, deserialize, AOF-rewrite and * free the object. 模块对象类型是一种特殊的类型,意味着由这个对象是一个直接由REDIS模块管理的。 在这种情况下,这个值指向一个包含对象值的模块值结构(只能由模块本身处理)和这个redis模块类型结构, 这个类型结构列出了为了序列化,反序列化,AOF重写和释放对象的函数指针。 * Inside the RDB file, module types are encoded as OBJ_MODULE followed * by a 64 bit module type ID, which has a 54 bits module-specific signature * in order to dispatch the loading to the right module, plus a 10 bits * encoding version. */ 在备份文件RDB文件中,模块类型都被编码为OBJ_MODULE类型,尾随着一个64比特的模块类型ID, 由54bit模块特定签名决定分配真确的模块方法去加载,外加一个10比特的编码版本 #define OBJ_MODULE 5 /* Module object. */ #define OBJ_STREAM 6 /* Stream object. */ /* Objects encoding. Some kind of objects like Strings and Hashes can be * internally represented in multiple ways. The 'encoding' field of the object * is set to one of this fields for this object. */ #define OBJ_ENCODING_RAW 0 /* Raw representation */ #define OBJ_ENCODING_INT 1 /* Encoded as integer */ #define OBJ_ENCODING_HT 2 /* Encoded as hash table */ #define OBJ_ENCODING_ZIPMAP 3 /* Encoded as zipmap */ #define OBJ_ENCODING_LINKEDLIST 4 /* No longer used: old list encoding. */ #define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */ #define OBJ_ENCODING_INTSET 6 /* Encoded as intset */ #define OBJ_ENCODING_SKIPLIST 7 /* Encoded as skiplist */ #define OBJ_ENCODING_EMBSTR 8 /* Embedded sds string encoding */ #define OBJ_ENCODING_QUICKLIST 9 /* Encoded as linked list of ziplists */ #define OBJ_ENCODING_STREAM 10 /* Encoded as a radix tree of listpacks */ #define LRU_BITS 24 #define LRU_CLOCK_MAX ((1<<LRU_BITS)-1) /* Max value of obj->lru */ #define LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */ #define OBJ_SHARED_REFCOUNT INT_MAX /* Global object never destroyed. */ #define OBJ_STATIC_REFCOUNT (INT_MAX-1) /* Object allocated in the stack. */ #define OBJ_FIRST_SPECIAL_REFCOUNT OBJ_STATIC_REFCOUNT typedef struct redisObject { unsigned type:4; //类型 unsigned encoding:4; //编码方式,就是存储方式 unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ //这个地段需要根据不同的淘汰算法来决定,如果使用LRU,那么就是一个时间值(和全局lru_clock相关) //或者LFU数据(最低8位表示频率同时最高16位表示存取时间) int refcount; //引用次数,次数为了表示没有引用,可以释放 void *ptr; //指向对象保存的数据 } robj; *************************************************************************************** /* The a string name for an object's type as listed above * Native types are checked against the OBJ_STRING, OBJ_LIST, OBJ_* defines, * and Module types have their registered name returned. */ 每个对象的类型有一个字符串名字如上表所示(之前定义的,即下面的类型) /* The actual Redis Object */ #define OBJ_STRING 0 /* String object. */ #define OBJ_LIST 1 /* List object. */ #define OBJ_SET 2 /* Set object. */ #define OBJ_ZSET 3 /* Sorted set object. */ #define OBJ_HASH 4 /* Hash object. */ #define OBJ_MODULE 5 /* Module object. */ #define OBJ_STREAM 6 /* Stream object. */) 原始类型都被检测使用类似 OBJ_STRING, OBJ_LIST, OBJ_* 的定义,模块的类型根据它们注册名返回 ******************************************************************************************** char* getObjectTypeName(robj *o) { char* type; if (o == NULL) { type = "none"; //对象为空 } else { switch(o->type) { case OBJ_STRING: type = "string"; break; case OBJ_LIST: type = "list"; break; case OBJ_SET: type = "set"; break; case OBJ_ZSET: type = "zset"; break; case OBJ_HASH: type = "hash"; break; case OBJ_STREAM: type = "stream"; break; //非模块对象 比较直接,如上所示 //模块对象比较特殊,需要获取具体的值,然后从值中获取名称 case OBJ_MODULE: { moduleValue *mv = o->ptr; type = mv->type->name; }; break; default: type = "unknown"; break; } } return type; } /* In Redis objects 'robj' structures of type OBJ_MODULE, the value pointer * is set to the following structure, referencing the moduleType structure * in order to work with the value, and at the same time providing a raw * pointer to the value, as created by the module commands operating with * the module type. 在OBJ_MODULE类型的Redis objects'robj'结构中,值指针被设置为以下结构, 引用moduleType结构来处理该值,同时提供一个原始的值指针, 就如操作模块类型时候创建的模块命令(我个人的理解是如同redisObject中的void *ptr;) * So for example in order to free such a value, it is possible to use * the following code: 所以如果想要释放这样一个值,作为一个例子,我们可以采用如下的代码 * if (robj->type == OBJ_MODULE) { //是模块类型 * moduleValue *mt = robj->ptr; //取出具体的内容 * mt->type->free(mt->value); //释放内容中分配的内存 * zfree(mt); // We need to release this in-the-middle struct as well. //释放redisObject中的void *ptr,因为这个指针指向的内容也需要分配获取 * } */ typedef struct moduleValue { moduleType *type; //模块类型 void *value; //模块具体的值 } moduleValue; /* The module type, which is referenced in each value of a given type, defines * the methods and links to the module exporting the type. */ 对于每个给定类型引用的每个值的模块类型,定义方法和连接供模块外部使用 typedef struct RedisModuleType { uint64_t id; /* Higher 54 bits of type ID + 10 lower bits of encoding ver. */ struct RedisModule *module; moduleTypeLoadFunc rdb_load; moduleTypeSaveFunc rdb_save; moduleTypeRewriteFunc aof_rewrite; moduleTypeMemUsageFunc mem_usage; moduleTypeDigestFunc digest; moduleTypeFreeFunc free; moduleTypeAuxLoadFunc aux_load; moduleTypeAuxSaveFunc aux_save; int aux_save_triggers; char name[10]; /* 9 bytes name + null term. Charset: A-Z a-z 0-9 _- */ //这里说明模块名字的长度为9个字符 } moduleType; ************************************************************************************** robj *createObject(int type, void *ptr) { robj *o = zmalloc(sizeof(*o)); //根据结构体的大小获取内存空间 o->type = type; o->encoding = OBJ_ENCODING_RAW; //原始编码类型 o->ptr = ptr; //指向具体的内容 o->refcount = 1; //应用计算置成1 /* Set the LRU to the current lruclock (minutes resolution), or * alternatively the LFU counter. */ 根据不同的淘汰算法设置不同的淘汰标志,时间(LRU)或者时间加频率(LFU) if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) { o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL; } else { o->lru = LRU_CLOCK(); } return o; } ************************************************************************************** /* Set a special refcount in the object to make it "shared": * incrRefCount and decrRefCount() will test for this special refcount * and will not touch the object. This way it is free to access shared * objects such as small integers from different threads without any * mutex. 在对象内设置一个特殊的引用使得它可以被共享: 函数incrRefCount和decrRefCount()利用特殊引用做测试,不需要直接操作对象内容。 这种方法可以自由访问共享对象,比如不同线程间的小整数而不需要任何互斥。 * A common patter to create shared objects: 一个通用模式来创建共享对象 * robj *myobject = makeObjectShared(createObject(...)); * */ robj *makeObjectShared(robj *o) { serverAssert(o->refcount == 1); o->refcount = OBJ_SHARED_REFCOUNT; //设置为共享计数 return o; } #define OBJ_STATIC_REFCOUNT (INT_MAX-1) /* Object allocated in the stack. */ #define OBJ_FIRST_SPECIAL_REFCOUNT OBJ_STATIC_REFCOUNT void incrRefCount(robj *o) { if (o->refcount < OBJ_FIRST_SPECIAL_REFCOUNT) { o->refcount++; //引用计数++ } else { if (o->refcount == OBJ_SHARED_REFCOUNT) { //如果已经是共享,不需要做任何操作 /* Nothing to do: this refcount is immutable. */ } else if (o->refcount == OBJ_STATIC_REFCOUNT) { //如果已经是共享计数减一的情况,只给提示,不做操作 serverPanic("You tried to retain an object allocated in the stack"); } } } 对于减少计数的情况 void decrRefCount(robj *o) { if (o->refcount == 1) { //如果只剩下一个计数,可以释放这个对象了 switch(o->type) { //针对不同类型释放对象具体的内容数据 case OBJ_STRING: freeStringObject(o); break; case OBJ_LIST: freeListObject(o); break; case OBJ_SET: freeSetObject(o); break; case OBJ_ZSET: freeZsetObject(o); break; case OBJ_HASH: freeHashObject(o); break; case OBJ_MODULE: freeModuleObject(o); break; case OBJ_STREAM: freeStreamObject(o); break; default: serverPanic("Unknown object type"); break; } zfree(o); //释放对象结构体 } else { if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0"); /如果引用计数小于等于0,给提示 if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount--; //最正常的情况,引用计数减去1 } } ************************************************************************************** /* Create a string object with EMBSTR encoding if it is smaller than * OBJ_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is * used. 创建一个嵌入式编码的字符串,如果该字符串的长度小于44个字节,否则需要创建一个原始编码的字符串 * The current limit of 44 is chosen so that the biggest string object * we allocate as EMBSTR will still fit into the 64 byte arena of jemalloc. */ 如果当前限制44被选择,那么我们为最大嵌入式字符串对象分配的内存仍然适合jemalloc关于64字节的内存策略, #define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44 //最大能存储的字符串长度 robj *createStringObject(const char *ptr, size_t len) { if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT) //小于等于44,可以采用嵌入式字符串 return createEmbeddedStringObject(ptr,len); else return createRawStringObject(ptr,len);//大于44,采用独立的原始SDS字符串 } 因为struct redisObject的size为16个字节,struct sdshdr8为3个字节,加上一个结尾'\0',结构需要20个字节 所以总的可以用的字节数为 64 -20 = 44,所以可用字节数为44个字节,即为上述判断的标准 16个字节 typedef struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; //4 void *ptr; //8 } robj; 3个字节 struct __attribute__ ((__packed__)) sdshdr8 { uint8_t len; /* used */ uint8_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[]; }; ************************************************************************************** /* Create a string object with encoding OBJ_ENCODING_RAW, that is a plain * string object where o->ptr points to a proper sds string. */ 创建一个编码类型为OBJ_ENCODING_RAW的字符串对象, 这是一个纯字符串对象,其指针o->ptr指向一个具体的sds字符串 robj *createRawStringObject(const char *ptr, size_t len) { return createObject(OBJ_STRING, sdsnewlen(ptr,len)); } /* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is * an object where the sds string is actually an unmodifiable string * allocated in the same chunk as the object itself. */ 创建一个编码类型为OBJ_ENCODING_EMBSTR的字符串对象,这是一个对象, 其sds字符串实际是是一个不可改变的sds字符串,分配在同样对象同一块内存中 robj *createEmbeddedStringObject(const char *ptr, size_t len) { robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr8)+len+1); //这里就是分配了一整块内存 struct sdshdr8 *sh = (void*)(o+1); //sds结构体开始位置 o->type = OBJ_STRING; o->encoding = OBJ_ENCODING_EMBSTR; o->ptr = sh+1; //指向sds字符串的具体内容开始位置 o->refcount = 1; if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) { o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL; } else { o->lru = LRU_CLOCK(); } sh->len = len; sh->alloc = len; sh->flags = SDS_TYPE_8; if (ptr == SDS_NOINIT) sh->buf[len] = '\0'; else if (ptr) { memcpy(sh->buf,ptr,len); sh->buf[len] = '\0'; } else { memset(sh->buf,0,len+1); \\0等同于'\0', } return o; } ************************************************************************************** /* Create a string object from a long long value. When possible returns a * shared integer object, or at least an integer encoded one. 从一个长整形的值创建一个字符串对象,可能返回一个共享的整型对象,或者至少返回一个整型编码的对象 * If valueobj is non zero, the function avoids returning a a shared * integer, because the object is going to be used as value in the Redis key * space (for instance when the INCR command is used), so we want LFU/LRU * values specific for each key. */ 如果valueobj的值非0,那么函数避免返回一个共享整型对象,因为这个对象会被当做值使用在redis的键空间, (举例 当INCR命令被执行时),所以我们希望每个键都有一个LFU/LRU值 robj *createStringObjectFromLongLongWithOptions(long long value, int valueobj) { robj *o; if (server.maxmemory == 0 || !(server.maxmemory_policy & MAXMEMORY_FLAG_NO_SHARED_INTEGERS)) { //如果最大内存不做限制 或者 虽然有内存策略,但是不淘汰,这两种请情况下,我们还是可以使用共享对象 /* If the maxmemory policy permits, we can still return shared integers * even if valueobj is true. */ valueobj = 0; } if (value >= 0 && value < OBJ_SHARED_INTEGERS && valueobj == 0) { //值在0到OBJ_SHARED_INTEGERS之间,并且允许共享 incrRefCount(shared.integers[value]); //共享引用计数加1 o = shared.integers[value]; //返回共享对象指针 } else { if (value >= LONG_MIN && value <= LONG_MAX) { o = createObject(OBJ_STRING, NULL); //创建对象 o->encoding = OBJ_ENCODING_INT; //编码为整型 o->ptr = (void*)((long)value); } else { o = createObject(OBJ_STRING,sdsfromlonglong(value)); } } return o; } 小整数的共享对象在初始化的时候生成 void createSharedObjects(void) { ... for (j = 0; j < OBJ_SHARED_INTEGERS; j++) { shared.integers[j] = makeObjectShared(createObject(OBJ_STRING,(void*)(long)j)); shared.integers[j]->encoding = OBJ_ENCODING_INT; } ... } ************************************************************************************** /* Wrapper for createStringObjectFromLongLongWithOptions() always demanding * to create a shared object if possible. */ 对函数createStringObjectFromLongLongWithOptions进行包装,尽可能创建一个共享对象 robj *createStringObjectFromLongLong(long long value) { return createStringObjectFromLongLongWithOptions(value,0); } /* Wrapper for createStringObjectFromLongLongWithOptions() avoiding a shared * object when LFU/LRU info are needed, that is, when the object is used * as a value in the key space, and Redis is configured to evict based on * LFU/LRU. */ 对函数createStringObjectFromLongLongWithOptions进行包装避免成为一个共享对象, 因为需要淘汰策略信息(LFU/LRU),因为对象被当做一个值使用在键空间, redis系统的键被配置成基于LFU/LRU的淘汰策略 robj *createStringObjectFromLongLongForValue(long long value) { return createStringObjectFromLongLongWithOptions(value,1); } ************************************************************************************** /* Duplicate a string object, with the guarantee that the returned object * has the same encoding as the original one. 复制一个字符串对象,保证返回的对象拥有和原始对象一致的编码。 * This function also guarantees that duplicating a small integer object * (or a string object that contains a representation of a small integer) * will always result in a fresh object that is unshared (refcount == 1). 这个函数也保证赋值一个小整型对象(或者 一个字符串对象 包含一个小整数的表示) 总是能保证返回一个新的不共享的对象 * The resulting object always has refcount set to 1. */ 结果返回的对象总是有一个等于1的引用值 robj *dupStringObject(const robj *o) { robj *d; serverAssert(o->type == OBJ_STRING); switch(o->encoding) { case OBJ_ENCODING_RAW: return createRawStringObject(o->ptr,sdslen(o->ptr)); case OBJ_ENCODING_EMBSTR: return createEmbeddedStringObject(o->ptr,sdslen(o->ptr)); case OBJ_ENCODING_INT: d = createObject(OBJ_STRING, NULL); d->encoding = OBJ_ENCODING_INT; d->ptr = o->ptr; return d; default: serverPanic("Wrong encoding."); break; } } ************************************************************************************** 创建各种类型的redis对象,统一化,使用指针指向具体的内容,外面看上去一致 robj *createQuicklistObject(void) { quicklist *l = quicklistCreate(); robj *o = createObject(OBJ_LIST,l); o->encoding = OBJ_ENCODING_QUICKLIST; return o; } robj *createZiplistObject(void) { unsigned char *zl = ziplistNew(); robj *o = createObject(OBJ_LIST,zl); o->encoding = OBJ_ENCODING_ZIPLIST; return o; } robj *createSetObject(void) { dict *d = dictCreate(&setDictType,NULL); robj *o = createObject(OBJ_SET,d); o->encoding = OBJ_ENCODING_HT; return o; } robj *createIntsetObject(void) { intset *is = intsetNew(); robj *o = createObject(OBJ_SET,is); o->encoding = OBJ_ENCODING_INTSET; return o; } robj *createHashObject(void) { unsigned char *zl = ziplistNew(); robj *o = createObject(OBJ_HASH, zl); o->encoding = OBJ_ENCODING_ZIPLIST; return o; } robj *createZsetObject(void) { zset *zs = zmalloc(sizeof(*zs)); robj *o; zs->dict = dictCreate(&zsetDictType,NULL); zs->zsl = zslCreate(); o = createObject(OBJ_ZSET,zs); o->encoding = OBJ_ENCODING_SKIPLIST; return o; } robj *createZsetZiplistObject(void) { unsigned char *zl = ziplistNew(); robj *o = createObject(OBJ_ZSET,zl); o->encoding = OBJ_ENCODING_ZIPLIST; return o; } robj *createStreamObject(void) { stream *s = streamNew(); robj *o = createObject(OBJ_STREAM,s); o->encoding = OBJ_ENCODING_STREAM; return o; } robj *createModuleObject(moduleType *mt, void *value) { moduleValue *mv = zmalloc(sizeof(*mv)); mv->type = mt; mv->value = value; return createObject(OBJ_MODULE,mv); } ************************************************************************************** 释放redis对象,根据类型做不同的释放操作 void freeStringObject(robj *o) { if (o->encoding == OBJ_ENCODING_RAW) { sdsfree(o->ptr); } } void freeListObject(robj *o) { if (o->encoding == OBJ_ENCODING_QUICKLIST) { quicklistRelease(o->ptr); } else { serverPanic("Unknown list encoding type"); } } void freeSetObject(robj *o) { switch (o->encoding) { case OBJ_ENCODING_HT: dictRelease((dict*) o->ptr); break; case OBJ_ENCODING_INTSET: zfree(o->ptr); break; default: serverPanic("Unknown set encoding type"); } } void freeZsetObject(robj *o) { zset *zs; switch (o->encoding) { case OBJ_ENCODING_SKIPLIST: zs = o->ptr; dictRelease(zs->dict); zslFree(zs->zsl); zfree(zs); break; case OBJ_ENCODING_ZIPLIST: zfree(o->ptr); break; default: serverPanic("Unknown sorted set encoding"); } } void freeHashObject(robj *o) { switch (o->encoding) { case OBJ_ENCODING_HT: dictRelease((dict*) o->ptr); break; case OBJ_ENCODING_ZIPLIST: zfree(o->ptr); break; default: serverPanic("Unknown hash encoding type"); break; } } void freeModuleObject(robj *o) { moduleValue *mv = o->ptr; mv->type->free(mv->value); zfree(mv); } void freeStreamObject(robj *o) { freeStream(o->ptr); } ************************************************************************************** /* This variant of decrRefCount() gets its argument as void, and is useful * as free method in data structures that expect a 'void free_object(void*)' * prototype for the free method. */ 这个函数是为了满足特定类型函数指针void free_object(void*)调用的需要 void decrRefCountVoid(void *o) { decrRefCount(o); } ************************************************************************************** /* This function set the ref count to zero without freeing the object. * It is useful in order to pass a new object to functions incrementing * the ref count of the received object. Example: * * functionThatWillIncrementRefCount(resetRefCount(CreateObject(...))); * * Otherwise you need to resort to the less elegant pattern: * * *obj = createObject(...); * functionThatWillIncrementRefCount(obj); * decrRefCount(obj); */ 这个函数是为了节省步骤 robj *resetRefCount(robj *obj) { obj->refcount = 0; return obj; } ************************************************************************************** shared.wrongtypeerr = createObject(OBJ_STRING, sdsnew("-WRONGTYPE Operation against a key holding the wrong kind of value\r\n")); 检查类型是否符合,不符合输出错误 int checkType(client *c, robj *o, int type) { if (o->type != type) { addReply(c,shared.wrongtypeerr); return 1; } return 0; } ************************************************************************************** 字符串能否转化成长整形 int isSdsRepresentableAsLongLong(sds s, long long *llval) { return string2ll(s,sdslen(s),llval) ? C_OK : C_ERR; } ************************************************************************************** 对象是否能转化为长整形 int isObjectRepresentableAsLongLong(robj *o, long long *llval) { serverAssertWithInfo(NULL,o,o->type == OBJ_STRING); if (o->encoding == OBJ_ENCODING_INT) { if (llval) *llval = (long) o->ptr; return C_OK; } else { return isSdsRepresentableAsLongLong(o->ptr,llval); } } ************************************************************************************** /* Optimize the SDS string inside the string object to require little space, * in case there is more than 10% of free space at the end of the SDS * string. This happens because SDS strings tend to overallocate to avoid * wasting too much time in allocations when appending to the string. */ 优化字符串对象中SDS字符串的内部存储空间获取空闲的小空间 在SDS字符串末尾超过百分之10的空闲空间时候。发生这种情况的原因在于sds过度分配, 为了避免花费更多时间处理分配额外添加的字符串所需的空间 void trimStringObjectIfNeeded(robj *o) { if (o->encoding == OBJ_ENCODING_RAW && sdsavail(o->ptr) > sdslen(o->ptr)/10) { o->ptr = sdsRemoveFreeSpace(o->ptr); } }