标签： redis

Redis expire原理

Posted on | by bob | Leave a Comment on Redis expire原理

https://redis.io/commands/expire

Available since 1.0.0.
Time complexity: O(1)

Set a timeout on key. After the timeout has expired, the key will automatically be deleted. A key with an associated timeout is often said to be volatile in Redis terminology.

The timeout will only be cleared by commands that delete or overwrite the contents of the key, including DEL, SET, GETSET and all the *STORE commands. This means that all the operations that conceptually alter the value stored at the key without replacing it with a new one will leave the timeout untouched. For instance, incrementing the value of a key with INCR, pushing a new value into a list with LPUSH, or altering the field value of a hash with HSET are all operations that will leave the timeout untouched.

The timeout can also be cleared, turning the key back into a persistent key, using the PERSIST command.

If a key is renamed with RENAME, the associated time to live is transferred to the new key name.

If a key is overwritten by RENAME, like in the case of an existing key Key_A that is overwritten by a call like RENAME Key_B Key_A, it does not matter if the original Key_A had a timeout associated or not, the new key Key_A will inherit all the characteristics of Key_B.

Note that calling EXPIRE/PEXPIRE with a non-positive timeout or EXPIREAT/PEXPIREAT with a time in the past will result in the key being deleted rather than expired (accordingly, the emitted key event will be del, not expired).

Refreshing expires

It is possible to call EXPIRE using as argument a key that already has an existing expire set. In this case the time to live of a key is updated to the new value. There are many useful applications for this, an example is documented in the Navigation session pattern section below.

Differences in Redis prior 2.1.3

In Redis versions prior 2.1.3 altering a key with an expire set using a command altering its value had the effect of removing the key entirely. This semantics was needed because of limitations in the replication layer that are now fixed.

EXPIRE would return 0 and not alter the timeout for a key with a timeout set.

Return value

Integer reply, specifically:

  • 1 if the timeout was set.
  • 0 if key does not exist.

Examplesredis> SET mykey “Hello”

 redis> SET mykey "Hello"
 "OK"
 redis> EXPIRE mykey 10
 (integer) 1
 redis> TTL mykey
 (integer) 10
 redis> SET mykey "Hello World"
 "OK"
 redis> TTL mykey(integer)
  -1
 redis>

Pattern: Navigation session

Imagine you have a web service and you are interested in the latest N pages recently visited by your users, such that each adjacent page view was not performed more than 60 seconds after the previous. Conceptually you may consider this set of page views as a Navigation session of your user, that may contain interesting information about what kind of products he or she is looking for currently, so that you can recommend related products.

You can easily model this pattern in Redis using the following strategy: every time the user does a page view you call the following commands:

MULTI
RPUSH pagewviews.user:<userid> http://.....
EXPIRE pagewviews.user:<userid> 60
EXEC

If the user will be idle more than 60 seconds, the key will be deleted and only subsequent page views that have less than 60 seconds of difference will be recorded.

This pattern is easily modified to use counters using INCR instead of lists using RPUSH.

Appendix: Redis expires

Keys with an expire

Normally Redis keys are created without an associated time to live. The key will simply live forever, unless it is removed by the user in an explicit way, for instance using the DEL command.

The EXPIRE family of commands is able to associate an expire to a given key, at the cost of some additional memory used by the key. When a key has an expire set, Redis will make sure to remove the key when the specified amount of time elapsed.

The key time to live can be updated or entirely removed using the EXPIRE and PERSIST command (or other strictly related commands).

Expire accuracy

In Redis 2.4 the expire might not be pin-point accurate, and it could be between zero to one seconds out.

Since Redis 2.6 the expire error is from 0 to 1 milliseconds.

Expires and persistence

Keys expiring information is stored as absolute Unix timestamps (in milliseconds in case of Redis version 2.6 or greater). This means that the time is flowing even when the Redis instance is not active.

For expires to work well, the computer time must be taken stable. If you move an RDB file from two computers with a big desync in their clocks, funny things may happen (like all the keys loaded to be expired at loading time).

Even running instances will always check the computer clock, so for instance if you set a key with a time to live of 1000 seconds, and then set your computer time 2000 seconds in the future, the key will be expired immediately, instead of lasting for 1000 seconds.

How Redis expires keys

Redis keys are expired in two ways: a passive way, and an active way.

A key is passively expired simply when some client tries to access it, and the key is found to be timed out.

Of course this is not enough as there are expired keys that will never be accessed again. These keys should be expired anyway, so periodically Redis tests a few keys at random among keys with an expire set. All the keys that are already expired are deleted from the keyspace.

Specifically this is what Redis does 10 times per second:

  1. Test 20 random keys from the set of keys with an associated expire.
  2. Delete all the keys found expired.
  3. If more than 25% of keys were expired, start again from step 1.

This is a trivial probabilistic algorithm, basically the assumption is that our sample is representative of the whole key space, and we continue to expire until the percentage of keys that are likely to be expired is under 25%

This means that at any given moment the maximum amount of keys already expired that are using memory is at max equal to max amount of write operations per second divided by 4.

How expires are handled in the replication link and AOF file

In order to obtain a correct behavior without sacrificing consistency, when a key expires, a DEL operation is synthesized in both the AOF file and gains all the attached replicas nodes. This way the expiration process is centralized in the master instance, and there is no chance of consistency errors.

However while the replicas connected to a master will not expire keys independently (but will wait for the DEL coming from the master), they’ll still take the full state of the expires existing in the dataset, so when a replica is elected to master it will be able to expire the keys independently, fully acting as a master.

上面的内容来自于redis官网，其实如果我们在java代码中也是可以有所发现的，下面是Jedis的expire() 及API.

public Long expire(final String key, final int seconds) {
	checkIsInMulti();
	client.expire(key, seconds);
	return client.getIntegerReply();
}

Set a timeout on the specified key. After the timeout the key will be automatically deleted by the server. A key with an associated timeout is said to be volatile in Redis terminology.
Voltile keys are stored on disk like the other keys, the timeout is persistent too like all the other aspects of the dataset. Saving a dataset containing expires and stopping the server does not stop the flow of time as Redis stores on disk the time when the key will no longer be available as Unix time, and not the remaining seconds.
Since Redis 2.1.3 you can update the value of the timeout of a key already having an expire set. It is also possible to undo the expire at all turning the key into a normal key using the PERSIST command.
Time complexity: O(1)

Integer reply, specifically: 1: the timeout was set. 0: the timeout was not set since the key already has an associated timeout (this may happen only in Redis versions < 2.1.3, Redis >= 2.1.3 will happily update the timeout), or the key does not exist.

springboot2+redis实现注解缓存

Posted on | by bob | Leave a Comment on springboot2+redis实现注解缓存

一、项目核心依赖(pom文件)

       <!--LettuceConnectionFactory-->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-pool2</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>

二、核心配置文件（yaml文件）

spring:
  application:
    name: postgresql-demo
  datasource:
    url: jdbc:postgresql://localhost:5432/test
    username: postgres
    password: 123456
    driver-class-name: org.postgresql.Driver
  jpa:
    show-sql: true
    properties:
      hibernate:
        temp:
          use_jdbc_metadata_defaults: false
        dialect: org.hibernate.dialect.PostgreSQLDialect
        hbm2ddl:
          auto: update
  cache:
    type: redis
  redis:
    timeout: 10s
    lettuce:
      pool:
        max-active: 8
        max-wait: 30s
        max-idle: 8
        min-idle: 0

三、Redis的配置文件(Java文件)

package com.wangyousong.redis.config;

import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CachingConfigurerSupport;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.interceptor.KeyGenerator;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheConfiguration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.RedisSerializationContext;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

import java.time.Duration;
import java.util.Arrays;


@EnableCaching
@Configuration
public class RedisConfig extends CachingConfigurerSupport {

    private final LettuceConnectionFactory lettuceConnectionFactory;

    @Autowired
    public RedisConfig(LettuceConnectionFactory lettuceConnectionFactory) {
        this.lettuceConnectionFactory = lettuceConnectionFactory;
    }

    @Bean
    public KeyGenerator keyGenerator() {
        return (target, method, params) -> {
            StringBuilder sb = new StringBuilder();
            sb.append(target.getClass().getName());
            sb.append(method.getName());
            Arrays.stream(params).map(Object::toString).forEach(sb::append);
            return sb.toString();
        };
    }
 
 
    // key键序列化方式
    private RedisSerializer<String> keySerializer() {
        return new StringRedisSerializer();
    }
 
    /**
     * json序列化
     */
    @Bean
    @SuppressWarnings({"rawtypes", "unchecked"})
    public RedisSerializer<Object> jackson2JsonRedisSerializer() {
        //使用Jackson2JsonRedisSerializer来序列化和反序列化redis的value值
        Jackson2JsonRedisSerializer serializer = new Jackson2JsonRedisSerializer(Object.class);
        ObjectMapper mapper = setObjectMapper();
        serializer.setObjectMapper(mapper);
        return serializer;
    }

    private ObjectMapper setObjectMapper() {
        ObjectMapper mapper = new ObjectMapper();
        mapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        mapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
        return mapper;
    }

    /**
     * 配置缓存管理器
     */
    @Bean
    public CacheManager cacheManager() {
        // 生成一个默认配置，通过config对象即可对缓存进行自定义配置
        RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig();
        // 设置缓存的默认过期时间--10分钟，也是使用Duration设置,默认-1表示不过期
        config = config.entryTtl(Duration.ofMinutes(10))
         // 设置 key为string序列化
         .serializeKeysWith(RedisSerializationContext.SerializationPair.fromSerializer(new StringRedisSerializer()))
         // 设置value为json序列化
         .serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(jackson2JsonRedisSerializer()))
         // 不缓存空值
         .disableCachingNullValues();
        // 使用自定义的缓存配置初始化一个cacheManager
        return RedisCacheManager
         .builder(lettuceConnectionFactory)
         .cacheDefaults(config)
         .transactionAware()
         .build();
    }
 
    /**
     * RedisTemplate配置
     */
    @Bean
    @SuppressWarnings({"rawtypes", "unchecked"})
    public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory lettuceConnectionFactory) {
        RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
        redisTemplate.setConnectionFactory(lettuceConnectionFactory);

        Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
        ObjectMapper objectMapper = setObjectMapper();
        jackson2JsonRedisSerializer.setObjectMapper(objectMapper);

        //key
        redisTemplate.setKeySerializer(keySerializer());
        redisTemplate.setHashKeySerializer(keySerializer());
        //value 值采用json序列化
        redisTemplate.setValueSerializer(jackson2JsonRedisSerializer);
        redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);
 
        redisTemplate.afterPropertiesSet();
        return redisTemplate;
    }
}

四、实例代码及运行效果

五、后记（代码小结）

每次写小结的时候，我都知道要认真写了，应该写代码不仅仅是为了写代码，而是为了回顾与总结在写代码时弄清楚的问题，解决问题是结果，但是我却享受过程。
（1）yaml文件中与spring.redis.timeout=10s相等的那条配置，如果写错了，比如timeout=0了，那么会遇到超时的错误；
（2）spring.redis.pool其实都有默认的值，不写也是可以的，只要要注意一下spring.redis.pool.max-wait,它的默认值是-1.


（3）关于KeyGenerator，在RedisConfig中配置，我们是如下配置：

类名+方法名+各个参数

但是实际应用的时候，我们需要根据实际情况应用，有时候还是需要自己定义。
（4）关于@Cacheable中的unless = “#result == null”，这个需要注意。如果没有写这个属性，我们缓存一个null，会报错的。显然缓存null也不是我们希望的，所以在RedisConfig中，我们是设置了不缓存空值的。

当然，这点会在程序报错的时候看到友好的提示的。比如调用UserService.findById(String id),如果id是一个不存的，那么返回的结果Optional中value便是null.
（5）关于默认缓存的时间，默认如果不配置的话TTL=-1,表示永久不过期，在上面RedisConfig中我们设置的是10分钟，通常这个值需要根据业务需求设置。
（6）最后也是看了缓存应用中的问题。UserService中findById(String id)负责查询并缓存查询结果，void update(UserDTO userDTO)负责更新数据，并清空缓存，不然 findById(String id)拿到数据就是“脏数据”，或者说是过期数据，实际的效果就是用户的体验不是很好，因为用户看不到自己实时提交(修改)的数据。所以这时候@CacheEvict注解就很重要了。当时我演示的时候，发现就是没有清空OptionalUser，我当时就很好奇，为什么？注解难道用错了？还是环境配置有问题，还是什么其它的问题？想了一番，乱搜了一番，无果。冷静下来，看看Spring Cache是如果是实现@CacheEvict机制的，换句话说，就是spring cache 是如何解析 @CacheEvict 这个注解的？后来看了一下源码，从@interface CacheEvict 看到interface CacheAnnotationParser，再看到class SpringCacheAnnotationParser，再看到abstract class CacheOperation，再追踪到class CacheEvictOperation，再一搜被调用的地方：

看到CacheAspectSupport中Aspect，就一下子想通了，肯定是通过SOP,所以结果就是在这个类中

然后就打了一下断点运行，看到update()到底做了什么，为什么没有更新findById查询出来的OptionalUser，后来一下他们的key不一样，如果我不指定key值，将采用默认的key生成策略，就是前面介绍的目标类::方法限定名+参数列表，显然查询和更新的方法名是不一样的，尽管我定义了value的值。所以我重写了@Cacheable(value = “OptionalUser”, key = “#id”, unless = “#result == null”)
@CacheEvict(value = “OptionalUser”, key = “#userDTO.id”) 这样doEvict(cache,key)就可以如期清除掉指定key的缓存。

写在最后，理解Spring Cache的缓存机制对于缓存的应用是有莫大的帮助的，不然都理解不了为什么@CacheEvict会失效。