梦飞环报

Redis多数据源，看这篇就够了

2024-05-06 11:40 由 codeba 发表于 #后端开发

背景

Redis多数据源常见的场景：

  1. 分区数据处理：当数据量增长时，单个Redis实例可能无法处理所有的数据。通过使用多个Redis数据源，可以将数据分区存储在不同的实例中，使得数据处理更加高效。
  2. 多租户应用程序：对于多租户应用程序，每个租户可以拥有自己的Redis数据源，以确保数据隔离和安全性。

目前在网上的一些解决方案无法完全满足特定需求，例如不支持动态添加或更新数据源、缺乏数据源的负载均衡功能，或者不能开箱即用需要自行封装一些常用方法。

为了解决这些问题，封装了一个轻量级的Redis多数据源库，已在生产环境中长时间稳定运行。现分享出来，希望这个库能对你的项目有所帮助。

https://github.com/codebaorg/redis-keeper

如果这篇文章帮助到了你，欢迎star一下，感谢你的支持。

特征

  • 基于redisson封装，保留redisson的所有强大功能。
  • 支持redis多数据源配置的实时更新。
  • 支持redis多数据源的负载均衡。
  • 支持redis数据源的“只读”、“只写”、“读写”、“跳过”的状态切换。
  • 提供 130+ 常用的redis操作方法，包括但不限string、list、hash、set、zset、geo、bitmap、hyberloglog、分布式锁、布隆过滤器、限流等。

快速开始

  1. 添加maven依赖
<dependency>
    <groupId>org.redisson</groupId>
    <artifactId>redisson</artifactId>
    <version>3.15.5</version>
    <scope>provided</scope>
</dependency>

<dependency>
	<groupId>org.codeba</groupId>
	<artifactId>redis-keeper-spring-boot-starter</artifactId>
	<version>2024.0.0</version>
</dependency>
  1. springboot的配置文件，添加redis多数据源的配置
redis-keeper:
  redis:
    datasource:
      ds1: // 此处的命名可以随便取，但是要保证唯一不重复
        host: localhost
        port: 6379
        database: 0
        password: yourPass
      
      ds2: // 此处的命名可以随便取，但是要保证唯一不重复
        host: localhost
        port: 6379
        database: 0
        password: yourPass
  1. 按自定义的数据源别名获取相应的cacheTemplate，执行redis命令。

方式一：通过CacheTemplateProvider获取cacheTemplate，示例如下：

public class AppTest {

    @Autowired
    private CacheTemplateProvider<CacheTemplate> provider;

    public void test() {
        final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1");

        if (templateOptional.isPresent()) {
            final CacheTemplate cacheTemplate = templateOptional.get();
            String key = "foo";
            String value = "bar";
            cacheTemplate.set(key, value);
        }
    }
    
}

方式二：通过代理类CacheTemplateProxy获取cacheTemplate，示例如下：

public class AppTest {

    private final CacheTemplate cacheTemplate = CacheTemplateProxy.asTemplate("ds1");

    public void test() {
        String key = "foo";
        String value = "bar";
        cacheTemplate.set(key, value);
    }
    
}

更多示例

多数据源实时更新

需要：spring cloud + nacos

现有的数据源配置示例如下：

redis-keeper:
  redis:
    datasource:
      ds1:
        host: localhost
        port: 6379
        database: 10
        password: xxx
        invoke-params-print: true

测试方法：

@RestController
@RequiredArgsConstructor
public class TestController {

    private final CacheTemplateProvider<CacheTemplate> provider;

    @RequestMapping("/test")
    public void testRefresh() {
        final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1");
        templateOptional.ifPresent(cacheTemplate -> {
            cacheTemplate.set("foo", "bar");
            cacheTemplate.del("foo");
        });
    }


}

访问上面的方法，日志打印如下（可以看到写入的redis库是10库）：

2024-03-12T11:33:37.850+08:00  INFO 62914 --- [nio-8000-exec-1] o.c.r.k.support.DefaultRedissonTemplate  : cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=10}]
2024-03-12T11:33:37.858+08:00  INFO 62914 --- [nio-8000-exec-1] o.c.r.k.support.DefaultRedissonTemplate  : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=10}]

修改数据源配置，比如修改库为0库，此时可以看到nacos配置实时更新日志：

2024-03-12T11:43:51.683+08:00  INFO 62914 --- [r-localhost-216] com.alibaba.nacos.common.remote.client   : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] Receive server push request, request = ConfigChangeNotifyRequest, requestId = 1
2024-03-12T11:43:51.683+08:00  INFO 62914 --- [r-localhost-216] c.a.n.client.config.impl.ClientWorker    : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] [server-push] config changed. dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP,tenant=null
2024-03-12T11:43:51.684+08:00  INFO 62914 --- [r-localhost-216] com.alibaba.nacos.common.remote.client   : [6253dca3-136e-4ac1-b894-fb4c11483703_config-0] Ack server push request, request = ConfigChangeNotifyRequest, requestId = 1
2024-03-12T11:43:51.700+08:00  INFO 62914 --- [s.client.Worker] c.a.n.client.config.impl.ClientWorker    : [fixed-localhost_8848] [data-received] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, tenant=, md5=dddd513021564c6d6322e18783ac4f2b, content=server:

redis-keeper:
  redis:
    datasource:
      ds1:
        host: localhost
    ..., type=yaml
2024-03-12T11:43:51.701+08:00  INFO 62914 --- [s.client.Worker] c.a.nacos.client.config.impl.CacheData   : [fixed-localhost_8848] [notify-listener] time cost=0ms in ClientWorker, dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b, listener=com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1@178412d1 
2024-03-12T11:43:51.701+08:00  INFO 62914 --- [ternal.notifier] c.a.nacos.client.config.impl.CacheData   : [fixed-localhost_8848] [notify-context] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b
2024-03-12T11:43:51.915+08:00  INFO 62914 --- [ternal.notifier] c.a.c.n.c.NacosConfigDataLoader          : [Nacos Config] Load config[dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP] success
2024-03-12T11:43:51.950+08:00  INFO 62914 --- [ternal.notifier] o.s.c.e.event.RefreshEventListener       : Refresh keys changed: [redis-keeper.redis.datasource.ds1.database]
2024-03-12T11:43:51.950+08:00  INFO 62914 --- [ternal.notifier] c.a.nacos.client.config.impl.CacheData   : [fixed-localhost_8848] [notify-ok] dataId=redis-keeper-demo.yaml, group=DEFAULT_GROUP, md5=dddd513021564c6d6322e18783ac4f2b, listener=com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1@178412d1 ,cost=249 millis.

再次访问上面的方法，日志打印如下（可以看到写入的redis库是0库）：

2024-03-12T11:46:11.849+08:00  INFO 62914 --- [nio-8000-exec-4] o.c.r.k.support.DefaultRedissonTemplate  : cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12T11:46:11.861+08:00  INFO 62914 --- [nio-8000-exec-4] o.c.r.k.support.DefaultRedissonTemplate  : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]

多数据源负载均衡

有的业务场景需要多套redis集群，来承接同一业务的流量，起到分流的作用，这时就需要从多个数据源中获取其中的某一个进行redis的读写。

负载均衡的算法目前支持两种：轮询、随机。

redis多数据源配置，示例如下：

注意：只有 redis-keeper.redis.datasources 或者 redis-keeper.redisson.datasources 配置下的多数据源支持负载均衡。

redis-keeper:
  redis:
    datasources:
      ds2:
        - host: localhost
          port: 6379
          database: 1
          password: xxx
          invoke-params-print: true

        - host: localhost
          port: 6379
          database: 2
          password: xxx
          invoke-params-print: true

        - host: localhost
          port: 6379
          database: 3
          password: xxx
          invoke-params-print: true

轮询示例如下：

public class AppTest {

    @Autowired
    private CacheTemplateProvider<CacheTemplate> provider;

    public void testPollTemplate() {
        String key = "foo";

        for (int i = 1; i <= 10; i++) {
            final Optional<CacheTemplate> templateOptional = provider.pollTemplate("ds2");
            if (templateOptional.isPresent()) {
                final CacheTemplate template = templateOptional.get();

                final String str = String.valueOf(i);

                template.set(key, str);

                template.get(key).ifPresent(el -> {
                    assert str.equals(el);
                });

                template.del(key);
            }
        }

    }
    
}

日志打印结果，可以看到配置的redis数据源的出现的次数是相近的：

2024-03-12 12:15:15.614  INFO 67575 --- [main] : cmd:set, params:[foo, 1], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.621  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.624  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.627  INFO 67575 --- [main] : cmd:set, params:[foo, 2], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.629  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.635  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.635  INFO 67575 --- [main] : cmd:set, params:[foo, 3], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.636  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.638  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:15:15.639  INFO 67575 --- [main] : cmd:set, params:[foo, 4], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.652  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.653  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.654  INFO 67575 --- [main] : cmd:set, params:[foo, 5], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.655  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.656  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.656  INFO 67575 --- [main] : cmd:set, params:[foo, 6], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.658  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.658  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:15:15.659  INFO 67575 --- [main] : cmd:set, params:[foo, 7], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.660  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.661  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.667  INFO 67575 --- [main] : cmd:set, params:[foo, 8], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.669  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.669  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:15:15.670  INFO 67575 --- [main] : cmd:set, params:[foo, 9], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.672  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:15:15.673  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:15:15.674  INFO 67575 --- [main] : cmd:set, params:[foo, 10], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.675  INFO 67575 --- [main] : cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:15:15.676  INFO 67575 --- [main] : cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]

随机示例如下：

public class AppTest {

    @Autowired
    private CacheTemplateProvider<CacheTemplate> provider;

    public void testRandomTemplate() {
        String key = "foo";

        for (int i = 1; i <= 10; i++) {
            final Optional<CacheTemplate> templateOptional = provider.randomTemplate("ds2");
            if (templateOptional.isPresent()) {
                final CacheTemplate template = templateOptional.get();

                final String str = String.valueOf(i);

                template.set(key, str);

                template.get(key).ifPresent(el -> {
                    assert str.equals(el);
                });

                template.del(key);
            }
        }

    }
    
}

日志打印结果，可以看到配置的redis数据源的出现的次数是随机的：

2024-03-12 12:05:06.450  INFO 66199 --- [main]: cmd:set, params:[foo, 1], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.457  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.461  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.464  INFO 66199 --- [main]: cmd:set, params:[foo, 2], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.465  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.467  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.468  INFO 66199 --- [main]: cmd:set, params:[foo, 3], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.469  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.470  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.472  INFO 66199 --- [main]: cmd:set, params:[foo, 4], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.473  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.475  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]

2024-03-12 12:05:06.475  INFO 66199 --- [main]: cmd:set, params:[foo, 5], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:05:06.476  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]
2024-03-12 12:05:06.477  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=1}]

2024-03-12 12:05:06.478  INFO 66199 --- [main]: cmd:set, params:[foo, 6], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.479  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.481  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.482  INFO 66199 --- [main]: cmd:set, params:[foo, 7], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.484  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.485  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.486  INFO 66199 --- [main]: cmd:set, params:[foo, 8], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.488  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.489  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

2024-03-12 12:05:06.491  INFO 66199 --- [main]: cmd:set, params:[foo, 9], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.492  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]
2024-03-12 12:05:06.493  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=2}]

2024-03-12 12:05:06.494  INFO 66199 --- [main]: cmd:set, params:[foo, 10], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.495  INFO 66199 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]
2024-03-12 12:05:06.496  INFO 66199 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=3}]

获取指定状态标识的数据源

默认情况下数据源的状态标识是读写，还有三个可配状态标识：只读、只写、跳过。

注意：此处的状态只是一种标识，用于筛选获取指定的数据源，并不用于控制redis的读写命令。

数据源配置如下：

redis-keeper:
  redis:
    datasource:
      ds1:
        host: localhost
        port: 6379
        database: 10
        password: xxx
        status: RW
        invoke-params-print: true

测试方法：

public void testTemplateWithStatus() {
        final Optional<CacheTemplate> templateOptional = provider.getTemplate("ds1", CacheDatasourceStatus.RW);

        if (templateOptional.isPresent()) {
            final CacheTemplate cacheTemplate = templateOptional.get();

            String key = "foo";
            String value = "bar";
            cacheTemplate.set(key, value);

            final Optional<Object> optional = cacheTemplate.get(key);

            optional.ifPresent(el -> {
                assert value.equals(el);
            });

            cacheTemplate.del(key);

        }

        assert !provider.getTemplate("ds1", CacheDatasourceStatus.SKIP).isPresent();

    }

访问上面的方法，日志打印如下：

2024-03-12 14:28:34.183  INFO 77230 --- [main]: cmd:set, params:[foo, bar], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12 14:28:34.190  INFO 77230 --- [main]: cmd:get, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]
2024-03-12 14:28:34.193  INFO 77230 --- [main]: cmd:del, params:[foo], connectionInfo:[SingleServer:{Address=redis://localhost:6379,Database=0}]

更多详细示例，请查看：https://github.com/codebaorg/redis-keeper/blob/main/README_zh.md

对比其他方案

如果这篇文章帮助到了你，欢迎评论、点赞、转发。

本文由博客一文多发平台 OpenWrite 发布！

热门相关：全民女神：重生腹黑千金   惊悚乐园   大文豪   都市之九天大帝   带着仓库到大明