布隆过滤器是一种数据结构，由一个很长的二进制位数组和一系列随机映射函数组成，用于判断某个key一定不存在或者可能存在于某个集合中，它在存储空间占用和查询效率上非常有优势，但是存在一定的误算率，数据越多误算率越高。

当使用布隆过滤初始化一个key时，先对key使用一系列的映射函数映射后得出它在二进制数组中的一系列位置，然后把这些位置上的值设置为1。当进行查找时，同样对key使用一系列的映射函数进行映射得出它在数组中的一系列位置，如果这些位置上存在为0的值说明key一定不在集合中，如果这些位置上的值都是1说明key可能存储于集合中，但是不一定存在于集合中，因为有可能别的key把这些位置上的值设置成了1。

常见的布隆过滤器有goole的布隆过滤器、Redis布隆过滤器，google布隆过滤器使用的是JVM内存。

google布隆过滤器

<!--google bloom filter--><dependency><groupId>com.google.guava</groupId><artifactId>guava</artifactId><version>20.0</version></dependency>

package com.tech.tech.redis.filter.service;import mon.hash.BloomFilter;import mon.hash.Funnels;import com.tech.tech.redis.filter.entity.SysUser;import lombok.extern.slf4j.Slf4j;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.stereotype.Service;import org.springframework.util.CollectionUtils;import javax.annotation.PostConstruct;import java.util.List;/*** @author lw* @since /12/20*/@Slf4j@Servicepublic class BloomFilterService {//google bloom filter 基于JVM内存的布隆过滤器private BloomFilter<Long> bloomFilter;@AutowiredISysUserService sysUserService;@PostConstructvoid init(){List<SysUser> list = sysUserService.list();if(CollectionUtils.isEmpty(list)){return;}bloomFilter=BloomFilter.create(Funnels.longFunnel(),list.size());list.forEach(sysUser -> {bloomFilter.put(sysUser.getId());});log.info("init bloom filter OK");}public boolean userIdExists(Long id){return bloomFilter.mightContain(id);}}

package com.tech.tech.redis.filter.controller;import com.tech.tech.redis.filter.service.BloomFilterService;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.RestController;/*** <p>* 前端控制器* </p>** @author lw* @since -12-20*/@RestController@RequestMapping("/filter/sysUser")public class SysUserController {@AutowiredBloomFilterService bloomFilterService;/*** google bloom filter* @param id* @return*/@GetMapping("id")Boolean id(long id){return bloomFilterService.userIdExists(id);} }

在程序启动时，在数据库中查询出用户列表，根据全部的用户ID，使用布隆过滤器进行数据初始化。当访问接口时，根据传入的用户ID，使用布隆过滤器进行判断是否为数据库中的用户ID，如果返回false，表示肯定不是真实的用户ID,可以进行业务拦截处理；如果返回true，表示很可能为真实的用户，可以进行查询数据库。通过布隆过滤器，过滤掉大部分伪造的非法请求，通过布隆过滤器校验通过的请求再访问数据库，做核实，防止伪造大量请求访问数据库。

Redis布隆过滤器

Redis布隆过滤器，在早期版本Redis需要用BitMap实现，新版本Redis可以通过加载相关插件提供布隆过滤器功能。

下载插件git clone /RedisBloom/RedisBloom.git

RedisBloom-2.2.zip

解压后进入解压包进行编译

make

修改Redis的配置加载插件redisbloom.so

loadmodule /path/to/redisbloom.so

重启Redis

#使用redis-cli客户端进行操作：#往过滤器添加一个keyBF.ADD bloom redis#判断过滤器是否存在key redisBF.EXISTS bloom redis

springboot访问Redis布隆过滤器

bloomFilterAdd.lua 添加数据到布隆过滤器

------ Generated by EmmyLua(/EmmyLua)--- Created by lw.--- DateTime: /12/21 10:21---local bloomName = KEYS[1]local value = KEYS[2]local result_1 = redis.call('BF.ADD',bloomName,value)return result_1

bloomFilterExist.lua 判断数据是否一定不存在或可能存在

------ Generated by EmmyLua(/EmmyLua)--- Created by lw.--- DateTime: /12/21 10:24---local bloomName = KEYS[1]local value = KEYS[2]local result_1 = redis.call('BF.EXISTS',bloomName,value)return result_1

RedisTemplate配置，防止乱码

package com.tech.tech.redis.filter.config;import com.fasterxml.jackson.annotation.JsonAutoDetect;import com.fasterxml.jackson.annotation.PropertyAccessor;import com.fasterxml.jackson.databind.ObjectMapper;import org.springframework.cache.CacheManager;import org.springframework.cache.annotation.EnableCaching;import org.springframework.cache.interceptor.KeyGenerator;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.data.redis.cache.RedisCacheConfiguration;import org.springframework.data.redis.cache.RedisCacheManager;import org.springframework.data.redis.cache.RedisCacheWriter;import org.springframework.data.redis.connection.RedisConnectionFactory;import org.springframework.data.redis.core.RedisTemplate;import org.springframework.data.redis.core.StringRedisTemplate;import org.springframework.data.redis.serializer.*;import javax.annotation.Resource;import java.time.Duration;import java.util.HashMap;import java.util.Map;@Configuration@EnableCachingpublic class RedisConfig {// @Bean// public RedisTemplate<String,String> redisTemplate(RedisConnectionFactory factory){// RedisTemplate<String,String> redisTemplate = new RedisTemplate<>();// redisTemplate.setConnectionFactory(factory);// return redisTemplate;// }@Beanpublic RedisTemplate<String, String> redisTemplate(RedisConnectionFactory factory){RedisTemplate<String, String> redisTemplate = new RedisTemplate<String,String>();redisTemplate.setConnectionFactory(factory);// 使用Jackson2JsonRedisSerialize 替换默认序列化/**Jackson序列化 json占用的内存最小 */Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);ObjectMapper om = new ObjectMapper();om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);jackson2JsonRedisSerializer.setObjectMapper(om);/**Jdk序列化 JdkSerializationRedisSerializer是最高效的*///JdkSerializationRedisSerializer jdkSerializationRedisSerializer = new JdkSerializationRedisSerializer();/**String序列化*/StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();/**将key value 进行stringRedisSerializer序列化*/redisTemplate.setKeySerializer(stringRedisSerializer);redisTemplate.setValueSerializer(stringRedisSerializer);/**将HashKey HashValue 进行序列化*/redisTemplate.setHashKeySerializer(stringRedisSerializer);redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);redisTemplate.afterPropertiesSet();return redisTemplate;}@Beanpublic KeyGenerator genValueKeyGenerator() {return (o, method, objects) -> {StringBuilder stringBuilder = new StringBuilder();stringBuilder.append(o.getClass().getSimpleName());stringBuilder.append(".");stringBuilder.append(method.getName());stringBuilder.append("[");for (Object obj : objects) {stringBuilder.append(obj.toString());}stringBuilder.append("]");return stringBuilder.toString();};}@Beanpublic CacheManager cacheManager(RedisConnectionFactory redisConnectionFactory) {return new RedisCacheManager(RedisCacheWriter.nonLockingRedisCacheWriter(redisConnectionFactory),this.getRedisCacheConfigurationWithTtl(600), // 默认策略，未配置的 key 会使用这个this.getRedisCacheConfigurationMap() // 指定 key 策略);}private Map<String, RedisCacheConfiguration> getRedisCacheConfigurationMap() {Map<String, RedisCacheConfiguration> redisCacheConfigurationMap = new HashMap<>();redisCacheConfigurationMap.put("UserInfoList", this.getRedisCacheConfigurationWithTtl(100));redisCacheConfigurationMap.put("UserInfoListAnother", this.getRedisCacheConfigurationWithTtl(18000));return redisCacheConfigurationMap;}private RedisCacheConfiguration getRedisCacheConfigurationWithTtl(Integer seconds) {Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer<>(Object.class);ObjectMapper om = new ObjectMapper();om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);jackson2JsonRedisSerializer.setObjectMapper(om);RedisCacheConfiguration redisCacheConfiguration = RedisCacheConfiguration.defaultCacheConfig();redisCacheConfiguration = redisCacheConfiguration.serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(jackson2JsonRedisSerializer)).entryTtl(Duration.ofSeconds(seconds));return redisCacheConfiguration;}}

package com.tech.tech.redis.filter.controller;import com.tech.tech.redis.filter.service.BloomFilterService;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.core.io.ClassPathResource;import org.springframework.data.redis.core.RedisTemplate;import org.springframework.data.redis.core.script.DefaultRedisScript;import org.springframework.scripting.support.ResourceScriptSource;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.RestController;import java.util.ArrayList;import java.util.List;/*** <p>* 前端控制器* </p>** @author lw* @since -12-20*/@RestController@RequestMapping("/filter/sysUser")public class SysUserController {@AutowiredBloomFilterService bloomFilterService;@Autowiredprivate RedisTemplate redisTemplate;private static final String bloomFilterName="userIdBloomFilter";/*** google bloom filter* @param id* @return*/@GetMapping("id")Boolean id(long id){return bloomFilterService.userIdExists(id);}/*** redis布隆过滤器 添加数据到布隆过滤器* @param id* @return*/@GetMapping("redis/idAdd")Boolean redisIdAdd(int id){DefaultRedisScript<Boolean> script = new DefaultRedisScript<>();script.setScriptSource(new ResourceScriptSource(new ClassPathResource("bloomFilterAdd.lua")));script.setResultType(Boolean.class);List<Object> keyList = new ArrayList<>();keyList.add(bloomFilterName);keyList.add(String.valueOf(id));Boolean res = (Boolean) redisTemplate.execute(script, keyList);return res;}/*** redis 布隆过滤器 判断数据是否一定不存在或者可能存在* @param id* @return*/@GetMapping("redis/idExist")Boolean redisIdExist(int id){DefaultRedisScript<Boolean> script = new DefaultRedisScript<>();script.setScriptSource(new ResourceScriptSource(new ClassPathResource("bloomFilterExist.lua")));script.setResultType(Boolean.class);List<Object> keyList = new ArrayList<>();keyList.add(bloomFilterName);keyList.add(String.valueOf(id));Boolean res = (Boolean) redisTemplate.execute(script, keyList);return res;}}