已解决:Elasticsearch报错:[reason=Fielddata is disabled on text fields by default]
前言使用AggregationBuilders.terms("name").field("field")做分组统计:// 根据action分组统计总数AggregationBuilder actionAggregation = AggregationBuilders.terms("actionCount").field("action");// cardinality相当于去重计算Cardinal
前言
使用AggregationBuilders.terms("name").field("field")
做分组统计:
@Test
public void getData() {
// 根据action分组统计总数
AggregationBuilder actionAggregation = AggregationBuilders.terms("actionCount").field("action");
// cardinality相当于去重计算
CardinalityAggregationBuilder uniqueUser = AggregationBuilders.cardinality("uniqueUser").field("userid");
actionAggregation.subAggregation(uniqueUser);
// 创建查询请求实例
SearchRequest searchRequest = new SearchRequest("test-index");
searchRequest.source(SearchSourceBuilder.searchSource().aggregation(actionAggregation));
try {
// 使用RestHighLevelClient发送查询请求
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
long totalHits = searchResponse.getHits().totalHits;
log.info("从es命中的总数: {}", totalHits);
Terms actionTerms = searchResponse.getAggregations().get("actionCount");
for (Terms.Bucket bucket : actionTerms.getBuckets()) {
String action = bucket.getKeyAsString();
log.info("bucket key: {}", action);
long docCount = bucket.getDocCount();
log.info("bucket doc count: {}", docCount);
Cardinality uniqueUser1 = bucket.getAggregations().get("uniqueUser");
long value = uniqueUser1.getValue();
log.info("distinct doc count: {}", value);
}
} catch (IOException e) {
e.printStackTrace();
log.error("从es统计数据失败: {}", e.getMessage());
}
}
报错信息如下
ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [deviceid] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [userid] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]];
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2030)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092)
Caused by: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [deviceid] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [deviceid] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]];
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:509)
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:420)
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:450)
at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:616)
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169)
从报错信息Caused by: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [userid] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]];
中可以看出:在分组统计时,默认情况下text类型的Fielddata是被禁用的,可以在字段上面设置fielddata = true
,通过取消倒排索引把fielddata加载到内存,但是这个可能会使用大量的内存,应该使用keyword类型来替代。
简单的说, 在elasticsearch中对text类型的字段进行聚合、排序时,会产生上面的错误
解决方法一:把要聚合的字段加上".keyword"就可以了
我们可以用fieldname.keyword
进行聚合,排序,代码修改为如下:
// 去重聚合
AggregationBuilders.cardinality("uniqueUser").field("userid.keyword");
解决方法二:设置字段的"fielddata=true"(不推荐,这样可能会使用大量的内存)
使用如下的命令设置:
PUT /test-index/_mapping/doc
{
"properties": {
"userid": {
"type": "text",
"fielddata": true
}
}
}
设置成功返回:
{
"acknowledged" : true
}
开放原子开发者工作坊旨在鼓励更多人参与开源活动,与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动,如meetup、训练营等,主打技术交流,干货满满,真诚地邀请各位开发者共同参与!
更多推荐
所有评论(0)