千家信息网

请输入关键字词

热门搜索排行

最新搜索排行

导航： 首页 > 互联网科技 >

Elastic search performance testing finding

发表于：2024-11-24 作者：千家信息网编辑
千家信息网最后更新 2024年11月24日，Setup1 Node cluster on my local laptop: 8core, Xms=8G, Xmx=8GIndexing performance (Single index):10
千家信息网最后更新 2024年11月24日Elastic search performance testing finding

Setup
1 Node cluster on my local laptop: 8core, Xms=8G, Xmx=8G

Indexing performance (Single index):
10 million payments, each one about 5KB, with batch size = 10000. Each batch takes roughly 2.5 s → 4 s, total time to index 10 million payment is around 50 min

Indexing performance (Multiple indices):
20 separate indices store totally 10 million payments. Indexing execution is slightly faster than single index case. Each batch takes roughly 1.7 s → 3.8 s, total time to index 10 million payment is around 38 min

Parameters required for bulk load operation
Elasticsearch config: http.max_content_length: 500mb

Client time out adjustment:

RestClient.builder(HttpHost("localhost", 9200))        .setRequestConfigCallback {    it.apply {        this.setConnectTimeout(5000)        this.setSocketTimeout(60000)    }}.setMaxRetryTimeoutMillis(60000))

Initially batch size is set to 100000, elastic search server becomes unstable with high GC frequency, occupying a large percent of CPU time. So larger batch size does not always imply higher performance

Query aggregation performance:

Test query: real aggregation query used by rule engine

{  "aggregations": {    "date_range": {      "range": {        "field": "createdAt",        "ranges": [          {            "key": "LAST_7_DAYS",            "from": 1544400968485,            "to": 1545005768486          }        ],        "keyed": false      },      "aggregations": {        "filter_aggregator": {          "filters": {            "filters": {              "602c7d66-e990-4dfb-b6e2-72b62ff159d5": {                "terms": {                  "beneficiaryId.keyword": [                    "602c7d66-e990-4dfb-b6e2-72b62ff159d5"                  ],                  "boost": 1                }              },              "67cab0c8-2510-443d-8f00-bce19c04815e": {                "terms": {                  "bankAccountUserId.keyword": [                    "67cab0c8-2510-443d-8f00-bce19c04815e"                  ],                  "boost": 1                }              },              "8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {                "terms": {                  "payerId.keyword": [                    "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"                  ],                  "boost": 1                }              },              "8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {                "bool": {                  "filter": [                    {                      "terms": {                        "payerId.keyword": [                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"                        ],                        "boost": 1                      }                    },                    {                      "terms": {                        "beneficiaryId.keyword": [                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"                        ],                        "boost": 1                      }                    }                  ],                  "adjust_pure_negative": true,                  "boost": 1                }              },              "9a1b4bad-ccf5-4c67-8718-02696cb351e4": {                "terms": {                  "clientId.keyword": [                    "9a1b4bad-ccf5-4c67-8718-02696cb351e4"                  ],                  "boost": 1                }              },              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {                "bool": {                  "filter": [                    {                      "terms": {                        "clientId.keyword": [                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"                        ],                        "boost": 1                      }                    },                    {                      "terms": {                        "beneficiaryId.keyword": [                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"                        ],                        "boost": 1                      }                    }                  ],                  "adjust_pure_negative": true,                  "boost": 1                }              },              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {                "bool": {                  "filter": [                    {                      "terms": {                        "clientId.keyword": [                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"                        ],                        "boost": 1                      }                    },                    {                      "terms": {                        "payerId.keyword": [                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"                        ],                        "boost": 1                      }                    }                  ],                  "adjust_pure_negative": true,                  "boost": 1                }              },              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {                "bool": {                  "filter": [                    {                      "terms": {                        "clientId.keyword": [                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"                        ],                        "boost": 1                      }                    },                    {                      "terms": {                        "payerId.keyword": [                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"                        ],                        "boost": 1                      }                    },                    {                      "terms": {                        "beneficiaryId.keyword": [                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"                        ],                        "boost": 1                      }                    }                  ],                  "adjust_pure_negative": true,                  "boost": 1                }              }            },            "other_bucket": false,            "other_bucket_key": "_other_"          },          "aggregations": {            "beneficiary_amount": {              "stats": {                "field": "beneficiaryAmountUsd"              }            },            "payer_amount": {              "stats": {                "field": "payerAmountUsd"              }            },            "distinct_count_beneficiary": {              "cardinality": {                "field": "beneficiaryId.keyword"              }            },            "distinct_count_payer": {              "cardinality": {                "field": "payerId.keyword"              }            },            "distinct_count_client": {              "cardinality": {                "field": "clientId.keyword"              }            },            "distinct_count_bank_acc": {              "cardinality": {                "field": "bankAccountUserId.keyword"              }            },            "distinct_count_bene_country": {              "cardinality": {                "field": "beneficiaryCountry.keyword"              }            },            "distinct_count_payer_country": {              "cardinality": {                "field": "payerCountry.keyword"              }            },            "distinct_count_bene_currency": {              "cardinality": {                "field": "beneficiaryCurrency.keyword"              }            },            "distinct_count_payer_currency": {              "cardinality": {                "field": "payerCurrency.keyword"              }            },            "structured_payment_amount_personal": {              "range": {                "field": "payerAmountUsd",                "ranges": [                  {                    "from": 9000,                    "to": 9999.999                  }                ],                "keyed": false              }            },            "structured_payment_amount_company": {              "range": {                "field": "payerAmountUsd",                "ranges": [                  {                    "from": 112500000,                    "to": 124999999.999                  }                ],                "keyed": false              }            }          }        }      }    }  }}

Test result : (Single Index)

ScenarioNumber of runExecution timesMinMaxAverage
Single thread Search result hit Result size unset106s, 5.9s, 5.9s, 6s, 5.9s, 6.2s, 6.2s, 6.2s, 6.6s, 8.2s5.9s8.2s6.93s
Single thread Search result hit Result size = 0107200ms, 12ms, 19ms, 20ms, 23ms, 22ms, 28ms, 21ms, 19ms, 29ms12ms7200ms19.3ms (First execution takes 7.2s, deviates too much from the rest, so exclude it)
Single thread Search result none hit Result size unset10510ms, 529ms, 549ms, 512ms, 489ms, 520ms, 506ms, 500ms, 493ms, 499ms489ms549ms510.7ms
Single thread Search result none hit Result size = 010389ms, 1ms, 2ms, 3ms, 3ms, 1ms, 1ms, 1ms, 2ms, 3ms389ms1ms1.89ms (First execution takes 389ms, deviates too much from the rest, so exclude it)
20 threads Search result hit Result size unset10Exception, listener timeout after waiting for [60000] ms
20 threads Search result hit Result size = 01018ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms17ms82ms43.6ms
20 threads Search result none hit Result size unset104.4s, 5.8s, 4.9s, 5.6s 4.6s, 5.4s, 5.3s 4.7s, 5.8s, 4.7s5.8s4.4s5.1s
20 threads Search result none hit Result size = 0101.4s, 25ms, 6ms, 22ms, 276ms, 250ms, 19ms, 30ms, 11ms, 18ms1.4s11ms73ms (First execution takes 1.4s, deviates too much from the rest, so exclude it)

Test result: (Multiple Index)

ScenarioNumber of runExecution timesMinMaxAverage
Single thread Search result hit Result size unset1012.6s, 12.7s 12.6s 12.7s, 13.1s, 13.1s, 13.2s, 13,1s, 13.1s, 13.1s12.6s13.2s12.9s
Single thread Search result hit Result size = 010212ms, 147ms, 170ms, 272ms, 223ms, 222ms, 207ms, 255ms, 219ms, 219ms147ms272ms214.6ms
Single thread Search result none hit Result size unset101.2s, 1.1s, 1.1s, 1.1s, 1.1s, 1.2s, 1.1s, 1.1s, 1.2s, 1.1s1.1s1.2s1.13s
Single thread Search result none hit Result size = 010909ms, 23ms, 23ms, 35ms, 14ms, 26ms, 19ms, 12ms, 15ms, 22m12ms909ms18.9ms (First execution takes 909ms, deviates too much from the rest, so exclude it)
20 threads Search result hit Result size unset10Exception, listener timeout after waiting for [60000] ms
20 threads Search result hit Result size = 01018ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms17ms93ms43.6ms
20 threads Search result none hit Result size unset107.8s, 7.6s, 7.1s, 6.9s, 7.4s, 8s, 7.3, 7.2s, 7.3s, 7.4s6.9s8s7.4s
20 threads Search result none hit Result size = 010202ms, 91ms, 100ms, 146ms, 182ms, 131ms, 158ms, 48ms, 171ms, 152ms48ms202ms138.1ms

Conclusion:
Aggregation performance hinges on the number of documents that matches the aggregation

Result size parameter has significant impact on aggregation performance. not only because it skipped returning hit documents, but also because it enables caching for aggregation result, otherwise, you have to force result caching by explicitly setting request_cache=true

https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-request-cache.html

Executing query concurrently can also have negative impact on performance

Increasing number of indices have positive impact on index speed but have large negative impact on aggregation if the aggregation is performed across indices

很赞哦！
数据库的安全要保护哪些东西 数据库安全各自的含义是什么 生产安全数据库录入 数据库的安全性及管理 数据库安全策略包含哪些 海淀数据库安全审计系统 建立农村房屋安全信息数据库 易用的数据库客户端支持安全管理 连接数据库失败ssl安全错误 数据库的锁怎样保障安全 存储服务器硬盘绿灯闪烁和常亮 更新数据库表时间戳 软件开发项目奖金分配公式 软件开发外包比较 网络安全影响因素及防范 访问sql数据库对象 网络安全周活动总结关于工业企业 软件开发资金用在哪里 河南服务器硬盘性能 甘肃肃南网络安全宣传 怎么看邮箱服务器 互联网科技特派员申论 凌源网络技术公司 如何查看微信云数据库数据长度 飞机 服务器 云服务器安全组没有gre协议 湖南美音网络技术有限公司微信 怎么看服务器的物理地址 网络安全法制度在整个 删除数据库表中的相同的数据 数据库服务器的性能指标 传销网络安全知识竞赛 计算机有哪些软件开发 服务器总是 网络安全扫描的内容 微信和网络安全作文 安全关联数据库在哪 广州新媒体网络技术 网络安全领域专项整治行动 计算机网络技术万晓明教材答案

相关文章

0