1.5 Elasticsearch Optimization

1. Design Tuning

• During the mapping-creation stage:

  • Design fields reasonably. Use Java to encapsulate complex operations and write directly into ES. Avoid complex operations such as join, nested, parent-child, etc.
  • Use the smallest appropriate data type: byte, short, integer, long. If the smallest type is sufficient, use the smallest type. Use keyword whenever possible.
  • Configure a reasonable analyzer.
  • Configure field properties appropriately, such as whether a field needs to be searchable or stored.

• Index management:

  • Use aliases for index management.
  • Create indices based on date templates and roll indices through the rollover API.
  • Run force_merge on indices on a schedule every night to release space.
  • Use hot/cold separation. Store hot data on SSD to improve search efficiency; place cold data separately and periodically run shrink operations to reduce storage usage.
  • Use Curator for index lifecycle management.

• Cache warming:

  • For some hot data, we can write a cache-warming system that accesses it periodically and loads it into the OS cache.

2. Write Tuning

• Use automatically generated IDs whenever possible.
• Before writing, disable replica and refresh: set replica shards to 0; disable refresh by setting refresh_interval to -1.
• Use bulk writes during ingestion.
• Use multiple threads to write data into ES.
• Restore the replica count and refresh interval after writing.

3. Query Tuning

• When the amount of data is large, first determine the index based on time and then search it.
• Pagination-performance optimization:
  • Deep pagination is inefficient. For example, with 10 records per page, querying page 100 means querying records 990-1000. ES reads the top 1000 records from every shard into the coordinating node. With 5 shards, that means 5000 records. It merges and processes those 5000 records and then obtains the actual 10 records for page 100. Elasticsearch CRUD Flow
  • Scroll search can be used instead. Its principle is to save a snapshot of the current data. The disadvantage is that pages can only be traversed one by one downward, similar to scrolling down a Weibo feed. You cannot jump arbitrarily between pages, otherwise efficiency becomes even worse.
• Disable wildcard and large batches of terms (hundreds or thousands).
• Configure a reasonable routing mechanism.

4. Deployment Tuning

• Linux
  • OS Cache
    • ES first queries data from the operating system cache and then reads from disk when the data is not found, so system memory should be as large as possible.
    • In general, memory should be at least half of the data size. For example, if ES data is 1 TB, memory should be at least 512 GB.
    • ES stores only a small number of fields used for querying; fields used for display are queried from HBase or MySQL.
  • Disk-storage RAID: use RAID10 when storage conditions allow, to improve single-node performance and avoid single-node storage failures.
  • Configure Linux so that at least 2048 threads can be created: ulimit -u 2048.
  • Configure the maximum virtual memory: sysctl -w vm.max_map_count=262144.
  • Configure the maximum number of file descriptors: ulimit -n 65536.
• ES
  • Disable swapping: bootstrap.memory_lock: true.
  • Set the minimum and maximum heap memory to Min(system memory / 2, 32GB).
  • Add nodes at any time for dynamic horizontal scaling.

5. References

• 系统设置
• ES索引写入性能优化_数据库_Jimmy的专栏-CSDN博客
• ES搜索性能优化_数据库_Jimmy的专栏-CSDN博客
• elasticsearch 优化写入速度 | easyice
• Elasticsearch：从写入原理谈写入优化 - 知乎