Setting up ILM policies for App Search in Elastic Search

At SpotDraft we use App Search on top of Elastic Search to give our users fast and reliable search. Search is at the core of Contract Management, as we let users search by various AI extracted and manually inputted Key Points on each contract. In this article we discuss how to manage App Search Indexes in Elastic Search.

Identifying the Reason for High Memory Usage

We started seeing issues around high JVM memory load and had 1000s of indexes even though the data size was only in the 100K documents range. Since we were a paying user of Elastic Cloud, we reached out to their support and were told that we had too many shards and we should get consulting support from them to improve this (or fix it ourselves). 

Following their index lifecycle management and ILM documentation, we realized that the issue was that there were a lot of app search logs that were saved causing this heavy memory and disk usage. However there isn’t much documentation online on how to manage indexes for App Search.

Since we were using Elastic.co we can go to Kibana to see what indexes are in the system. Here we can clearly see that a lot of app-search-app-logs-* indexes are present and taking up a lot of room.

We can even drill down to each index and what the creation date is and the stats around when it was last queried and written to

Setting up ILM Policy Using Kibana

Since we already ship logs elsewhere, we didn’t need to save these logs in elastic. So we can set up an ILM policy to auto delete them after 30 days. Thankfully App Search creates a index template named “app-search-logs” which we can use for this.

However this only applies to new indexes. In order to update the existing indexes, we can use the Elastic Search console in Kibana.

‍

Setting up ILM Policy without Kibana

Same can also be done without Kibana using curl

curl -X PUT "localhost:9200/_ilm/policy/delete-after-15-days?pretty" \
  -H 'Content-Type: application/json' \
  -d'{"policy": {"phases": {"delete": {"min_age": "15d","actions": {"delete": {}}}}}}'


PUT .app-search-app-logs-*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "delete-after-15-days"
    }
  }
}


PUT .app-search-analytics-logs-*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "delete-after-15-days"
    }
  }
}

After making this change our Elastic Search memory usage dropped by 50% and the latency also dropped by 40%.

Happy searching!