Elastic Search daily index size view

I could never find a decent way to forecast the index sizes in Elastic Search. In the Kibana GUI, under Stack Management, you can see the total index size which you need to divide by the number of nodes that the index data is stored on, and that gives you an idea, but you can’t visualise it.

So, to get some sort of rough average, I wrote some python code to do what i needed. The following is done.

  • Pick an index
  • Work out the average size of a document in that index
  • Count the number of documents in the previous day
  • daily index size = (number of docs that day) x (average doc size)

It’s not 100% but it’s going to allow you to see the index sizes and forecast some trends.

A docker image

My Elastic stack is running in docker, so an image is included with a docker-compose file, but the python code can be run on where ever you want.

https://github.com/mohclips/elastic_index_sizer

https://github.com/mohclips/elastic_index_sizer/blob/main/src/get_size.py

So you can run the code as often as you want and create a visualisation in Kibana to see the results.

Results

So after a few days running, i can see we are ingesting 9GB a day and also which index is ingesting the most often.

So the green index is the one that is ingesting more data each day.

About mohclips

Born in the UK, educated in NZ, working for a tier 1 multinational
This entry was posted in Automation, Containers, DevOps. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.