Can't reduce index size after deleting content

cancel
Showing results for 
Search instead for 
Did you mean: 
gabrielmarinb
Active Member

Can't reduce index size after deleting content

Hi:

I started performing massive deletion (during the day) in my repository. Every night a massive content ingestion is performed.

Index size was growing in about 1 to 1.5 GB daily after ingestion and now that I am deleting (almost the same quantity of documents) the growth rate remain the same. I see and increasing deleted docs number but size used is increasing at the same pace before deleting.

Solr adminSolr admin

What should I do in order to free up index space?

 

Thanks a lot for your advice.

3 Replies
cesarista
Customer

Re: Can't reduce index size after deleting content

Hi:

Probably, you are leaving your content in Thrash when deleting. 

If it is not the case, you may optimize index, for cleaning those deleting docs. This is in fact a recommended maintainance task for SOLR.

When a document is deleted in SOLR, the index of this document is flagged as deleted, but it still occupies size in disk. A big number of deleted docs in SOLR indices may impact in search performance. 

Regards.

--C.

gabrielmarinb
Active Member

Re: Can't reduce index size after deleting content

Deletion process is performed using REST API with permanent=True parameter.

A couple of question regarding Optimization. Can you run optimization during normal operation (indexation/searching)?

Do you have an estimation of how long this process take? We have a dedicated solr server (16 core - 32 GB)

I also read that double the size used is needed in the file system where indexes are stored. 

 

Thanks a lot for your guidance

cesarista
Customer

Re: Can't reduce index size after deleting content

Hi:

Can you run optimization during normal operation (indexation/searching)?

 - No. Manteinance normally means out of office hours.

Do you have an estimation of how long this process take? We have a dedicated solr server (16 core - 32 GB)

 - Can't say. Not so much comparing with a full reindex for example. It depends on the resources for SOLR, also how fast are your indices disks, how big are your indices... 

I also read that double the size used is needed in the file system where indexes are stored. 

 - Yes, this refers for the (SSD) disk where indices are stored. When SOLR is doing merges internally needs extra space.

Regards.

--C.