Elasticsearch Indexing Failure with ClusterBlockException and Read-only Error Message

Problem: The following error may occur for an Elasticsearch (ES) indexing job:

ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Cause: This is due to a ES 6's mechanism that automatically switches its indices to read-only mode when disk watermark is high. More detailed information can be found here.

Resolution: To work around this issue, clean up the disk mount that ES is running on, and run the following commands:

./stop-unify.sh
./stop-dependencies.sh

./start-dependencies.sh

curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d ' { "index.blocks.read_only_allow_delete": null } '

./start-unify.sh