What is Elasticsearch and how is Tamr using it?

Elasticsearch is an open-source, RESTful, search engine based on the Apache Lucene library. Initially released in 2010 by Elastic, Elasticsearch was designed as a distributed Java solution for bringing full-text search functionality into schema-free JSON documents across multiple database types. It is packaged with Tamr which enables usage of the search engine to search for datasets, records, clusters, and so on in Tamr.

Why you may want to use the disable/enable setting?

The ability to disable/enable Elasticsearch on a per-project basis helps when multiple projects are hosted in the same environment with some of them being in training mode and some in production mode. This is usually the case in large / scale-out deployments.

How to use it?

You can enable/disable Elasticsearch (ES) for a specific project via API calls.

This feature adds a new project metadata key named “{{disableEs}}”. This key is not present by default which means that ES is enabled for each project. If this key is associated with true, ES will be disabled for this project.

To disable ES for a specific project id, use the following endpoint:

POST /api/recipe/projects/disableEs/{id}

To enable ES for a specific project id, you can use this endpoint:

POST /api/recipe/projects/enableEs/{id}

Note that Tamr config {{TAMR_ES_ENABLED}} still exists. If this config is false, all projects will have ES disabled, regardless of the value of {{disableEs}}.

Note that when ES is disabled for a project, enabling ES afterward requires running,

POST /reindex/all-datascale

and

POST /reindex/all-humanscale

in order to synchronize ES and storage.