Magento and Elasticsearch REST API: The must-have developers’ toolset

Magento 2 is heavily integrated with the Elasticsearch engine. It uses Elasticsearch for searches, catalog lists, and layered navigation. Every product or category is stored as an Elasticsearch document. Documents are stored in the Elasticsearch index and indexes are stored in the Elasticsearch database. Magento queries Elasticsearch and fetches those documents for further usage. Please check the Elastic stack tutorial to learn more about these concepts.

Thus, Magento developers have to utilize Elasticsearch, understand its query language, and debug the requests or responses that come to or from it. To do this effectively, we would like to suggest the several tools that we have found very useful during the development of our Elasticsearch Magento 2 extension and running a large Elasticsearch cluster in production.

Command Line. REST Requests

Elasticsearch has a great REST API. Almost all necessary information and most operations can be done using this API. It’s fully described in the official documentation. Generally, to use Elasticsearch REST API, you need to send an HTTP request to Elasticsearch. For example, the following request will show the status of the cluster:

 curl -X GET "localhost:9200/_cluster/health"

{"cluster_name":"elasticsearch_brew","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":9,"active_shards":9,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":8,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":52.94117647058824}

If you would like to receive a human-friendly response, add the GET parameter “pretty” to any request URL.

 curl -X GET "localhost:9200/_cluster/health?pretty"

{
  "cluster_name" : "elasticsearch_brew",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 9,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 8,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 52.94117647058824
}

Elasticsearch architecture allows you to send requests to one node of the cluster and get information about all other nodes or the cluster.

Everything is simple, and that's the power of Elasticsearch.

Kibana

REST requests via the command line are enough to solve most tasks. However, from our experience, if we need to debug some unique problems (for example, when we receive an incorrect search result for a particular query), it’s much more comfortable to use a Kibana in running these requests to Elasticsearch. Kibana is an official product of the Elastic company - the developer of Elasticsearch. The main goal of this product is to visualize different data, show dashboards, and graphics. For our tasks, we use only one utility inside this product - Dev Console. This tool is an Elasticsearch REST client. And it’s great.

For example, this is how the previous query will appear inside Kibana:

Also, Kibana has some very useful options; they are:

• Copy as cURL - allows you to convert any request from Elasticsearch query to cURL request for a command line.
• Auto indent - allows you to format large requests smoothly, so it will be much easier to understand and debug them.

Note: You have to use the same version of Kibana as your Elasticsearch. For example, if you use Elasticsearch 7.4.x, you must have Kibana version 7.4.x. If you use Elasticsearch's latest version, you should also use the latest kibana version. Otherwise, it will not work correctly. For more details, please check out the compatibility matrix.

Elasticsearch Head

Elasticsearch Head is a Google Chrome plugin. It’s useful during day-to-day operations with the Elasticsearch cluster. You can connect to a cluster and see the cluster's state in different parameters in just one glance. You can manage the cluster and check the parameters of every server, node, shard, etc. It's a really powerful tool in managing the Elasticsearch clusters. We highly recommend it.

It also allows you to make your REST queries to Elasticsearch. However, from our opinion, the interface of Kibana is more useful for this task.

Summary

Magento developers have to master their skills with Elasticsearch. As any tool, you can use a basic toolset, or if you would like to improve your performance, you can invest some time to explore more advanced tools, which will more than give back said investment once you have mastered them. The analogy is simple; you can use Notepad to write code or use a modern IDE like PHPStorm.

We've shared our toolset to work with Elasticsearch. It’s possible that from time to time, we will expand this article and add more tools. Meanwhile, please share your favorite tools or tips about Elasticsearch in the comments below.