Enrich Search Results with Personalized Sorting in Elasticsearch

Greetings!

Personalization is a key e-commerce feature for enhancing user experience and driving sales. One aspect of this is improving search results through recommendations, helping to surface more buyable products for the customer.

Business opportunities

Boost sales by sorting search results based on customer recommendations.

Technical opportunity

A business opportunity often comes with fascinating technical challenges. While Elasticsearch makes sorting easier, this doesn’t seem like something we can achieve with standard sorting alone (but we can).

Sort by script

Elasticsearch offers multiple search options, including custom scripts that allow us to define our own sorting logic.

  "sort": {
    "_script": {
      "type": "number",
      "script": {
        "lang": "painless",
        "inline": "doc['field_name'].value * params.factor",
        "params": {
          "factor": 1.1
        }
      },
      "order": "asc"
    }
  }

Painless is a simple, secure scripting language designed specifically for use with Elasticsearch. It is the default scripting language for Elasticsearch and can safely be used for inline and stored scripts. [Elasticsearch documentation]

A personalized movie sorting

Since movies are a common and easy-to-understand data type, let's build a movie search system that can sort results based on custom recommendations. For this example, let's assume we have a wishlist and want those movies to appear first. When a customer sorts movies by their wishlist, those movies will be prioritized in the results.

PUT /sort-movies
{
  "mappings": {
    "properties": {
      "imdbid":    { "type": "keyword" },  
      "title":  { "type": "text"  }, 
      "genre":   { "type": "keyword"  },
      "year": { "type": "integer" },
      "rating": { "type": "float" },
      "language": { "type": "keyword" }
    }
  }
}

POST _bulk
{ "index": { "_index": "sort-movies", "_id": "tt0167260" } }
{ "imdbid": "tt0167260", "title": "The Lord of the Rings: The Return of the King", "genre": ["Adventure", "Action", "Drama"], "year": 2003, "rating": 8.9, "language": "English" }
{ "index": { "_index": "sort-movies", "_id": "tt0167261" } }
{ "imdbid": "tt0167261", "title": "Kenshin", "genre": ["Action", "Adventure"], "year": 2021, "rating": 7.8, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt8367810" } }
{ "imdbid": "tt8367810", "title": "The Great Battle", "genre": ["Action", "War"], "year": 2018, "rating": 7.1, "language": "Korean" }
{ "index": { "_index": "sort-movies", "_id": "tt0167263" } }
{ "imdbid": "tt0167263", "title": "Planet of the Apes", "genre": ["Sci-Fi", "Adventure"], "year": 2001, "rating": 8.5, "language": "English" }
{ "index": { "_index": "sort-movies", "_id": "tt0167264" } }
{ "imdbid": "tt0167264", "title": "Baahubali", "genre": ["Action", "Fantasy", "Drama"], "year": 2015, "rating": 8.1, "language": "Telugu" }
{ "index": { "_index": "sort-movies", "_id": "tt0167265" } }
{ "imdbid": "tt0167265", "title": "Seven Samurai", "genre": ["Drama", "Action", "Adventure"], "year": 1954, "rating": 8.6, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt0167266" } }
{ "imdbid": "tt0167266", "title": "Andhadhun", "genre": ["Thriller", "Mystery", "Comedy"], "year": 2018, "rating": 8.2, "language": "Hindi" }
{ "index": { "_index": "sort-movies", "_id": "tt0167267" } }
{ "imdbid": "tt0167267", "title": "The Lord of the Rings: The Two Towers", "genre": ["Adventure", "Action", "Drama"], "year": 2002, "rating": 8.7, "language": "English" }
{ "index": { "_index": "sort-movies", "_id": "tt1979319" } }
{ "imdbid": "tt1979319", "title": "Rurouni Kenshin", "genre": ["Action", "Adventure"], "year": 2012, "rating": 7.4, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt3029556" } }
{ "imdbid": "tt3029556", "title": "Rurouni Kenshin: Kyoto Inferno", "genre": ["Action", "Adventure"], "year": 2014, "rating": 7.5, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt3029630" } }
{ "imdbid": "tt3029630", "title": "Rurouni Kenshin: The Legend Ends", "genre": ["Action", "Adventure"], "year": 2014, "rating": 7.6, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt9314288" } }
{ "imdbid": "tt9314288", "title": "Rurouni Kenshin: The Final", "genre": ["Action", "Drama"], "year": 2021, "rating": 7.2, "language": "Japanese" }
{ "index": { "_index": "sort-movies", "_id": "tt10758050" } }
{ "imdbid": "tt10758050", "title": "Rurouni Kenshin: The Beginning", "genre": ["Action", "Drama"], "year": 2021, "rating": 7.4, "language": "Japanese" }

GET /sort-movies/_search
{
  "from": 0,
  "size": 20,
  "query": {
    "bool": {
      "must": [],
      "filter": []
    }
  },
  "sort": []
}

Building the custom sorting

First, we need to define the sorting order. I want these movies to appear first.

Rurouni Kenshin (tt1979319)
The Great Battle (tt8367810)
Baahubali (tt0167264)
Andhadhun (tt0167266)

We can assign a number to each movie in increasing order and use the document to compare the current movie with the predefined order.

"custom_order": {
    "tt1979319": 0,
    "tt8367810": 1,
    "tt0167264": 2,
    "tt0167266": 3
}

Thus, our complete search query would look like this.

GET /sort-movies/_search
{
  "from": 0,
  "size": 10,
  "sort": [
    {
      "_script": {
        "type": "number",
        "order": "asc",
        "script": {
          "lang": "painless",
          "params": {
            "custom_order": {
              "tt1979319": 0,
              "tt8367810": 1,
              "tt0167264": 2,
              "tt0167266": 3
            }
          },
          "source": """
            return params.custom_order.containsKey(doc['imdbid'].value) ? params.custom_order[doc['imdbid'].value] : 1000;
          """
        }
      }
    }
  ]
}

A wishlist page

Now, imagine we offer customers a wishlist page. Naturally, the best sorting option would be the wishlist order. The challenge, then, is to preserve this order. In this scenario, the search term consists of IDs or IMDb IDs. Note: A real word scenario would be to provide recommended page preserving the order.

GET /sort-movies/_search
{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "should": [
        {
          "terms": {
            "imdbid": [
              "tt1979319",
              "tt8367810",
              "tt0167264",
              "tt0167266"
            ]
          }
        }
      ]
    }
  },
  "sort": [
    {
      "_script": {
        "type": "number",
        "order": "asc",
        "script": {
          "lang": "painless",
          "params": {
            "custom_order": {
              "tt1979319": 0,
              "tt8367810": 1,
              "tt0167264": 2,
              "tt0167266": 3
            }
          },
          "source": """
      return params.custom_order.containsKey(doc['imdbid'].value) ? params.custom_order[doc['imdbid'].value] : 1000;
     """
        }
      }
    }
  ]
}

Conclusion

In this article, we discussed how to personalize search sorting using Elasticsearch scripts. While there are multiple solutions to the same problem, this experience provided me with new insights and an opportunity to explore Elasticsearch in real business scenarios.

References

Script_based_sorting
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html

Manju