Exploring Enterprise Search Solution Critical Capabilities

In this series of blog posts we will review the enterprise search solution capabilities, available software solutions, from basic to advanced search capabilities along with using AI/ML models to achieve both consumer and business value. We will also cover the search solution for healthcare domain common user interactions, domain dataset and ML alignments. In this post we will cover the critical enterprise search solution capabilities on high level.

History

A long has changed in the search solution capabilities since the initial blog posts shared here in this blog for ElasticSearch capabilities. Additionally using Elasticsearch for user behavior clickstream data with Hadoop big data capabilities to process those information. A quick glance at those here,

ElasticSearch

Getting used to Elastic search and having hands directly with different tricks,

Getting started with ElasticSearch

ElasticSearch: Faceted Search for Hierarchical data

ElasticSearch: Text analysis for content enrichment

ElasticSearch: Boosting score for content relevancy

ElasticSearch: Learn Java API usage with test cases

ElasticSearch: Indexing setup using Akka tutorial

ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch

This post covers to use ElasticSearch-Hadoop to read data from Hadoop system and index that in ElasticSearch. The functionality it covers is to index product views count and top search query per customer in last n number of days. The analyzed data can further be used on website to display customer recently viewed, product views count and top search query string.

In continuation to the previous posts on

•  Customer product search clicks analytics using big data,
•  Flume: Gathering customer product search clicks data using Apache Flume,
•  Hive: Query customer top search query and product views count using Apache Hive

we already have customer search clicks data gathered using Flume and stored in Hadoop HDFS and ElasticSearch, and how to analyze same data using Hive and generate statistical data. Here we will further see how to use the analyzed data to enhance customer experience on website and make it relevant for the end customers.

Recently Viewed Items

We already have covered in first part, how we can use flume ElasticSearch sink to index the recently viewed items directory to ElasticSearch instance and the data can be used to display real time clicked items for the customer.

ElasticSearch-Hadoop

Elasticsearch for Apache Hadoop  allows Hadoop jobs to interact with ElasticSearch with small library and easy setup.

elasticsearch-hadoop-hive, allows to access ElasticSearch using Hive. As shared in previous post, we have product views count and also customer top search query data extracted in Hive tables. We will read and index the same data to ElasticSearch so that it can be used for display purpose on website.

Flume: Gathering customer product search clicks data using Apache Flume

This post covers to use Apache flume to gather customer product search clicks and store the information using hadoop and elasticsearch sinks. The data may consist of different product search events like filtering based on different facets, sorting information, pagination information and further the products viewed and some of the products marked as favorite by the customers. In later posts we will analyze data further to use the same information for display and analytic.

Product Search Functionality

Any eCommerce platform offers different products to customers and search functionality is one of the basics of that. Allowing user for guided navigation using different facets/filters or free text search for the content is trivial of the any of existing search functionality.

SearchQueryInstruction

Consider a similar scenario where customer can search for a product and allows us to capture the product search behavior with following information,

Customer product search clicks analytics using big data

The application demonstrate to setup customer product search clicks analytics using big data Hadoop, Hive, Pig, Oozie, ElasticSearch, Akka, Spring Data etc.

Github Repository

URL: https://github.com/jaibeermalik/searchanalytics-bigdata

Analyzing Search Clicks Data Using Flume, Hadoop, Hive, Pig, Oozie, ElasticSearch, Akka, Spring Data.

Repository contains unit/integration test cases to generate analytics based on clicks events related to the product search on any e-commerce website.

Getting Started

The project is maven project and can be build with Eclipse. Check pom dependencies for relevant version of earch application. It uses cloudera hadoop distribution version 2.3.0-cdh5.0.0.

Functionality

The scenario covered in the application for the search analytics using big data is as follow,

ElasticSearch: Learn Java API usage with test cases

Find ElasticSearch tutorial on github using Java API with test cases.

Github

elasticsearch-tutorial repository uploaded on github explaining the ElasticSearch Java API usage with test cases.

URL: https://github.com/jaibeermalik/elasticsearch-tutorial

README

Following ElasticSearch concepts have been covered.