HBase: Generating search click events statistics for customer behavior

In this post we will explore HBase to store customer search click events data and utilizing same to derive customer behavior information based on search query string and facet filter clicks. We will cover to use MiniHBaseCluster, HBase Schema design, integration with Flume using HBaseSink to store JSON data.

In continuation to the previous posts on,

• Customer product search clicks analytics using big data,
• Flume: Gathering customer product search clicks data using Apache Flume,
• Hive: Query customer top search query and product views count using Apache Hive,
• ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch,
• Oozie: Scheduling Coordinator/Bundle jobs for Hive partitioning and ElasticSearch indexing,
• Spark: Real time analytics for big data for top search queries and top product views

We have explored to store search click events data in Hadoop and to query same using different technologies. Here we will use HBase to achieve same,

•  HBase mini cluster setup
•  HBase template using Spring Data
•  HBase Schema Design
•  Flume Integration using HBaseSink
•  HBaseJsonSerializer to serialize json data
•  Query Top 10 search query string in last an hour
•  Query Top 10 search facet filter in last an hour
•  Get recent search query string for a customer in last 30 days

Spark: Real time analytics for big data for top search queries and top product views

Hadoop being the batch processing framework makes it a little hard to get the real time analytics for big data. Apache Spark overcomes this batch nature and provides distributed computation capabilities and events processed in streaming fashion. In this post, we will cover to explore Spark streaming capability to process Flume Events data to generate Top search query strings generated in last an hour or top product views in the last one hour.

In continuation to the previous posts on

•  Customer product search clicks analytics using big data,
•  Flume: Gathering customer product search clicks data using Apache Flume,
•  Hive: Query customer top search query and product views count using Apache Hive,
•  ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch,
•  Oozie: Scheduling Coordinator/Bundle jobs for Hive partitioning and ElasticSearch indexing

We have so far utilized the Hadoop system batching capabilities to process huge amount of data. But the overall batching operation makes it a bit of latency issue depending on your data. This is where Spark comes into picture. We will explore Spark streaming capability here to get some real time analytics and those can be used on the website for display purpose or for monitoring purpose.

Spark

Apache spark  “is a fast and general engine for large-scale data processing.”

Functionality

As shared in other above exmaples, we have the customer search clicks data available to us. We have Flume system in place to process the data and store in Hadoop for later processing perspective. Take a scenario, you want to display real time customer behavior on the website, how other customers are doing

• What other customers searching?
• Other customers also searching for…
• Top search query string on the website in last an hour
• What other customers viewing?
• Other customers also viewing products…
• Top product views in the last an hour

Oozie: Scheduling Coordinator/Bundle jobs for Hive partitioning and ElasticSearch indexing

This post covers to use Oozie to schedule Hive add partition every hour with the help of Coordinator jobs and to automatically update the ElasticSearch data served to customer based on nightly jobs using Bundle jobs functionality. The automated procedure using oozie jobs will help to update the statistical data used on website to display product views count and top search query string.

In continuation to the previous posts on

•  Customer product search clicks analytics using big data,
•  Flume: Gathering customer product search clicks data using Apache Flume,
•  Hive: Query customer top search query and product views count using Apache Hive,
•  ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch

As described in earlier posts, the hive partitioning strategy is added based on current time and accordingly the elasticsearch indexing based on analytic data also. We will cover here to automate the process using Oozie to add hive partition once data is available in hadoop directory.

Oozie

Oozie  is a workflow scheduler system to manage Apache Hadoop jobs.

ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch

This post covers to use ElasticSearch-Hadoop to read data from Hadoop system and index that in ElasticSearch. The functionality it covers is to index product views count and top search query per customer in last n number of days. The analyzed data can further be used on website to display customer recently viewed, product views count and top search query string.

In continuation to the previous posts on

•  Customer product search clicks analytics using big data,
•  Flume: Gathering customer product search clicks data using Apache Flume,
•  Hive: Query customer top search query and product views count using Apache Hive

we already have customer search clicks data gathered using Flume and stored in Hadoop HDFS and ElasticSearch, and how to analyze same data using Hive and generate statistical data. Here we will further see how to use the analyzed data to enhance customer experience on website and make it relevant for the end customers.

Recently Viewed Items

We already have covered in first part, how we can use flume ElasticSearch sink to index the recently viewed items directory to ElasticSearch instance and the data can be used to display real time clicked items for the customer.

ElasticSearch-Hadoop

Elasticsearch for Apache Hadoop  allows Hadoop jobs to interact with ElasticSearch with small library and easy setup.

elasticsearch-hadoop-hive, allows to access ElasticSearch using Hive. As shared in previous post, we have product views count and also customer top search query data extracted in Hive tables. We will read and index the same data to ElasticSearch so that it can be used for display purpose on website.

Hive: Query customer top search query and product views count using Apache Hive

This post covers to use Apache Hive to query the search clicks data stored under Hadoop. We will take examples to generate customer top search query and statistics on total product views.

In continuation to the previous posts on

• Customer product search clicks analytic using big data,
• Flume: Gathering customer product search clicks data using Apache Flume,

we already have customer search clicks data gathered using Flume in Hadoop HDFS.

Here will analyze further to use Hive to query the stored data under Hadoop.

Hive

Hive allow us to query big data using SQL-like language HiveQL.

Hadoop Data

As shared in last post, we have search clicks data stored under hadoop with the following format “/searchevents/2014/05/15/16/”. The data is stored in separate directory created per hour.

The files are created as,

hdfs://localhost.localdomain:54321/searchevents/2014/05/06/16/searchevents.1399386809864