Hive: Query customer top search query and product views count using Apache Hive

This post covers to use Apache Hive to query the search clicks data stored under Hadoop. We will take examples to generate customer top search query and statistics on total product views.

In continuation to the previous posts on

• Customer product search clicks analytic using big data,
• Flume: Gathering customer product search clicks data using Apache Flume,

we already have customer search clicks data gathered using Flume in Hadoop HDFS.

Here will analyze further to use Hive to query the stored data under Hadoop.

Hive

Hive allow us to query big data using SQL-like language HiveQL.

Hadoop Data

As shared in last post, we have search clicks data stored under hadoop with the following format “/searchevents/2014/05/15/16/”. The data is stored in separate directory created per hour.

The files are created as,

hdfs://localhost.localdomain:54321/searchevents/2014/05/06/16/searchevents.1399386809864

Customer product search clicks analytics using big data

The application demonstrate to setup customer product search clicks analytics using big data Hadoop, Hive, Pig, Oozie, ElasticSearch, Akka, Spring Data etc.

Github Repository

URL: https://github.com/jaibeermalik/searchanalytics-bigdata

Analyzing Search Clicks Data Using Flume, Hadoop, Hive, Pig, Oozie, ElasticSearch, Akka, Spring Data.

Repository contains unit/integration test cases to generate analytics based on clicks events related to the product search on any e-commerce website.

Getting Started

The project is maven project and can be build with Eclipse. Check pom dependencies for relevant version of earch application. It uses cloudera hadoop distribution version 2.3.0-cdh5.0.0.

Functionality

The scenario covered in the application for the search analytics using big data is as follow,

Java Application performance analysis and optimization using AppDynamics

The enterprise java application stack is growing bigger and bigger which makes it equally difficult to keep control on all the layers of the infrastructure to get maximum result out of it. One of the basic requirement of any web application is well performing, we will cover here an ideal enterprise java web application setup and see how to analyze and optimize the same using AppDynamics tool.

Java Enterprise web application N-tier set up

Take an example of below n-tier java web application interacting with complex middleware system, integrate with numerous external web api’s and equally powerful backend storage system.

enterprise_java_web_application_setup

The diagram covers quite common and complex enterprise application set up.

• Web Servers (eg. Apache web server)
• Application Servers (tomcat application server)
• Mobile application server (tomcat application server)
• Email Server
• Web content management server (eg. Team Site, Alfresco)
• Web application Administration server (tomcat application server)
• File servers (Shared disk eg. NFS)
• Real time/Messaging/Queue server (eg. ActiveMQ)
• Data/File processing backend servers (tomcat application server)
• Data storage/Database servers (eg. MySQL/Oracle)

Problem Context

Tips&Tricks: Ehcache post processing method cached results

The post is to share a tip on advisory for not post processing the cached results from Ehcache method interceptor. In case you are using method cache interceptor within you application, please make sure the cached results are not updated anywhere after retrieving data from cache.

An example of how you can use spring aop method interceptor for caching the results of a method, Caching the result of methods using Spring and EHCache

Let us take an example,

List com.jai.test.TestService.getTestList();

public class TestObj implements Serializable
{
...
private List customers;
...
}

Tips&Tricks: Flex-Cairngorm-Blazeds-Spring-JPA-Hibernate

This post includes some of the lessons learned during recent integration and usage of these framework. This is just to help out in case you too fall into these small small problems which may eat of lot of your time.

Cairngorm maven pom dependency entry

<dependency>

<groupId>com.adobe.cairngorm</groupId>

<artifactId>cairngorm-bin</artifactId>

<version>2.2.1</version>

<type>swc</type>

</dependency>

The prefix “mx” for element “mx:Script” is not bound