Spark: Real time analytics for big data for top search queries and top product views

Hadoop being the batch processing framework makes it a little hard to get the real time analytics for big data. Apache Spark overcomes this batch nature and provides distributed computation capabilities and events processed in streaming fashion. In this post, we will cover to explore Spark streaming capability to process Flume Events data to generate Top search query strings generated in last an hour or top product views in the last one hour.

In continuation to the previous posts on

•  Customer product search clicks analytics using big data,
•  Flume: Gathering customer product search clicks data using Apache Flume,
•  Hive: Query customer top search query and product views count using Apache Hive,
•  ElasticSearch-Hadoop: Indexing product views count and customer top search query from Hadoop to ElasticSearch,
•  Oozie: Scheduling Coordinator/Bundle jobs for Hive partitioning and ElasticSearch indexing

We have so far utilized the Hadoop system batching capabilities to process huge amount of data. But the overall batching operation makes it a bit of latency issue depending on your data. This is where Spark comes into picture. We will explore Spark streaming capability here to get some real time analytics and those can be used on the website for display purpose or for monitoring purpose.

Spark

Apache spark  “is a fast and general engine for large-scale data processing.”

Functionality

As shared in other above exmaples, we have the customer search clicks data available to us. We have Flume system in place to process the data and store in Hadoop for later processing perspective. Take a scenario, you want to display real time customer behavior on the website, how other customers are doing

• What other customers searching?
• Other customers also searching for…
• Top search query string on the website in last an hour
• What other customers viewing?
• Other customers also viewing products…
• Top product views in the last an hour

ElasticSearch: Indexing setup using Akka tutorial

Find ElasticSearch tutorial on github using Akka with test cases. The tutorial covers the search indexing setup using Akka.

Github

elasticsearch-akka repository uploaded on github explaining the ElasticSearch indexing setup using Akka Actors usage with test cases.

URL: https://github.com/jaibeermalik/elasticsearch-akka

README

Setup Index

ElasticSearch: Learn Java API usage with test cases

Find ElasticSearch tutorial on github using Java API with test cases.

Github

elasticsearch-tutorial repository uploaded on github explaining the ElasticSearch Java API usage with test cases.

URL: https://github.com/jaibeermalik/elasticsearch-tutorial

README

Following ElasticSearch concepts have been covered.

Tips&Tricks: How to set web service url end point

A quick tip to change the url end point for a webservice. Keep the web service end point as configurable to be able to use it under different test/demo/live environments.

WebClientTestService service = new WebClientTestService() ;
String serviceEndPoint = "http://localhost/soap/testEndPoint";
Map<String, Object> context = ((BindingProvider)service).getRequestContext();
//Set service end point
context.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, serviceEndPoint);

Keep the service end point etc. configurations as system property or pick from properties file to make it configuration in the system to be able to use different test/demo/live environment without changes the client code.

Email domain validation in java on server side

Email Address name and domain name validation on client side or server side are import part of maintaining quality customer data. There are plenty of utilities to allow you to validate the valid name formatting for the same. This post covers to check if the domain name used in the given email address actually valid/exists or not using MX records.

Email Address name validation

Email Address regular expression validation covers the valid regular expression matching.

Apache commons library Email validator covers the email name validation in java.

Check How To Validate Email Address With Regular Expression to find the matching regular expression for Email Address.

Email domain MX record validation

The post Java Extended Email Validation Using DNS MX Lookup covers to validate the MX records for the email domain.

dnsjava library allows you do the real dns validation on java side.

Maven dependency