Tuesday, February 24, 2015

Java Quiz

1) What will be the output when you execute below code ?

for (int i = 0; i >= (-1) * 5; i--) {
          System.out.println(i);
}

Output : 0 -1 -2 -3 -4 -5


for loop repetitive control structure flow :
for (initialize; boolean_expression; update) {
}

  • Initialize will be called only once in the for loop to be initialize for the first time loop starts. You can leave blank by providing semicolon(;) placeholder.
  • boolean_expression will be called next to initialize to validate expression and if its true body of the for loop exucutes. False loop exits
  • update will be called upon completion of the body execution of for loop.
  • Once update complete it calls the boolean expression to validate to proceeds body and body completion will call update and moves on back to booleanExpression

Note : There is no difference between i-- or --i or i++ or ++i in the for loop update.


2) What will be the output when you execute below code ?
int i = 1;
int j = ++i;
System.out.println("i: " + i + " j: " + j);

and

int i = 1;
int j = i++;
System.out.println("i: " + i + " j: " + j);

Ouput : 
i: 2 j: 2
i: 2 j: 1


Why Difference?
    ++i increments the value first and then return it
    i++ return the value first and then increments it

Usually, try to use i = i + 1 for clear understanding in the code.

3) What will be the output when you execute below code ?
String testSplit = "hello,,world";
String[] testSplits = testSplit.split(",");
System.out.println(testSplits[2]);

Output : world

String testSplit = "hello,world,";
String[] testSplits = testSplit.split(",");
System.out.println(testSplits[2]);

Output : java.lang.ArrayIndexOutOfBoundsException: 2

String testSplit = "hello,world,";
String[] testSplits = testSplit.split(",", -2);
System.out.println(testSplits[2]);

Output :      (empty)

Note : By default, split drops all empty trailing columns, so any attempt to access the final column will result ArrayIndexOutOfBoundsException. Passing negative number (limit) as the second arg to split causes it to retain the trailing empty columns.

4) What will be the output when you execute below code ?
Pattern p = Pattern.compile("^([\"']?)\\d\\d:\\d\\d\\1,([\"']?)[A-Z]\\w+\\2,.*$");
  String regexpInput = "1:23,Logout Now";
  if (p.matcher(regexpInput.toString()).matches()) {
   System.out.println("Good");
  } else {
   System.out.println("Bad");
  }
Ouput : Bad

Why Bad ?
1. hour has only 1 digit 1 instead 2 digit
2. 2nd column has space in regexp which is not mentioned
3. Regexp expects 3rd column with comma(,) after some text 1 or more which is missing

\\1 and \\2 is backreference to repeat the grouping element
group starts from 1, 2, 3 Backreference also considered as group in regexp.

Difference between iBatis and Hibernate ? (Both are persistence framework)
iBatis / myBatis is sql driven model. It means based on sql you want to control application changes (SqlMap XML file)
Hibernate is object driven model. It means you design your object and create fields in database. (hbm - hibernate mapping xml file)
Pros and Cons :
iBatis is database dependent due to SQL usage, but faster development and lighter with cache support.
Hibernate is database independent due to HQL based approach. Its more heavy compare to JPA / iBatis where as highly scalable with advance cache support.

iBATOR - Code generator for iBatis.

castor - Castor is open source java data binding  framework. Moving data from XML to Java programming language objects and from Java to database. same as JAXB.

Difference between JPA and Hibernate ? 
JPA is specification / interface based on JSR
Hibernate / iBatis is implementation using JPA.

Saturday, February 21, 2015

Oozie Examples

Setting Up the Examples
The examples/ directory must be extracted from oozie and copied to the user HOME directory in HDFS:
cd /usr/local/oozie
tar -xvzf oozie-examples.tar.gz
chown -R hduser:hadoop examples/
hadoop fs -put /usr/local/oozie/examples/ /user/hduser/examples/
NOTE: If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be copied.

Running the Examples
Add Oozie bin/ to the environment PATH in .bashrc or /etc/profiles
export OOZIE_HOME=/usr/local/oozie
export PATH=$PATH:$HADOOP_HOME/bin:$HIVE_HOME/bin:$PIG_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin

Execute job from Terminal (hduser)
oozie job -oozie http://localhost:11000/oozie -config /usr/local/oozie/examples/apps/map-reduce/job.properties -run

NOTE: The job.properties file needs to be a local file during submissions, and not a HDFS path. Modify job.properties for namenode & jobtracker url.

Check Oozie URL to track the oozie workflow job status : http://localhost:11000/oozie

Note : The example applications are under the $OOZIE_HOME/examples/app directory, one directory per example. The directory contains the application XML file (workflow, or worklfow and coordinator), the job.properties file to submit the job and any JAR files the example may need. Go through each workflow to run hive, java, sqoop, streaming examples and try

Reference : Oozie Example
Oozie workflow with Pig and Hive

Friday, February 20, 2015

Big Data Growth Trends and Events

Hadoop is Growing faster than expected..

Click to Zoom the image : Hadoop Growth


US President speaks about importance of BigData Trend.






















Apache Oozie - Schedule your job using powerful workflow engine

Apache Oozie
Oozie is an extensible, scalable and reliable system to define, manage, schedule, and execute complex Hadoop workloads via web services. More specifically, this includes:

  * XML-based declarative framework to specify a job or a complex workflow of dependent jobs.
  * Support different types of job such as Hadoop Map-Reduce, Pipe, Streaming, Pig, Hive and custom java applications.
  * Workflow scheduling based on frequency and/or data availability.
  * Monitoring capability, automatic retry and failure handing of jobs.
  * Extensible and pluggable architecture to allow arbitrary grid programming paradigms.
  * Authentication, authorization, and capacity-aware load throttling to allow multi-tenant software as a service.

Oozie Engines :
Oozie has Workflow Engine,  Coordinator Engine and Bundle Engine

PreRequiste :
JAVA_HOME (Java)
M2_HOME (Install Maven)
HADOOP_HOME (hadoop)
Pig (PIG_HOME)

Download Oozie

Make sure below command works
$ java -version
$ javac -version
$ mvn -version


Extract Oozie archieve
$ sudo cp ~/Downloads/oozie-*.tar.gz /usr/local/
$ sudo su -
$ cd /usr/local
$ tar -xzf oozie-3.3.2.tar.gz


Building Oozie
The simplest way to build Oozie is to run the mkdistro.sh script:
$ cd oozie-3.3.2
$ ./bin/mkdistro.sh -DskipTests


Oozie Server Setup
Copy the built binaries to the home directory as ‘oozie’
$ cd ..
$ cp -R oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/ oozie

Create the required libext directory
$ cd oozie
$ mkdir libext


Copy all the required jars from hadooplibs to the libext directory using the following command:
$ cp ../oozie-3.3.2/hadooplibs/target/oozie-3.3.2-hadooplibs.tar.gz .
$ tar xzvf oozie-3.3.2-hadooplibs.tar.gz
$ cp oozie-3.3.2/hadooplibs/hadooplib-1.1.1.oozie-3.3.2/* libext/


Get Ext2Js – This library is not bundled with Oozie and needs to be downloaded separately. This library is used for the Oozie Web Console:
$ cd libext
$ wget http://extjs.com/deploy/ext-2.2.zip
$ cd ..


Update ../hadoop/conf/core-site.xml as follows. Hadoop Version 1.2.x:

<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>localhost</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>hadoop</value>
</property>


Note : Here, ‘hduser’ is the username and it belongs to ‘hadoop’ group.


Prepare the WAR file
$../bin/oozie-setup.sh prepare-war

INFO: Oozie is ready to be started


Provide permission to oozie directory
$ chown -R hduser:hadoop oozie


Create sharelib on HDFS
$ su hduser
$ /usr/local/oozie/
$ ./bin/oozie-setup.sh sharelib create -fs hdfs://localhost:54310


Create the OoozieDB
$ ./bin/ooziedb.sh create -sqlfile oozie.sql -run

The SQL commands have been written to: oozie.sql


Start Oozie as a daemon process run / start Oozie as a foreground process run:
oozie-start.sh, oozie-run.sh , and oozie-stop.sh
$ ./bin/oozie-start.sh
or
$ ./bin/oozie-run.sh
or
$ ./bin/oozie-stop.sh


Note : oozie log will be in /usr/local/oozie/logs/oozie.log


URL for Oozie Web Console is http://localhost:11000/oozie

Check Oozie status, should be NORMAL.
$ bin/oozie admin -oozie http://localhost:11000/oozie -status

Try Oozie Examples : Oozie Examples which i tried from my same blog

Oozie Client Setup may required in the remote machine
$ cd ..
$ cp oozie/oozie-client-3.3.2.tar.gz .
$ tar xvzf oozie-client-3.3.2.tar.gz
$ mv oozie-client-3.3.2 oozie-client
$ cd bin

Add the /home/hduser/oozie-client/bin to PATH in .bashrc or /etc/profiles and restart your terminal.

References : Oozie Installation : (Apache Oozie, Rohit Blog, CloudBlog)

Tuesday, February 17, 2015

Install Maven from archive

Why Maven ?

Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.
  •     Quick project set-up, no complicated build.xml files, just a POM and go
  •     All developers in a project use the same jar dependencies due to centralized POM.
  •     Getting a number of reports and metrics for a project "for free"
  •     Reduce the size of source distributions, because jars can be pulled from a central location
Make sure you installed Java and exported JAVA_HOME in your PATH variable.

Verify java installation on your machine
$java -version 
$javac -version
Download maven
Extract Maven archive
$sudo cp ~/Downloads/apache-maven-*.tar.gz /usr/local/
$sudo su -
$cd /usr/local
$tar -xzf apache-maven-3.1.1-bin.tar.gz

Set maven environment variables (vi ~/.bashrc or /etc/profiles)
export M2_HOME=/usr/local/apache-maven-x.x.x
export MAVEN_OPTS="-Xms256m -Xmx512m"

Add maven bin directory to system path
export PATH=$PATH:$M2_HOME/bin
Verify Maven installed
$mvn -version

References : Maven

Monday, February 16, 2015

Install Apache Sqoop - Sql to hadoop (HDFS) and inverse the same

sqoop - Sql to hadoop

Efficient tool to transfer bulk data from structured (relational db's) to hadoop (hdfs / hive / hbase). You can also export data from hdfs to import into other datawarehouses.

Multiple ways to install sqoop :
To install sqoop in Debian distributions (Ubuntu / Debian)

$ sudo apt-get install sqoop

Advantage installing with debian packages than Tar ball. Still to handle yourself efficiently the package use tar ball.
  •     Handle dependencies
  •     Provide for easy upgrades
  •     Automatically install resources to conventional locations
Download Sqoop  - Download sqoop version based on the hadoop you installed in your box. If you hadoop 1.x then download sqoop with hadoop 1.x version to avoid incompatible error upon import / export.

$ sudo su
$ (cp ~/Downloads/sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz /usr/local && cd /usr/local/
 && tar -zxvf path_to_sqoop.tar.gz)
$ mv sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz sqoop-1.4.5
$ chown -R hduser:hadoop sqoop-1.4.5

Configure sqoop wrapper with hadoop
$cd /usr/local/sqoop-1.4.5/conf/
$mv sqoop-env-template.sh sqoop-env.sh

Enable HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME by providing hadoop available path. If you want to sqoop data to HBASE or Hive enable those variables as well and provide path in sqoop-env.sh

$vi sqoop-env.sh
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop

In /home/hduser/.bashrc or /etc/profiles

export SQOOP_HOME=/usr/local/sqoop-1.4.5
export PATH=$PATH:$HADOOP_HOME/bin:$HIVE_HOME/bin:$PIG_HOME/bin:$SQOOP_HOME/bin

Installing JDBC Driver for Sqoop
sudo apt-get install libmysql-java

(This will install the jar for mysql connector through java, cd /usr/share/java/ to see the mysql-connector-java.jar). mysql-connector-java.jar must be symlink to mysql.jar. If so follow below

sudo ln -s /usr/share/java/mysql.jar /usr/local/sqoop-1.4.5/lib/

Check your sqoop version
$sqoop-version

Create table movies in your mysql
Refer : Download sqoopSample.sql from github for mysql schema.

Import data (table or all tables) from MySQL into Hadoop (HDFS)
Make sure you started your hadoop. (cd $HADOOP_HOME/bin; ./start-all.sh)
-m Use n map tasks to import in parallel
$cd /home/hduser
$sqoop import --connect jdbc:mysql://hostName/dbName --username userName --password password --table tableName --target-dir /samples/movies -m 1
 
$sqoop import-all-tables --connect jdbc:mysql://hostName/dbName --username userName --password password 

Important :
  • If you see error "Exception in thread "main" java.lang.IncompatibleClassChangeError Found class org.apache.hadoop.mapreduce.JobContext, but interface was expected" then hadoop version between you installed and sqoop hadoop has mismatched. Hadoop 1 and hadoop 2 has major changes. So, have 1.x or 2.x on hadoop and sqoop-hadoop.
  • Make sure table has primary key otherwise you need to mention map tasks.

Export data from hadoop (hdfs) into MySQL
$sqoop export --connect jdbc:mysql://localhost/sqoop_test --table movies_export --username root --password root --export-dir '/samples/movies/' -m 1;

sqoop command to create hive tables matches the database table. If already exist it will throw error
hduser$ sqoop create-hive-table --connect jdbc:mysql://localhost/sqoop_test --username root --password root --table movies

Note : Impala uses the same metadata as hive, So you can use create-hive-table to import and query in Impala

sqoop command to import the entire content from the database table to hive, that uses commas(,) to separate the files in data files
hduser$ sqoop import --connect jdbc:mysql://localhost/sqoop_test --username root --password root --table movies --fields-terminated-by ',' --hive-import
 
hduser$  

--hive-overwrite -> This will overwrite the existing content and write the data into hive table
--hive-import -> To import the table into hive we need --hive-import when there is no table in hive.
--hive-table -> If this is not specified by default the database table name will be created in the hive

Default delimiter using sqoop into hive is ^a. You can specify --fields-terminated-by ',' for the specific separator. 

Refer : Download sqoopSample.sql from github for mysql schema.

References : Tutorial Reference,
GoodPlaceToStart 

Fomat your blog

How to add CSS to blogger to be available in all pages
Blogger -> Posts -> Layout -> Template Designer -> Advanced -> Add CSS

Add below CSS
.preTagCodeStyle {
font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;
background-image:URL(https://github.com/gubs4u/DownloadSamples/blob/master/codebg.gif);
padding:0px;color:#000000;text-align:left;line-height:20px;
}
.codeTagStyle {
 color:#000000;word-wrap:normal;
}
div.note {
    background-color: #fefaee;
    padding: 10pt;
    margin-top: 0.6em;
    margin-bottom: 0.6em;
    margin-right: 0.6em;
    border: 1px dashed;
    border-radius: 3px;
    border-color: #CCCCCC;
}
div.configuration {
    padding: 1em;
    border: 1px dashed #2f6fab;
    color: black;
    background-color: #f9f9f9;
    line-height: 1.1em;
    font-family: Courier New, Courier, mono;
    font-size: 12px;
    font-style: italic;
    display: block;
} 
To format your code add below:
<pre class="preTagCodeStyle"><code class="codeTagStyle">
Your code here
</code></pre>
To add a note use below:

<div class="note">
Your note here
</div>
To add a configuration use below:

<div class="configuration">
Your configuration here
</div>
Note : Go to HTML tab and add this tag. In 'Compose' have option 'Show HTML Literally'

// Below script tag for SyntaxHighLighter