More tips towards your Java world

Initialize multiple variable in same line
private int a,b,c,d;
a = b = c = d = 0;

Access Modifier in Java

Access Levels

Modifier	Class	Package	Subclass	World
public	Y	Y	Y	Y
protected	Y	Y	Y	N
no modifier	Y	Y	N	N
private	Y	N	N	N

Note : "No moidifier" can also call as package-private. Only difference between protected and package-private is protected modifier can be access in subClass.

Why static method can only have static variables & static methods ? 
When you run your java program, static methods call first. So, the methods and variables inside should also be static in order for the static method to be executable.

'final' keyword
final class cannot be superclass (Cannot Inherited by class)
final variable cannot be overwritten. Like constant
final method cannot be override. (Mostly final method will be used in abstract class)

Quotes

Good Quotes to read always

• If you have anger explain it, don't express it - Control your emotion and explain
• A penny saved is a penny earned -  Benjamin Franklin
• Am I efficient ? List your task, estimate yourself and justify yourself - You can only answer about your efficiency not others.
• To be a successful person first surround with successful people and tried to read more successful people biography to get motivate and you will get your success. 
• If you been good with your friend, just be as it is. Don't change yourself because your friend did bad to you. Don't change your individuality. If you really hurt on him, Just ignore him and move on.

Install Hadoop in a Single Node (Linux / Ubuntu)

What is hadoop ?

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

You can group lot of small hardWare CPU's as cluster and process / analyse your data using those clusters instead of data getting processed in a single system. 

PreRequisites:
Java 1.6+ (Recommended : Oracle Java)

Update .bashrc or /etc/profiles

export JAVA_HOME=/usr/local/java/jdk1.6.0_25
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

Note : Make sure JAVA_HOME is set in /etc/profile. So that, java will be available for different users in the machine.

To Check the java version :

$ java -version

Create group and user for hadoop as a best practice

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser

Install and Configure ssh & rsync Tool used by hadoop distributed file system (HDFS)

$ sudo apt-get install ssh
$ sudo apt-get install rsync

Note : Make sure sshd is running in your machine

$ ps -ef | grep sshd
$ /etc/init.d/ssh start

Create SSH key for hduser

$ su - hduser
hduser@laptop: ssh-keygen -t rsa -P ""

Note : RSA key should be empty without password

Authorize SSH Key to avoid hdfs to provide password each time

$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
hduser$ ssh localhost

Hadoop Installation
Download Apache Hadoop stable version

$ cd /usr/local
$ sudo tar -xvzf hadoop-1.2.1.tar.gz
$ sudo mv hadoop-1.2.1 hadoop
$ sudo chown -R hduser:hadoop hadoop

Configure Hadoop  
Export HADOOP_HOME and add hadoop into bin Path in /etc/profile or hduser .bashrc. Make sure JAVA_HOME also configured

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

Update

$ vi /usr/local/hadoop/conf/hadoop-env.sh 
export JAVA_HOME=/usr/local/java/jdk1.6.0_25

Create directory for hadoop.tmp.dir (hadoop storage data files directory)

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
$ sudo chmod 750 /app/hadoop/tmp

Update $HADOOP_HOME/conf/conf/core-site.xml configuration tags with below configuration

<code>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system.</description>
</property>
</code>

Update $HADOOP_HOME/conf/mapred-site.xml configuration tags with below configuration

<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs. If "local", then jobs are run in-process as a single map and reduce task
</description>
</property>

Format namenode before you start your daemons

$HADOOP_HOME/bin/hadoop namenode -format

Note : Please execute only in the local environment where you installing hadoop. This command format / delete the entire data from the hadoop distributed file system. It will format and create HDFS directory based on the dfs.name.dir variable declared in the $HADOOP_HOME/src/hdfs/hdfs-default.xml.

Starting your single  node cluster

hduser$ $HADOOP_HOME/bin/start-all.sh

Note : Above command will start NameNode, DataNode, JobTracker and TaskTracker

Check the java process to see the daemons started and check the listening port

$ jps
$ netstat -plten | grep java

Note : Hadoop error log files in the $HADOOP_HOME/logs/ directory. You can see separate log file for each and every daemons.

Hadoop Web UI's and ports
http://localhost:50070/ – NameNode UI
http://localhost:50030/ – JobTracker UI
http://localhost:50060/ – TaskTracker UI

MapReduce Job Examples

• Make sure hadoop started and above mentioned ports are available
  
• Download sample for hadoop from gitHub user.txt
• Right click and click 'Save Page as'

Create directory in hdfs and copy sample files into hdfs

$ sudo su -
$ cp /home/user/Downloads/user.txt /home/hduser/
$ su hduser
$ hadoop fs -mkdir /samples/hadoop
$ hadoop fs -put /home/hduser/user.txt /samples/hadoop/

Command to run the wordCount example from hadoop

hduser$ cd /usr/local/hadoop
hduser$ hadoop jar hadoop*examples*.jar wordcount /samples/hadoop/user.txt /samples/hadoop-output

Delete existing output folder / folder from HDFS

hduser$ hadoop fs -rmr /samples/hadoop-output

Note : Make sure hadoop-output directory is not exist in hdfs. Hadoop example will create hadoop-output directory with output files. You can increase reduce task by passing "-D" mapred.reduce.tasks

hduser$ cd /usr/local/hadoop
hduser$ hadoop jar hadoop*examples*.jar wordcount -D mapred.reduce.tasks=16 /samples/hadoop/user.txt /samples/hadoop-output

Note : MapReduce job can accepts the user specified mapred.reduce.tasks and doesn’t manipulate. No. of mapper tasks will be decided by daemons based on the input content and available clusters. We cannot pass as input.

Verify the output generated file from HDFS

hduser$ /usr/local/hadoop/bin/hadoop dfs -cat /samples/hadoop-output/part-r-00000

Download the hdfs output file from hdfs to local

hduser$ hadoop dfs -get /samples/hadoop-output/part-r-00000 /tmp/

Command to stop your cluster

hduser$ /usr/local/hadoop/bin/stop-all.sh

Thanks Michael. I modified and added instruction upon my experience while following his blog.
Know more about Apache Hadoop and Developer.com BigData.
Install Cloudera VM, Counters, Partitioning, Combiners
Excellent hortonWorks tutorial

IoT - Internet Of Things

Internet of things (IoT) is very interesting topic. 

What is IoT ? 

Physical Object + MicroController / Sensor  Or actuators + Internet = IoT

Devices came out relating IoT : 
FitBit -> It measure your steps, calories and sleep and pushes your data to your account. 

MicroController -> microcontroller also call as "small computer".  It has Processor (CPU), Memory (RAM), EPROM (Erasable Programmable ROM), I/O (Input / Output). 

Arduino - Very Interesting tool to take a look as a part of IoT

Sensor -> Is a device that detects events or changes in quantities and provides a corresponding output, generally as an electrical or optical signal;

Input Sensor example -> Keyboard / Mouse

Output Sensor example -> Monitor / Printer. Based on the input signals it converts and show in the monitor to user readable format.

More Examples on IoT 
1) Alarm clock developed based on train schedule. It wakes you up based on your train schedule by connecting into Internet. 
2) Bus Board in the Bus station. Since, every bus has GPS sensor it sends the information to Bus department and data from their delivers to bus stand board for display.
3) Umberalla : Umberalla signal with 'Red' indicator if forecast from BBC says rainy.

GlowCaps, wheredial.com  

The business closed due to technology changes are 
daily newspapers, classifieds which deliver adv. about rental, jobs, post letters, Telephones (Home Phones)..

Read : Designing the Internet of Things written by "Adrian McEwen" was very interesting.

Any IoT interested candidates need help on programming, please feel free to reach me. I am very excited to help you.

Unsupported major.minor in Java

Exception in thread "main" java.lang.UnsupportedClassVersionError: Concordance : Unsupported major.minor version 51.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: Concordance.  Program will exit.

If your  java program shows above error, then you compiled the file in higher version of java from the version you are executing this jar / war / class.

Ex : Compiled in java 7 and running the program in java 6 (lower version).

How to resolve : Re-compile the program in java 6 to avoid the issues and execute the same. Check your JAVA_HOME added in your PATH  once.

Which is Allowed : You can compile in java 6 and run in java 7 is allowed. Program compiled in lower version  class file can run in higher version. Other-way around is not allowed.

nginx Web Server

Install nginx in centOS 6

Create a file in yum repository for nginx with its repo.
vi /etc/yum.repos.d/nginx.repo

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/6/$basearch/
gpgcheck=0
enabled=1

Once you saved the file

yum install nginx
service nginx start
service nginx status

By default nginx will be started and available in port :80.

  On the same server, protect the Java server from external access 

If you are running Nginx on the same server of the Java, the best practice is to deny access to port 8080 so only Nginx can access it. On Linux do:

/sbin/iptables -A INPUT -p tcp -i eth0 --dport 8080 -j REJECT --reject-with tcp-reset

Reference : http://wiki.nginx.org/JavaServers

Epoch Time in Shell Script for the expected Date

Below is the code to get epoch time in shell script :

Code is self-explanatory for the shell script dev's.

startdate_ymd=`date --date="31 day ago" +%Y%m%d`

startdate=`date --date="${startdate_ymd}" +%s`

startdatemillis=`expr ${startdate} \\* 1000`


enddate_ymd=`date --date="1 day ago" +%Y%m%d`

enddate=`date --date="${enddate_ymd}" +%s`

enddatemillis=`expr ${enddate} \\* 1000`