CDH - Hadoop - Hbase - Adding a host to an existing cluster

1. Install CentOS version matches to the existing hadoop clusters. 

2. Configure static Ip, DNS, gateway and hostnames.

3. Enable ssh and disable firewall. 

4. In /etc/selinux/config file disable the SELINUX

5. In /etc/hosts file add the distributed cluster machines

6. In /etc/fstab hadoop partition add noatime with default on ext4 http://www.howtoforge.com/reducing-disk-io-by-mounting-partitions-with-noatime)

7. Install "yum install ntp" and start the service and add the service to chkconfig ntpd on. Sync the time with server time pool "ntpdate pool.ntp.org"

8. Login into cloudera Click 'Hosts'

9. Add Hosts

10. Type the hostname and search

11. select and click 'Install CDH on machine' 

12. Before you select which version to install check in master CDH version and do the same. (go to master and type hbase shell). In hbase start it will show hbase and CDH version.

13. Copy the .bashrc file settings from other server to the new server

14. Go to each and every service (TaskTracker, DataNodes and RegionServer and add the new server). Make sure Master is not checked while adding regionServer

15. Copy the library of (hadoop_lib_jars from svn into $HADOOP_HOME/lib) and restart MapReduce in CDH.

16. Restart client like azkaban once you modified the zookeeper quorum adding this host cluster

HBASE - RegionServer - Hbase Master failed to reach RegionServer

RegionServer was failed to respond hbase Master. Basically, in regionServer the zookeeper failed to respond due to GC happens and java stop-the-world.

Read the below blog which explains very very clever how to get rid of GC failure.

http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/

Below is some configuration changes we did on our side to avoid the issue as a system load test.

(Hbase configuration) Hbase zookeeper session timeout increased to 90 seconds from 40seconds and default 60seconds as per hbase guide max : 3minutes you can have. To collect GC on 1 GB on an avg. system takes 8 to 10 seconds. Since, we have HeapMemory configured 8 GB and GC can collect @ around 7 GB we may ended up failure on connection time out.

Pass java arguments : -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:PrintFLSStatistics=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/usr/lib/hbase/logs/logs/gc-$(hostname)-hbase.log

Enable MSLAB Allocation Scheme with default flush values - In hbase-0.92 its enabled by default.

Cluster should be in odd number. Because, zookeeper multi-server suggest to have in odd numbers.

HBASE - Corrupted Blocks

If you see in your dfshealth if any blocks corrupted warnings as below. Run the command and check your hbase health.

http://devnamenode:50070/dfshealth.jsp

WARNING : There are about 1 missing blocks. Please check the log or run fsck. 

Run the below command in hdfs to see the corrupted files status :

hadoop fsck /

Note : Check the corrupted files in hdfs / hbase and based you can see hdfs job or hbase blocks. See below samples : 

1.  /user/hdfs/.staging/job_201307121242_42849/job.split: MISSING 1 blocks of total size 81

Above Corrupted in hdfs when it run the job job_201307121242_42849 (2013 07 12 @ 12:42)

2. /hbase/.corrupt/ednwavlhd01%3A60020.1349814163628: CORRUPT block blk_-7209961989095415639

/hbase/.corrupt/ednwavlhd01%3A60020.1349814163628:  Under replicated blk_-7209961989095415639_403425. Target Replicas is 3 but found 1 replica(s).

Above blocks Corrupted in hbase when it replication. We have epochTime before :CORRUPT (1349814163628). See the date of the data corrupted and find out table belongs to this blocks and delete if not required.

To Delete the corrupted block run the below command

hadoop fsck -delete /

To move the corrupted block run the below command 

hadoop fsck -move /

-move option moves under /lost+found in hdfs partition. You may delete once move command move the files to this directory.

Deleting Hosts

You can stop Cloudera Manager from managing a host and the Hadoop daemons on the host.

First, make sure there are no roles running on the Host; you can decommission the host to ensure all roles are stopped.

Second, you must stop the Cloudera Manager Agent from running on the host; if you don't stop the Agent, it will send heartbeats to the Cloudera Manager Server and show up again in the list of hosts.

To delete a host:

1. Decommission the host to ensure all roles on the host have been stopped. For instruction, see Decommissioning a Host.
2. Stop the Agent on the host. For instructions, see Stopping and Restarting Cloudera Manager Agents.
3. In the Cloudera Manager Admin Console, click the Hosts tab.
4. Select the host in the Hosts tab.
5. From the Actions for Selected menu, select Delete.

Reference : http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Free/4.5.1/Cloudera-Manager-Free-Edition-User-Guide/cmfeug_topic_6_7.html

Note : Atleast 1 hbase master should run while decommission  So, the region data will be swap to another region server. So, make sure hbase and other services running. Cloudera will take care of adding into excluding list safely

Also make sure the agents on cloudera machines were stopped else the host will be displayed automatically on cloudera

GitHub - Source Code Control

Why GitHub ? 

Github is basically a version control system same as SVN (SubVersion) or CVS to maintain your code in the web UI. Very popular openSource component to keep our code. It has 2 feature Public or Private. Public projects will be viewed by anyone. You will find most of the openSource code published in GitHub. You can integrate gitHub easily with standard editors "Eclipse, Netbeans, IntelliJ.."

Latest Eclipse (Luno) release comes with GitHub plugin as in-build. You can checkout your source and check-in easily.

GitHub in Command Line In Interface on Ubuntu

Install Git in Ubuntu

$ sudo apt-get install git

How to Checkout from Git?

$ git clone https://github.com//.git

How to Check status on your directory with git master ?

$ cd ~/
$ git status -u origin master

Pull your changes from GitHub

$ cd ~/
$ git pull -u origin master

Push your changes into GitHub

$ git add 
$ git commit -m "Adding  into Git"
$ git push -u origin master

Add your existing project into Git

Login to your gitHub in web UI

Create a Repository in your github UI 

Push project from Terminal into your created Git Repository

$ cd ~/
$ git init
$ git add 
$ git commit -m "Adding projects into git" .
$ git remote add origin https://github.com/username/projectName.git
$ git pull -u origin master 
$ git push -u origin master

Check out GitHub project repository into Eclipse

• Open Eclipse with workspace
• Go to Window > Open Perspective > Other > Git > Ok
• Right click and select 'Paste Respository Path or URI'
• Copy and paste the Git URI from Git Web UI > HTTPS Clone URL
• Move on with Next and finish to checkout repository into your workspace
• Select Git Repository > Right Click 'Import Project' > Select your project into working directory > Finish

Convert Regular checkout project into Java Project in Eclipse

• Select your project > right click > Properties > Project Facets > Click 'Convert to faceted form..'
• Select 'Java' and 'Version' change in dropdown if you have multiple version
• Click 'Apply' > Ok

Create sourceFolder (src/main/java) for your project in Eclipse

• Select your project > right Click > Source Folder > FolderName 'src/main/java' > Finish 
• Select your project > right Click > Source Folder > FolderName 'src/main/test' > Finish 

Push your changes into Git Repository through Eclipse

• Open Eclipse with Workspace had checkout project from git. 
• Select your project > right Click > Team > Add To Index (Need to add first)
• Select your project > right Click > Team > Commit > provide commit message > commit and push > ok

Pull your changes into workspace from Git Repository through Eclipse 

• Open Eclipse with Workspace had checkout project from git.  
• Select your project > right Click > Team > Pull

Note : You can do synchronize with Repository through Team > Synchronize Workspace to sync and verify the pull or push changes before you do.

Try : Familiar yourself with Git in 5 mins

How to get remote ip address when data get post through LB (Load Balancer)

In tomcat under conf/server.xml. Update below code

           prefix="localhost_access_log." suffix=".txt"
           pattern="%{X-Forwarded-For}i %h %l %u %t "%r" %s %b"    
resolveHosts="false" />

         

Java Code

The following Java code extracts the originating IP address of an HttpServletRequest object.

public final class HTTPUtils {

    private static final String HEADER_X_FORWARDED_FOR =

        "X-FORWARDED-FOR";

    public static String remoteAddr(HttpServletRequest request) {

        String remoteAddr = request.getRemoteAddr();

        String x;

        if ((x = request.getHeader(HEADER_X_FORWARDED_FOR)) != null) {

            remoteAddr = x;

            int idx = remoteAddr.indexOf(',');

            if (idx > -1) {

                remoteAddr = remoteAddr.substring(0, idx);

            }

        }

        return remoteAddr;

    }

}

Squirrel - Java SQL Client

Why Squirrel ?

SQuirreL SQL is an open-source Java SQL Client program for any JDBC compliant database. Easy configurable by loading jars and can connect to various SQL Servers (MSSQL, MySQL, Sybase...)

You can learn more about Squirrel.

Explore Squirrel

1) Download squirrel-sql-3.4.0-install.jar

2) Run / Execute the jar from console : java -jar squirrel-sql-3.4.0-install.jar

3) Squirrel will get install in your desktop as squirrel plugin under $HOME directory. You  have uninstall script under the same directory to do so.

4) Open the Squirrel client go to Drivers and select the driver, Click modify and load the (Jars for Mysql or MsSQL)  under External class path. 

For MySQL :
mysql-connector.jar - class name : com.mysql.jdbc.Driver

For SQL Server 
sqljdbc4.jar - class name : com.microsoft.sqlserver.jdbc.SQLServerDriver

5) Go to alias and add your 'new alias' with and show your driver. You will be connected to the server and enjoy playing with your squirrel client on connecting to different database servers.