HBASE - Corrupted Blocks

If you see in your dfshealth if any blocks corrupted warnings as below. Run the command and check your hbase health.

http://devnamenode:50070/dfshealth.jsp

WARNING : There are about 1 missing blocks. Please check the log or run fsck. 

Run the below command in hdfs to see the corrupted files status :

hadoop fsck /

Note : Check the corrupted files in hdfs / hbase and based you can see hdfs job or hbase blocks. See below samples : 

1.  /user/hdfs/.staging/job_201307121242_42849/job.split: MISSING 1 blocks of total size 81

Above Corrupted in hdfs when it run the job job_201307121242_42849 (2013 07 12 @ 12:42)

2. /hbase/.corrupt/ednwavlhd01%3A60020.1349814163628: CORRUPT block blk_-7209961989095415639

/hbase/.corrupt/ednwavlhd01%3A60020.1349814163628:  Under replicated blk_-7209961989095415639_403425. Target Replicas is 3 but found 1 replica(s).

Above blocks Corrupted in hbase when it replication. We have epochTime before :CORRUPT (1349814163628). See the date of the data corrupted and find out table belongs to this blocks and delete if not required.

To Delete the corrupted block run the below command

hadoop fsck -delete /

To move the corrupted block run the below command 

hadoop fsck -move /

-move option moves under /lost+found in hdfs partition. You may delete once move command move the files to this directory.