HBase导入大数据三大方式之(一)hive类SQL语句方式 – 数据库综合 – 次元立方网 – 电脑知识与技术互动交流平台

做大数据时,经常需要用到将大量格式化的文本数据导入到hbase中。此处就用到的三种方式:hive类SQL语句方式、importtsv +completebulkload 方式、mapreduce+completebulkload 方式,做下简单示例。其中当属hive类SQL语句方式最简单,首先介绍之:

实例中,我以虚拟话单作为需要导入的数据,格式如下:

view sourceprint?
01.1,12026546272,2013/10/19,20:52,3318秒,被叫,13727310234,北京市,省际,0,32.28,0.4,全球通商旅88套餐
02.2,12026546272,2013/10/19,20:23,3318秒,被叫,13727310234,北京市,省际,0,32.28,0.4,全球通商旅88套餐
03.3,16072996404,2013/10/19,20:52,1052秒,主叫,19271253211,北京市,省际,0,2.8,1.9,全球通商旅88套餐
04.4,10023895821,2013/10/19,20:52,0920秒,被叫,15115468122,绵阳市,省内,0,45.91,5.26,全球通商旅88套餐
05.5,13381653644,2013/10/19,20:53,0600秒,被叫,10991482287,北京市,省际,0,54.79,7.16,全球通商旅88套餐
06.6,18695195919,2013/10/19,21:37,2700秒,主叫,14858652217,绵阳市,省内,0,36.27,6.68,全球通商旅88套餐
07.7,11396010469,2013/10/19,21:37,2702秒,主叫,12939968466,绵阳市,省内,0,65.63,4.45,全球通商旅88套餐
08.8,15109754362,2013/10/19,21:37,0500秒,被叫,14240771580,绵阳市,省内,0,66.86,5.75,全球通商旅88套餐
09.9,13845944798,2013/10/19,21:37,1350秒,被叫,13648619896,广州市,省际,0,60.71,3.39,全球通商旅88套餐
10.10,17883953443,2013/10/19,21:38,3754秒,被叫,10110778698,广州市,省际,0,55.14,1.45,全球通商旅88套餐
11.11,19643495044,2013/10/19,21:38,4934秒,主叫,14581482419,广州市,省际,0,16.84,1.36,全球通商旅88套餐

步骤如下:

1、首先在hive创建表,创建hbase识别的表bill:

在hive的shell里面执行命令:

view sourceprint?
1.Drop table bill;
2.CREATE TABLE BILLS(selfnumber string,day string,hour string,duration string,calltype string, targetnumber string,address string, longdtype string, basecost float, longdcost float, infocost float,privilege string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, calltime:day,calltime:hour,dura:duration,info:calltype,info:targetnumber,info:address,info:longdtype,info:basecost,info:longdcost,info:infocost,info:privilege")TBLPROPERTIES ("hbase.table.name" = "bills");

首先如果bill表已经存在则删除之。之后建立一个hbase可识别的表,可见里面规定了hbase列族等信息。

注意:不能有敏感关键字,比如”date”。

\

“hbase.columns.mapping”=后面的第一个不要写第一列即作为row的那一列,否则报错:

FAILED: Error in metadata:java.lang.RuntimeException:MetaException(message:org.apache.hadoop.hive.serde2.SerDeExceptionorg.apache.hadoop.hive.hbase.HBaseSerDe: columns has 12 elements whilehbase.columns.mapping has 13 elements (counting the key if implicit))

FAILED: Execution Error,return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

\

2、在hive创建一个表用于导数据进去:

view sourceprint?
1.create table pokes(selfnumber string,day string,hour string,duration string,calltype string, targetnumber string,address string, longdtype string, basecost float, longdcost float, infocost float,privilege string)row format delimited fields terminated by ',';

3、批量导入数据到刚刚建的hive表pokes:

预处理数据:

把数据中字段名等去掉,把连续的空格全部变为“,”分开。可以写程序做预处理,也可以使用脚本。

load data local inpath’/home/cdh4/Desktop/bill.txt’ overwrite into table pokes;

4、使用类sql语句把pokes里的数据导入到hbase可识别的表BILLS中去:

insert overwrite table bills select * from pokes;

5、在hive shell中查看数据:

hive> select* from bills;

注意:

1、hive首先要起动远程服务接口,命令:

nohup hive –service hiveserver &

2、java工程中导入相应的需求jar包,列表如下(红色必须):

antlr-runtime-3.0.1.jar

hive-exec-0.7.1.jar

hive-jdbc-0.7.1.jar

hive-metastore-0.7.1.jar

hive-service-0.7.1.jar

jdo2-api-2.3-ec.jar

libfb303.jar

3、在java代码中写sql语句的时候注意sql语句中的空格。!!!

报错处理:

1、插入时数据不成功报错:

Error:

java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

atorg.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)

atorg.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

atorg.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)

atorg.apache.hadoop.mapred.MapTask.run(MapTask.java:332)

atorg.apache.hadoop.mapred.Child$4.run(Child.java:268)

atjava.security.AccessController.doPrivileged(Native Method)

atjavax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

说明:hive的classpath还需要加入hbase、zookeeper的那些jar包

解决办法:在hive的conf/hive-site.xml里面加入属性:

<property>

<name>hive.aux.jars.path</name>

<value>file:///usr/hadoop/hive-0.7.1-cdh3u6/lib/hive-hbase-handler-0.7.1-cdh3u6.jar,file:///usr/hadoop/hive-0.7.1-cdh3u6/lib/hbase-0.90.6-cdh3u6.jar,file:///usr/hadoop/hive-0.7.1-cdh3u6/lib/zookeeper-3.3.1.jar</value>

</property>

2、如果导入数据时遇到报错:

Anon-native table cannot be used as target for LOAD

说明:Hive不能向非本地表导入数据。

解决办法:请检查代码里面的建的表。

3、如果hive执行mapreduce的时候遇到报错:

Exception in thread “main”java.io.IOException: Cannot initialize Cluster. Please check your configurationfor mapreduce.framework.name and the correspond server addresses.

说明:mapreduce.framework.name这个属性是MRv2即yarn中才需要配置的,在版本1下不需要,所以就很自然地找到了问题的所在,MRv2和hadoop本身整合在了一起,而MRv1和hadoop还是分开的,所以查看了下/etc/profile在配HADOOP_HOME的时候要配MR1的目录。

解决办法:请检查集群的环境变量。

只需要配MR1的环境变量就行。如:

(1)export HADOOP_HOME=/usr/hadoop/hadoop-2.0.0-mr1-cdh4.1.5

(2) export PATH=$HADOOP_HOME/bin:$PATH

OK!GOOD LUCK!小伙伴们加油!

来源URL:http://www.it165.net/database/html/201404/5846.html