Loading

Kylin 3.1.2 cube构建过程报错 org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'

今天在使用kylin构建cube的时候报如下错误:

2022-10-09 16:45:00,932 ERROR [pool-12-thread-9] threadpool.DefaultScheduler:121 : ExecuteException job:f7b41a90-4c10-994d-e9b7-5c559e3d4a32
org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201)
	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:119)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201)
	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
	... 4 more
Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:83)
	at org.apache.kylin.engine.mr.steps.CuboidJob.configureMapperInputFormat(CuboidJob.java:177)
	at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:151)
	at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:156)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
	... 6 more
Caused by: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
	at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:80)
	... 10 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
	at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:2079)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:2066)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1578)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1570)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
	at com.sun.proxy.$Proxy73.getTable(Unknown Source)
	at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
	at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
	at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:88)
	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
	... 12 more

我在网上查找了相关的错误,发现找到的都是跟spark有关的,但是我的kylin没有使用spark进行构建。后来参考了网上的一些文章,能成功构建啦。
目前服务器的环境是:

hadoop 3.0.0-cdh6.3.2
hive 3.1.2
kylin 3.1.2

错误原因:应该是因为服务器的cdh最近升级到6.3.2,原先的hadoop2和hive2都升级为hadoop3和hive3,导致和kylin有点不兼容。

解决方法:

# 进入kylin的lib目录
cd $KYLIN_HOME/lib

# 将这两个jar包下载到该lib目录下
wget https://repo1.maven.org/maven2/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar
wget https://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.3/commons-configuration-1.3.jar

# 创建目录并进入
mkdir -p $KYLIN_HOME/bin/hadoop3_jars/cdh6
cd $KYLIN_HOME/bin/hadoop3_jars/cdh6

# 将这两个jar包下载到该目录下
wget https://repo1.maven.org/maven2/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar
wget https://cwiki.apache.org/confluence/download/attachments/173081375/hive-exec-1.21.2.3.1.0.0-78.jar?api=v2

# 重启kylin
$KYLIN_HOME/bin/kylin.sh restart

以上步骤完成后,重新到web页面构建cube,发现能成功构建了。

posted @ 2023-02-11 13:43  爱睡懒觉的我  阅读(438)  评论(0)    收藏  举报