[CDH Cluster] Unable to start HiveServer2 Connection

Solved!
BillMurray
Level 1
[CDH Cluster] Unable to start HiveServer2 Connection

Hi, I am evaluating DSS, so I installed it in my server and added a 2 weeks enterprise trial license.



I am facing a Cloudera CDH 5.12 cluster, kerberized.



I am able to connect and browse HDFS, but Hive connection is not working.



This is the error in log file backlog.log




[2018/05/30-07:47:19.991] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool] - Creating a new HiveServer2 connection (db=default)
[2018/05/30-07:47:20.000] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool] - Connecting to HiveServer2: jdbc:hive2://cdhmaster02.my.corp:10000/default;principal=hive/cdhmaster02.my.corp@MY.CORP;user=dataiku/dataiku02.my.corp@MY.CORP
[2018/05/30-07:47:20.001] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool] - Driver version 1.1
[2018/05/30-07:47:20.004] [qtp1440621772-171] [DEBUG] [dku.tracing] - [ct: 26] Done call: /api/connections/get-hive-names time=26ms user=admin thrown=CodedException
[2018/05/30-07:47:20.005] [qtp1440621772-171] [ERROR] [dip.controllers] - API call '/dip/api/connections/get-hive-names/' failed
com.dataiku.dip.CodedException: Could not create HiveServer2 connection
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:363)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.take(HiveServer2SchemaHandler.java:30)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:42)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:34)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.listHiveDatabase(HiveServer2SchemaHandler.java:47)
at com.dataiku.dip.server.controllers.ConnectionsController.listHiveNames(ConnectionsController.java:113)
at com.dataiku.dip.server.controllers.ConnectionsController$$FastClassBySpringCGLIB$$149f32e4.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.dataiku.dip.server.controllers.AuditedCallAspect.doCall(AuditedCallAspect.java:40)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.dataiku.dip.server.controllers.CallTracingAspect.doCall(CallTracingAspect.java:141)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
at com.dataiku.dip.server.controllers.ConnectionsController$$EnhancerBySpringCGLIB$$a73f5553.listHiveNames(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)
at com.dataiku.dip.server.MissingParameterExceptionHandler.invokeHandlerMethod(MissingParameterExceptionHandler.java:18)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
at com.dataiku.dip.server.controllers.DKUDispatcherServlet.doDispatch(DKUDispatcherServlet.java:50)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:844)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
at com.dataiku.dip.shaker.server.ResourceFilter.doFilter(ResourceFilter.java:33)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:462)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:464)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:202)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:128)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:356)
... 68 more


Of course I tried to impersonate the user on the machine via kinit and open a beeline session, with success. For this, I copy-pasted the connection string in the log file, and if I add double quotes in the connection string, it works. Without it doesn't (which is correct).



For the record, this command issued from dataiku's node CLI succedes:




beeline -u "jdbc:hive2://cdhmaster02.my.corp:10000/default;principal=hive/cdhmaster02.my.corp@MY.CORP"


What do I need to check ??



Thanks



 



Bill

0 Kudos
1 Solution
Clément_Stenac
Dataiker
Hi,

It looks like you have added a "Connection property" in the Hive connection settings (in Administration > Settings > Hadoop) but didn't fill anything in the key. You will need to either make sure the property is filled or remove it.

View solution in original post

4 Replies
Clément_Stenac
Dataiker
Hi,

It looks like you have added a "Connection property" in the Hive connection settings (in Administration > Settings > Hadoop) but didn't fill anything in the key. You will need to either make sure the property is filled or remove it.
BillMurray
Level 1
Author
Wooo! You're right!

May I ask you how did you found it out from the log ?
0 Kudos
Clément_Stenac
Dataiker
Having the source code of DSS helps 🙂 We'll make it so that it outputs a proper error message in future releases.
0 Kudos
BillMurray
Level 1
Author
It would be really appreciated.
0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku