Setting up Load balancer for HIVE in secure cluster with TLS support

Recently i  got requirement from business to enable high availability with front end load balancer for hive and Impala services.

There are couple are issues i faced during the hive ha implementation in the kerberos enabled cluster with TLS.

1) one was load balancer host name (.i.e hive.tanu.com) was missing in hive.keytab, due to that beeline throwed below exception.

2) second one was each hive instance have own certificate with the  common name of respective host name like cn=node1.tanu.com,cn=node2.tanu.com ..etc , when we use Load balancer host name i.e hive.tanu.com , beeline started throwing an exception ssl name not matching exception

hive@nm1 ~]$ beeline -u 'jdbc:hive2://hive.tanu.com:9009/default;principal=hive/_HOST@TANU.COM;ssl=true;sslTrustStore=/opt/hivekeystore/truststore.ts;trustStorePassword=sathish123;'
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 2ms
Connecting to jdbc:hive2://hive.tanu.com:9009/default;principal=hive/_HOST@TANU.COM;ssl=true;sslTrustStore=/opt/hivekeystore/truststore.ts;trustStorePassword=sathish123;
17/10/17 21:53:33 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]

[root@nm1 350-hive-HIVESERVER2]# klist -k -t -K hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 10/07/2017 20:24:24 hive/nm1.tanu.com@TANU.COM (0x370093cbe6049b97e2f5fd608e24d534)
   1 10/07/2017 20:24:24 HTTP/nm1.tanu.com@TANU.COM (0xb2f42608e82b6b60e4b540084d8165ce)

To overcome all above issue, 

first we need to add the load balancer hostname in cloudera --> cluster --> hive --> configuration --> search load balancer --> and update the load balancer hostname and port

then go to administration--> security --> kerberos Credentials --> Generate missing Credentials.

Once you able to complete above steps, you can see the hive lb hostname in hive.keytab

[root@nm1 369-hive-HIVESERVER2]# klist -k -t -K hive.keytab 

Keytab name: FILE:hive.keytab

KVNO Timestamp           Principal

---- ------------------- ------------------------------------------------------

   1 10/17/2017 22:24:55 HTTP/nm1.tanu.com@TANU.COM (0xb2f42608e82b6b60e4b540084d8165ce)

   1 10/17/2017 22:24:55 hive/hive.tanu.com@TANU.COM (0xc9499ade1c6b310252b661ed3096f548)

   1 10/17/2017 22:24:55 hive/nm1.tanu.com@TANU.COM (0x370093cbe6049b97e2f5fd608e24d534)

And for the SSL cn  miss match, we need to create the SAN certificate and include all the hive host names in the SAN field.

follow this blog post  OpenSSL CA Authority setup with SAN certificate for Clouderafor how to create SAN certificate for hive and refer cloudera document fo enabling TLS support.

Enabling hive TLS support, refer the cloudera document, it will be pretty simple steps.