Hadoop Knowledge base share: 2017

Thursday, December 14, 2017

Cloudera SSO/SAML /X.509 Login Handler/Shibboleth IDP/Configuration/Part3

In my Previous post i showed shibboleth installation and Cloudera SAML configuration. in this post i am going to show to how to configure shibboleth idp configuration for cloudera cm/hue/nav and release the attributes in the SAML assertion.

Update the following files

cat /app/shibboleth-idp/conf/ldap.properties|grep -v "^#"|sed "/^$/d"

idp.authn.LDAP.authenticator = adAuthenticator
idp.authn.LDAP.ldapURL = ldap://ad.tanu.com
idp.authn.LDAP.useStartTLS = false
idp.authn.LDAP.useSSL = true
idp.authn.LDAP.sslConfig = certificateTrust
idp.authn.LDAP.trustCertificates = /app/apache-tomcat-7.0.65/ssl/ca_certificate.pem
idp.authn.LDAP.returnAttributes = cn,mail,company
idp.authn.LDAP.baseDN = OU=Users,DC=tanu,DC=com
idp.authn.LDAP.subtreeSearch = true
idp.authn.LDAP.userFilter = (cn={0})
idp.authn.LDAP.bindDN = aduser@TANU
idp.authn.LDAP.bindDNCredential = tanu123
idp.authn.LDAP.dnFormat = %s@TANU
idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}
idp.attribute.resolver.LDAP.connectTimeout = %{idp.authn.LDAP.connectTimeout:PT3S}
idp.attribute.resolver.LDAP.responseTimeout = %{idp.authn.LDAP.responseTimeout:PT3S}
idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN:undefined}
idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN:undefined}
idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential:undefined}
idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}
idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates:undefined}

idp.attribute.resolver.LDAP.searchFilter = (cn=$resolutionContext.principal)

We need to update the ldap.properties file to get the users attributes like mail/ou..etc to include in saml assertion post authentication.

cat /app/shibboleth-idp/conf/idp.properties |grep -v "^#"|sed "/^$/d"

idp.additionalProperties= /conf/ldap.properties, /conf/saml-nameid.properties, /conf/services.properties, /conf/authn/duo.properties
idp.entityID= https://idp.tanu.com:8443/idp/shibboleth
idp.scope= am.hedani.net
idp.sealer.storeResource= %{idp.home}/credentials/sealer.jks
idp.sealer.versionResource= %{idp.home}/credentials/sealer.kver
idp.sealer.storePassword= tanu123
idp.sealer.keyPassword= tanu123
idp.signing.key= %{idp.home}/credentials/idp-signing.key
idp.signing.cert= %{idp.home}/credentials/idp-signing.crt
idp.encryption.key= %{idp.home}/credentials/idp-encryption.key
idp.encryption.cert= %{idp.home}/credentials/idp-encryption.crt
idp.encryption.optional = false
idp.authn.flows= X509
idp.ui.fallbackLanguages= en,fr,de

Note that important property line idp.authn.flows = X509. Since i am going to use ssl mutual authentication handler, i have updated X509. Shibboleth also support many authentication handler

cat /app/shibboleth-idp/conf/authn/general-authn.xml |grep 'authn/'

<bean id="authn/IPAddress" parent="shibboleth.AuthenticationFlow"

<bean id="authn/SPNEGO" parent="shibboleth.AuthenticationFlow"

<bean id="authn/External" parent="shibboleth.AuthenticationFlow"

<bean id="authn/RemoteUser" parent="shibboleth.AuthenticationFlow"

<bean id="authn/X509" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Password" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Duo" parent="shibboleth.AuthenticationFlow"

<bean id="authn/MFA" parent="shibboleth.AuthenticationFlow"

We can also use multi factor authentication by updating like below, if user can't able to login with username and password credential it will redirect to another authentication handler

idp.authn.flows= Password|X509

And also change the idp.authn.LDAP.userFilter based on your ldap and authentication handler. In my poc environment post authentication employee id will be returned to session and my active directory configured with cn="employee id". but shibboleth default filter will be sAMAccountName.

Update the /app/shibboleth-idp/conf/c14n/x500-subject-c14n-config.xml file

<util:list id="shibboleth.c14n.x500.Transforms">

</util:list>

Above marked regular expression is for extract the employee id from the certificate.

After successfully authenticated shibboleth will extract the common name from the client certificate. in our environment all employees common name(CN) would be cn=tanu sathish (12345).

So using above expression i am replacing shibboleth resolutionContext.principal session variable with only employee id instead of whole common name,.

Update the /app/shibboleth-idp/conf/metadata-providers.xml file

<MetadataProvider id="clouderaManager"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/cm-metadata.xml"

metadataURL="https://cm.tanu.com:7183/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="hueserver1"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/HuelocalCopyFromXYZHTTP.xml"

metadataURL="http://hue.tanu.com:8888/saml2/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="clouderaNavigator"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/clouderaNavigatorlocalCopyFromXYZHTTP.xml"

metadataURL="http://nav.tanu.com:7187/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

Update the /app/shibboleth-idp/conf/attribute-resolver.xml file to retrieve the attributes from ldap/active directory

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

<DataConnector id="myLDAP" xsi:type="LDAPDirectory"

ldapURL="%{idp.attribute.resolver.LDAP.ldapURL}"

baseDN="%{idp.attribute.resolver.LDAP.baseDN}"

principal="%{idp.attribute.resolver.LDAP.bindDN}"

principalCredential="%{idp.attribute.resolver.LDAP.bindDNCredential}"

useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"

connectTimeout="%{idp.attribute.resolver.LDAP.connectTimeout}"

trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}"

responseTimeout="%{idp.attribute.resolver.LDAP.responseTimeout}">

<![CDATA[

%{idp.attribute.resolver.LDAP.searchFilter}

]]>

</FilterTemplate>

<ConnectionPool

minPoolSize="%{idp.pool.LDAP.minSize:3}"

maxPoolSize="%{idp.pool.LDAP.maxSize:10}"

blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"

validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"

validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"

expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"

failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />

</DataConnector>

</Attribute>

</DataConnector>

Note that i use static connector for attribute role which mean role attribute will be always user to all the authenticated users.

Update the /app/shibboleth-idp/conf/attribute-filter.xml file to release the attribute the respective SP entity.

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

Cloudera SSO/SAML /X.509 Login Handler/ Cloudera/Hue/navigator/Configuration /Part2

In my previous blog shibboleth IDP installation i posted how to install and setup shibboleth. In this post i am going to show how to configure cloudera manager/Hue/navgator Service provider.

Note that I am using cloudera manager 5.13.1

Prerequisites :

1) Create common folder in clodera manager and all the nodes like /opt/cloudera-manager/saml/

2) Download the shibboleth idp metdata xml from https://idp.tanu.com:8443/idp/shibboleth URL or copy from /app/test/shibboleth-idp/metadata/idp-metadata.xml to /opt/cloudera-manager/saml/ directory in all the servers.

3) shibboleth some time wont generate the metadata xml properly. so open the /opt/cloudera-manager/saml/idp-metadata.xml. update the idp correct url and port like below.

3)Setup ssl for clouder manager/Hue/Cloudera navigator, follow this link to implement pki for cloudera components https://hadoopguides.blogspot.com/2017/10/openssl-ca-authority-setup-with-san.html

4) Create additional SAML Keystore for CM/HUE/NAVIGATOR or you can use existing keystore files used for setting for SSL

Cloudera Manager SAML Configuration

1) go to https://cm.tanu.com:7183/cmf/settings

2) Update below settings

Authentication Backend Order Database then External

External Authentication Type SAML

Path to SAML IDP Metadata File

Path to SAML Keystore File

SAML Keystore Password

Alias of SAML Sign/Encrypt Private Key

SAML Sign/Encrypt Private Key Password

SAML Entity ID

SAML Entity Alias

SAML Response Binding

Source of User ID in SAML Response

SAML Attribute Identifier for User ID

SAML Role Assignment Mechanism

SAML Attribute Identifier for User Role

SAML Attribute Values for Roles

HUE SAML Configuration

go to Cluster/Hue/Configuration/Advanced/Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

[desktop]

redirect_whitelist="^\/.*$,^https:\/\/usls1005818.am.hedani.net:9443\/.*$"

[[auth]]

backend=desktop.auth.backend.AllowFirstUserDjangoBackend,libsaml.backend.SAML2Backend

[libsaml]

xmlsec_binary=/usr/bin/xmlsec1

metadata_file=/opt/cloudera-manager/saml/idp-metadata.xml

key_file=/opt/cloudera-manager/ssl/dummy/idp/huesaml.key

cert_file=/opt/cloudera-manager/ssl/dummy/idp/huemsalcert.pem

key_file_password="test123"

username_source=attributes

name_id_format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"

entity_id=hueserver1

create_users_on_login=true

authn_requests_signed=true

Navigator SAML Configuration

Go to Cloudera manager/Cloudera Management Service/Scope - Navigator Metadata Server/Category-External Authentication

Wednesday, December 13, 2017

Cloudera SSO/SAML /X.509 Login Handler/ shibboleth Installation /Part1

I was working on a requirement to setup password less/SSO using saml based authentication for cloudera applications particularly for cloudera manager/Hue/Cloudera navigator.

I opted shibboleth since it's open source and support federated identity and we can also use other identity management like ca siteminder, ping identity,oracle access manager if you have already in your infrastructure.

In this post i will show you how to install and configure shibboleth idp for cloudermanager/Hue/Cloudera navagator SP.

Shibboleth Installation :

Please note that, my tomcat instance is running on default port 8080 and 8443. So my shbbolenth final SAML EntityID URL will be https://idp.tanu.com:8443/idp/shibboleth..

If you setup tomcat instance in different port, change the port accordingly.

1) Download the shibboleth from this link https://shibboleth.net/downloads/identity-
provider/latest/shibboleth-identity-provider-3.3.2.tar.gz

2) Download Apache tomcat http://mirrors.gigenet.com/apache/tomcat/tomcat-7/v7.0.82/bin/apache-tomcat-7.0.82.tar.gz

3) Extract the tarball tar -zxvf shibboleth-identity-provider-3.3.2.tar.gz

4) cd shibboleth-identity-provider-3.3.2/bin

then run ./install.sh

Source (Distribution) Directory (press <enter> to accept default): [/app/test/shibboleth-identity-provider-3.3.2]

Installation Directory: [/opt/shibboleth-idp]
/app/test/shibboleth-idp
Hostname: [idp.tanu.com]

SAML EntityID: [https://idp.tanu.com/idp/shibboleth]
https://idp.tanu.com:8443/idp/shibboleth
Attribute Scope: [tanu.com]

Backchannel PKCS12 Password:
Re-enter password:
Cookie Encryption Key Password:
Re-enter password:
Warning: /app/test/shibboleth-idp/bin does not exist.
Warning: /app/test/shibboleth-idp/dist does not exist.
Warning: /app/test/shibboleth-idp/doc does not exist.
Warning: /app/test/shibboleth-idp/system does not exist.
Warning: /app/test/shibboleth-idp/webapp does not exist.
Generating Signing Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Encryption Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Backchannel keystore, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating cookie encryption key files...
...done
Rebuilding /app/test/shibboleth-idp/war/idp.war ...
...done

BUILD SUCCESSFUL
Total time: 52 seconds

5) copy the idp.war from /app/test/shibboleth-idp/war/idp.war to /app/apache-tomcat-7.0.65/webapps/

6) update the shibboleth home path in /app/apache-tomcat-7.0.65/bin/catalina.sh
JAVA_OPTS="$JAVA_OPTS -Didp.home=/app/test/shibboleth-idp"

7) shibboleth provide url to check the installation status but it need additional jar jstl-1.2.jar otherwise status url wont work.
Download http://central.maven.org/maven2/javax/servlet/jstl/1.2/jstl-1.2.jar jar and copy into /app/apache-tomcat-7.0.65/webapps/idp/WEB-INF/lib directory

8) Configure tomcat to support support SSL. I have already created jks key store and trust store for my tomcat instance

vi /app/apache-tomcat-7.0.65/conf/server.xml

<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="true" sslProtocol="TLS" keystoreFile="/app/apache-tomcat-7.0.65/ssl/idp-cert.jks" keystorePass="test123*"
truststoreFile="/app/apache-tomcat-7.0.65/ssl/ca_truststore.jks" truststorePass="test123" />


<Connector port="9009" protocol="AJP/1.3" redirectPort="8443" />

Please note that i have enabled client Auth = true since i am going to use X.509 Login Handler in shibboleth. Our client infrastructure use smartcard based authentication and all the internal sites are use ssl mutual authentication, so i also opted to use ssl mutual authentication for cloudera sites.. if you are going to use different shibboleth authentication handler, you can skip this entry.

9) Then start the tomcat instance.

10) verify the installation by accessing https://idp.tanu.com:8443/idp/status url

### Operating Environment Information
operating_system: Linux
operating_system_version: 2.6.32-642.15.1.el6.x86_64
operating_system_architecture: amd64
jdk_version: 1.8.0_131
available_cores: 2
used_memory: 225 MB
maximum_memory: 823 MB

### Identity Provider Information
idp_version: 3.3.2
start_time: 2017-12-13T10:57:51-05:00
current_time: 2017-12-13T10:57:52-05:00
uptime: 604 ms

service: shibboleth.LoggingService
last successful reload attempt: 2017-12-13T15:57:28Z
last reload attempt: 2017-12-13T15:57:28Z

service: shibboleth.ReloadableAccessControlService
last successful reload attempt: 2017-12-13T15:57:34Z
last reload attempt: 2017-12-13T15:57:34Z

service: shibboleth.MetadataResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z


service: shibboleth.RelyingPartyResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.NameIdentifierGenerationService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.AttributeResolverService
last successful reload attempt: 2017-12-13T15:57:32Z
last reload attempt: 2017-12-13T15:57:32Z

 DataConnector statiConnector: has never failed

 DataConnector myLDAP: has never failed

service: shibboleth.AttributeFilterService
last successful reload attempt: 2017-12-13T15:57:30Z
last reload attempt: 2017-12-13T15:57:30Z

Friday, December 1, 2017

convert p12 into pem and jks format

Convert p12 into pem

openssl pkcs12 -in certificate.p12 -nocerts -out key.pem
openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out cert.pem

convert p12 into jks

keytool -v -importkeystore -srckeystore ../certificate.p12 -srcstoretype PKCS12 -destkeystore keystore.ks -deststoretype JKS

Friday, November 17, 2017

Cloudera SAML Integration with shibboleth/ Ldap Authentication Handler

IDP Configuration

Download and extract the shibboleth-identity-provider-3.3.2 file in following dir

[tanu@cloudera:[ET] /cs/app/shibboleth-idp]$ ls -lrt /cs/app/shibboleth-idp
total 292
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 old-20171116-1025
drwxr-x--- 6 tanu tanu 4096 Nov 16 10:26 dist
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 doc
drwxr-x--- 6 tanu tanu 4096 Nov 16 10:26 system
drwxr-x--- 7 tanu tanu 4096 Nov 16 10:26 webapp
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 credentials
drwxr-x--- 5 tanu tanu 4096 Nov 16 10:26 edit-webapp
drwxr-x--- 4 tanu tanu 4096 Nov 16 10:26 flows
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 messages
drwxr-x--- 4 tanu tanu 4096 Nov 16 10:26 views
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 war
-rw-r----- 1 tanu tanu 235520 Nov 16 11:29 conf.org.tar
drwxr-x--- 3 tanu tanu 4096 Nov 16 14:12 bin
drwxr-x--- 2 tanu tanu 4096 Nov 17 12:09 metadata
drwxr-x--- 2 tanu tanu 4096 Nov 17 12:48 logs
drwxr-x--- 6 tanu tanu 4096 Nov 17 13:04 conf

############################################################################

Update the access-control.xml file to allow users to connect to IDP servers

p:allowedRanges="#{ {'127.0.0.1/32', '::1/128','179.30.0.0/25','179.0.0.0/25'} }" />

Edit the /cs/app/shibboleth-idp/conf/ldap.properties file and update the following maked lines.

And Make sure there are no space at the end of each line otherwise services wont start properly

Ex idp.authn.LDAP.useStartTLS = false

########################################################

idp.authn.LDAP.authenticator = adAuthenticator

idp.authn.LDAP.ldapURL = ldaps://ad.tanu.com

idp.authn.LDAP.useStartTLS = false

idp.authn.LDAP.useSSL = true

idp.authn.LDAP.sslConfig = certificateTrust

idp.authn.LDAP.trustCertificates = /var/tmp/ca_cert.pem

idp.authn.LDAP.returnAttributes = sAMAccountName,mail,company

idp.authn.LDAP.baseDN = DC=users,DC=tanu,DC=net

idp.authn.LDAP.subtreeSearch = true

idp.authn.LDAP.userFilter = (sAMAccountName={0})

idp.authn.LDAP.bindDN = adbinduser@TANU

idp.authn.LDAP.bindDNCredential = test123

idp.authn.LDAP.dnFormat = %s@TANU

idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}

idp.attribute.resolver.LDAP.connectTimeout = %{idp.authn.LDAP.connectTimeout:PT3S}

idp.attribute.resolver.LDAP.responseTimeout = %{idp.authn.LDAP.responseTimeout:PT3S}

idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN:undefined}

idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN:undefined}

idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential:undefined}

idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}

idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates:undefined}

idp.attribute.resolver.LDAP.searchFilter = (sAMAccountName=$resolutionContext.principal)

##############################################################

Move the attribute-resolver.xml attribute-resolver.xml_bkf

cp attribute-resolver-ldap.xml attribute-resolver.xml

</AttributeDefinition>

<!--

In the rest of the world, the email address is the standard identifier,

despite the problems with that practice. Consider making the EPPN value

the same as your official email addresses whenever possible.

-->

</AttributeDefinition>

</AttributeDefinition>

<DataConnector id="myLDAP" xsi:type="LDAPDirectory"
ldapURL="%{idp.attribute.resolver.LDAP.ldapURL}"
baseDN="%{idp.attribute.resolver.LDAP.baseDN}"
principal="%{idp.attribute.resolver.LDAP.bindDN}"
principalCredential="%{idp.attribute.resolver.LDAP.bindDNCredential}"
useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"
connectTimeout="%{idp.attribute.resolver.LDAP.connectTimeout}"
trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}"
responseTimeout="%{idp.attribute.resolver.LDAP.responseTimeout}">
<FilterTemplate>
<![CDATA[
%{idp.attribute.resolver.LDAP.searchFilter}
]]>
</FilterTemplate>
<ConnectionPool
minPoolSize="%{idp.pool.LDAP.minSize:3}"
maxPoolSize="%{idp.pool.LDAP.maxSize:10}"
blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"
validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"
validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"
expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"
failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />
</DataConnector>

############################################################################

Edit the attribute-filter.xml

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

######################################################################

Update the metadata-providers.xml with cloudera manager SP metadata URL

<MetadataProvider id="clouderaManager"
xsi:type="FileBackedHTTPMetadataProvider"
backingFile="%{idp.home}/metadata/clouderalocalCopyFromXYZHTTP.xml"
metadataURL="http://cm.tanu.com:7180/saml/metadata">

<MetadataFilter xsi:type="EntityRoleWhiteList">
<RetainedRole>md:SPSSODescriptor</RetainedRole>
</MetadataFilter>
</MetadataProvider>

Cloudera SP Configuration

1) Create SP keystore file

keytool -genkeypair -keystore cm-sp.keystore -keyalg RSA -alias node1 -dname "CN=cm.tanu.com,O=Hadoop" -storepass changeme -keypass changeme -validity 365

2) /cs/app/shibboleth-idp/metadata/idp-metadata.xml file into cloudera manager custom path

3) Go to Cloudera Manager -> Administration --> Setting --> External Authentication -> select SAML

Then restart the Manager, it should redirect to IDP server for authentication.

Monday, November 6, 2017

spark-shell proxy setting for scala programing

Today we tried to parse xml url throguh the spark-shell but we endup with below error

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
java.net.UnknownHostException: static.klipfolio.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

Although unix server configured with below environment variable, didn't work for me.

export http_proxy="http://proxy.tanu.com:8080"

export https_proxy="http://proxy.tanu.com:8080"

Then i tried to pass the proxy conf via spark.driver.extraJavaOptions arguments.

spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.tanu.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.tanu.com:8080 -Dhttps.proxyPort=8080"

scala> import scala.xml.XML

import scala.xml.XML

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")

xml: scala.xml.Elem =

<root>

<qtd>

<area>

<name>Sweden</name>

<bookings_q1>323458</bookings_q1>

<bookings_q2>245684</bookings_q2>

<bookings_q3>260098</bookings_q3>

<bookings_q4>250840</bookings_q4>

<trend>/images/resources/indicators/small/ind-circle-green.png</trend>

<on_target>/images/resources/indicators/small/ind-check-green.png</on_target>

</area>

<area>

<name>Norway</name>

<bookings_q1>196845</bookings_q1>

<bookings_q2>185625</bookings_q2>

<bookings_q3>226300</bookings_q3>

<bookings_q4>241915</bookings_q4>

<trend>/images/resources/indicators/small/ind-diamond-yellow.png</trend>

<on_target>/images/resources/...

scala>

Hue notebook Session 'xxx' not found." (error 404)

After we setup livy server with hue notebook, developers were able to run and save pyspark codes through hue notebook, however couldn't able to rerun the codes after they re-login with new session.

while troubleshooting we found that, hue is deleting the sessions once users are logged-off

127.0.0.1 - - [06/Nov/2017:16:14:25 +0000] "GET /sessions/3 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:26 +0000] "GET /sessions/3 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "POST /sessions/3/statements HTTP/1.1" 201 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "GET /sessions/3/statements/0 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "GET /sessions/3/log?from=0 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:25:46 +0000] "DELETE /sessions/3 HTTP/1.1" 200 -

So how to open the previously saved notebook sessions?

Solution is very simple:

Go to the notebook --> open the saved session --> then click on below highlighted

then click recreate, now you should be able to re-run the code.

Tuesday, October 24, 2017

Unix IPTABLE ipforward

In my home lab, I have KVM installed on HP server and i was managing all the virtul machine through my Ubuntu installed laptop using virt-manager client.

but i couldn't access virtual machines in my wifi enabled networks and there is no virt-manager client for windows operating system.

So i tried to work on setup to configure my KVM machine as router to access all the VM within wifi network lan.

here is the my home lab and wifi networks

KVM machine has

one Ethernet card enp4s0f1 connected to 192.168.1.6( wifi network)
another one is kvm bridge network card connected to 192.168.100.1(vm private network)

Windows Laptop:

My Windows Laptop connected to 192.168.1.4 network)

Using putty on Windows laptop, i was able to connect KVM machine directly and not the virtual machines.

And also if i run any applications(webservers/cloudera manager) i was not able access in my windows laptop.

On Windows Laptop

Update the routing table to send all 192.168.100.0/24 (vm private network) traffic to gatway 192.168.1.6(KVM wifi network ipaddress)

route add 192.168.100.0 mask 255.255.255.0 192.168.1.6

sudo route add -net 192.168.100.0/24 gw 192.168.1.6 ( On Linux Dekstop)

On KVM Machine

Enable the IPFORWARD

update /etc/sysctl.conf file
net.ipv4.ip_forward=1

run this command to reflect the change.
sysctl -p

IF IPTABLE is enabled, update below rule.

-A FORWARD -i enp4s0f1 -o virbr1 -j ACCEPT

-I FORWARD 1 -j LOG --log-prefix "RULE4:" --log-level 7 ( to enable debug)
-I FORWARD -p tcp --dport 22 -j ACCEPT
-I FORWARD -p tcp --dport 7180 -j ACCEPT ( to access cloudera manager url)
-I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Make sure firewall stopped in gust host(systemctl stop firewalld)

Wednesday, October 18, 2017

Adding TLS/SSL for Impala Services

Setting up the TLS for Impala is also very simple.

Follow this post OpenSSL CA Authority setup with SAN certificate for Cloudera to create SAN certificate for Impala service in pem format.

update the Pem Key/Certificate/key password/CA PEM file like below.

then restart the impala services.

Since my cluster is kerberos enabled, we need valid tiket to access the impala shell, otherwise you will get following error.

[hive@nm1 ~]$ impala-shell -i node1.tanu.com:21000 -k --ssl --ca_cert=/opt/cloude-newcert/MyRootCA.pem
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
-k requires a valid kerberos ticket but no valid kerberos ticket found.

Now create ticket using kinit command. And use below impala shell, now we are able to access database over TLS.

[hive@nm1 ~]$ kinit

Password for hive@TANU.COM:

[hive@nm1 ~]$ impala-shell -i node1.tanu.com:21000 -k --ssl --ca_cert=/opt/cloude-newcert/MyRootCA.pem

Starting Impala Shell using Kerberos authentication

Using service name 'impala'

SSL is enabled

Connected to node1.tanu.com:21000

Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9)

***********************************************************************************

Welcome to the Impala shell.

(Impala Shell v2.9.0-cdh5.12.1 (5131a03) built on Thu Aug 24 09:27:32 PDT 2017)

When pretty-printing is disabled, you can use the '--output_delimiter' flag to set

the delimiter for fields in the same row. The default is ','.

***********************************************************************************

[node1.tanu.com:21000] > show databases;

Query: show databases

+------------------+----------------------------------------------+

| name | comment |

+------------------+----------------------------------------------+

| _impala_builtins | System database for Impala builtin functions |

| default | Default Hive database |

| tanu | |

+------------------+----------------------------------------------+

Fetched 3 row(s) in 0.04s

OpenSSL CA Authority setup with SAN certificate for Cloudera

As i mentioned in my previous blog Secure hadoop/cloudera cluster using PKI implementation setting up PKI will be good practice, In that blog i didn't mentioned about SAN certificate which will be more important while setting up fronted load balancer for hive/impala services in a multi node cluster.

I have created simple shell script with input file, this will create Single ROOT CA keystore and certificate and it will create jks/pem formate keystore and truststore for hive/impala/cm/hue

Certificate input file

#LBNAME:NODES:FORMAT(PEM|JKS)
hive.tanu.com:nm1.tanu.com,node1.tanu.com,node2.tanu.com,hive.tanu.com:jks
hue.tanu.com:nm1.tanu.com,node1.tanu.com,node2.tanu.com,hue.tanu.com:pem
impala.tanu.com:nm1.tanu.com,node1.tanu.com,node2.tanu.com,impala.tanu.com:jks
cm.tanu.com:nm1.tanu.com,node1.tanu.com,node2.tanu.com,cm.tanu.com:jks

ROOT CA creation script:

root@ubuntu:/mnt/pool/usb-disk/X509CA/Project-cloudera1# cat Create_ROOTCA.sh
openssl genrsa -des3 -out MyRootCA.key 2048
openssl req -x509 -new -nodes -key MyRootCA.key -sha256 -days 5000 -out MyRootCA.pem -subj "/C=US/ST=NY/L=NYC/O=Global Security/OU=IT Department/CN=Hadoop CA Authority"

SAN Certificate Creating script JKS FORMAT

nodes=`cat certificates.txt|grep -i ":jks" |grep -v "^#" |cut -d":" -f2`
#array=( $san )
#echo "Number of elements: ${#array[@]}"
#count=1
export PATH=$PATH:/opt/jdk1.8.0_144/bin
for cnname in `cat certificates.txt|grep -i ":jks"|grep -v "^#"`
do
unset san
cn=$(echo $cnname|cut -d":" -f1)
count=1
echo "[ req ]">openssl-ext.cnf
echo "req_extensions = v3_req" >>openssl-ext.cnf
echo "[ v3_req ]" >>openssl-ext.cnf
echo "subjectAltName = @alt_names" >>openssl-ext.cnf
echo "[alt_names]" >>openssl-ext.cnf
for x in $(echo $cnname|cut -d":" -f2|tr "," "\n")
do
san+="dns:$x,"
echo "DNS.$count = $x">>openssl-ext.cnf
count=$((count + 1 ))
done
echo $san
mkdir $cn
keytool -genkey -alias $cn -keyalg RSA -keystore $cn/$cn.jks -keysize 2048 -dname "CN=$cn,OU=IT,O=IT,L=NYC,S=NY,C=US" -storepass cloud123 -keypass cloud123
keytool -certreq -alias $cn -keystore $cn/$cn.jks -file $cn/$cn.csr -ext SAN="$san" -storepass cloud123 -keypass cloud123
openssl x509 -req -in $cn/$cn.csr -CA MyRootCA.pem -CAkey MyRootCA.key -CAcreateserial -out $cn/$cn.pem -days 5000 -sha256 -extfile openssl-ext.cnf -extensions v3_req -passin pass:sathish123
echo "Importing CA Cetificate into clinet key"
keytool -importcert -trustcacerts -keystore $cn/$cn.jks -alias MyRootCA.pem -file MyRootCA.pem -storepass cloud123 -keypass cloud123 -noprompt
echo "Importing Certiicate into client keystore"
keytool -importcert -trustcacerts -keystore $cn/$cn.jks -alias $cn -file $cn/$cn.pem -storepass cloud123 -keypass cloud123 -noprompt
echo "Creating Truststore and import CA ceertificate"
keytool -importcert -keystore $cn/$cn.truststore -alias MyRootCA -file MyRootCA.pem -storepass cloud123 -keypass cloud123 -noprompt

done

SAN Certificate Creating script PEM FORMAT

root@ubuntu:/mnt/pool/usb-disk/X509CA/Project-cloudera1# cat pem__cert_create.sh

export PATH=$PATH:/opt/jdk1.8.0_144/bin

for cnname in `cat certificates.txt|grep -v "^#"`

unset san

cn=$(echo $cnname|cut -d":" -f1)

count=1

echo "[ req ]" >openssl-ext.cnf

echo "distinguished_name = req_distinguished_name" >>openssl-ext.cnf

echo "req_extensions = v3_req" >>openssl-ext.cnf

echo "prompt = no" >>openssl-ext.cnf

echo "[req_distinguished_name]" >>openssl-ext.cnf

echo "C = US" >>openssl-ext.cnf

echo "ST = NY" >>openssl-ext.cnf

echo "L = NYC" >>openssl-ext.cnf

echo "O = Global Security" >>openssl-ext.cnf

echo "OU = IT Department" >>openssl-ext.cnf

echo "CN = $cn" >>openssl-ext.cnf

echo "[ v3_req ]" >>openssl-ext.cnf

echo "subjectAltName = @alt_names" >>openssl-ext.cnf

echo "[alt_names]" >>openssl-ext.cnf

for x in $(echo $cnname|cut -d":" -f2|tr "," "\n")

san+="dns:$x,"

echo "DNS.$count = $x">>openssl-ext.cnf

count=$((count + 1 ))

done

echo $san

mkdir $cn

openssl genrsa -des3 -out $cn/$cn.key -passout pass:cloud123 2048

openssl req -new -out $cn/$cn.csr -key $cn/$cn.key -passin pass:cloud123 -config openssl-ext.cnf

openssl x509 -req -in $cn/$cn.csr -CA MyRootCA.pem -CAkey MyRootCA.key -CAcreateserial -out $cn/$cn.pem -days 5000 -sha256 -extfile openssl-ext.cnf -extensions v3_req -passin pass:sathish123

done

Cloudera director installation on AWS

Our client wanted to setup the cloudera data science workbench in on-prem however existing infrastructure is don't have support for redhat 7.2 build which is main prerequisite for cds. due to that our client decided to move the cloudera manager and data science workbench setup in AWS cloud.

I will post another blog on how to install the cloudera manager using CDS and cloudera data science workbench in AWS. In this blog i will share on , how we setup the cloudera director in AWS cloud.

Cloudera director is nothing but provide UI and command line interface to dynamically create Cloudera manager environments and spin/scale cloudera clusters dynamically in Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure by just providing how many instances we want for master/worker and gateway.

we have already crated VPC/ security groups/subnets/redhat 7.2 instance for cloudera director in aws.

Cloudera director installation:

1.Install the jdk1.8

2.install the wget packages if wget command is not working

3.create a Cloudera Director repository by running this command :cd /etc/yum.repos.d/
sudo wget "http://archive.cloudera.com/director/redhat/7/x86_64/director/cloudera-director.repo"

4.Install Cloudera Director server and client by running the following command:sudo yum install cloudera-director-server cloudera-director-client

5.Start the Cloudera Director server by running the following command:sudo service cloudera-director-server start

6.If the RHEL 7 or CentOS firewall is running on the EC2 instance where you have installed Cloudera Director, disable and stop the firewall with the following commands:sudo systemctl disable firewalld
sudo systemctl stop firewalld

Once the server is started, it will fail by default, because the backend metastore needs to be configured.

Edit the property file under /etc/cloudera-director-server/application.properties

Tuesday, October 17, 2017

Setting up Load balancer for HIVE in secure cluster with TLS support

Recently i got requirement from business to enable high availability with front end load balancer for hive and Impala services.

There are couple are issues i faced during the hive ha implementation in the kerberos enabled cluster with TLS.

1) one was load balancer host name (.i.e hive.tanu.com) was missing in hive.keytab, due to that beeline throwed below exception.

2) second one was each hive instance have own certificate with the common name of respective host name like cn=node1.tanu.com,cn=node2.tanu.com ..etc , when we use Load balancer host name i.e hive.tanu.com , beeline started throwing an exception ssl name not matching exception

hive@nm1 ~]$ beeline -u 'jdbc:hive2://hive.tanu.com:9009/default;principal=hive/_HOST@TANU.COM;ssl=true;sslTrustStore=/opt/hivekeystore/truststore.ts;trustStorePassword=sathish123;'
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 2ms
Connecting to jdbc:hive2://hive.tanu.com:9009/default;principal=hive/_HOST@TANU.COM;ssl=true;sslTrustStore=/opt/hivekeystore/truststore.ts;trustStorePassword=sathish123;
17/10/17 21:53:33 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]

[root@nm1 350-hive-HIVESERVER2]# klist -k -t -K hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 10/07/2017 20:24:24 hive/nm1.tanu.com@TANU.COM (0x370093cbe6049b97e2f5fd608e24d534)
1 10/07/2017 20:24:24 HTTP/nm1.tanu.com@TANU.COM (0xb2f42608e82b6b60e4b540084d8165ce)

To overcome all above issue,

first we need to add the load balancer hostname in cloudera --> cluster --> hive --> configuration --> search load balancer --> and update the load balancer hostname and port

then go to administration--> security --> kerberos Credentials --> Generate missing Credentials.

Once you able to complete above steps, you can see the hive lb hostname in hive.keytab

[root@nm1 369-hive-HIVESERVER2]# klist -k -t -K hive.keytab

Keytab name: FILE:hive.keytab

KVNO Timestamp Principal

---- ------------------- ------------------------------------------------------

1 10/17/2017 22:24:55 HTTP/nm1.tanu.com@TANU.COM (0xb2f42608e82b6b60e4b540084d8165ce)

1 10/17/2017 22:24:55 hive/hive.tanu.com@TANU.COM (0xc9499ade1c6b310252b661ed3096f548)

1 10/17/2017 22:24:55 hive/nm1.tanu.com@TANU.COM (0x370093cbe6049b97e2f5fd608e24d534)

And for the SSL cn miss match, we need to create the SAN certificate and include all the hive host names in the SAN field.

follow this blog post OpenSSL CA Authority setup with SAN certificate for Clouderafor how to create SAN certificate for hive and refer cloudera document fo enabling TLS support.

Enabling hive TLS support, refer the cloudera document, it will be pretty simple steps.