Hadoop Knowledge base share: December 2017

Thursday, December 14, 2017

Cloudera SSO/SAML /X.509 Login Handler/Shibboleth IDP/Configuration/Part3

In my Previous post i showed shibboleth installation and Cloudera SAML configuration. in this post i am going to show to how to configure shibboleth idp configuration for cloudera cm/hue/nav and release the attributes in the SAML assertion.

Update the following files

cat /app/shibboleth-idp/conf/ldap.properties|grep -v "^#"|sed "/^$/d"

idp.authn.LDAP.authenticator = adAuthenticator
idp.authn.LDAP.ldapURL = ldap://ad.tanu.com
idp.authn.LDAP.useStartTLS = false
idp.authn.LDAP.useSSL = true
idp.authn.LDAP.sslConfig = certificateTrust
idp.authn.LDAP.trustCertificates = /app/apache-tomcat-7.0.65/ssl/ca_certificate.pem
idp.authn.LDAP.returnAttributes = cn,mail,company
idp.authn.LDAP.baseDN = OU=Users,DC=tanu,DC=com
idp.authn.LDAP.subtreeSearch = true
idp.authn.LDAP.userFilter = (cn={0})
idp.authn.LDAP.bindDN = aduser@TANU
idp.authn.LDAP.bindDNCredential = tanu123
idp.authn.LDAP.dnFormat = %s@TANU
idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}
idp.attribute.resolver.LDAP.connectTimeout = %{idp.authn.LDAP.connectTimeout:PT3S}
idp.attribute.resolver.LDAP.responseTimeout = %{idp.authn.LDAP.responseTimeout:PT3S}
idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN:undefined}
idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN:undefined}
idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential:undefined}
idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}
idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates:undefined}

idp.attribute.resolver.LDAP.searchFilter = (cn=$resolutionContext.principal)

We need to update the ldap.properties file to get the users attributes like mail/ou..etc to include in saml assertion post authentication.

cat /app/shibboleth-idp/conf/idp.properties |grep -v "^#"|sed "/^$/d"

idp.additionalProperties= /conf/ldap.properties, /conf/saml-nameid.properties, /conf/services.properties, /conf/authn/duo.properties
idp.entityID= https://idp.tanu.com:8443/idp/shibboleth
idp.scope= am.hedani.net
idp.sealer.storeResource= %{idp.home}/credentials/sealer.jks
idp.sealer.versionResource= %{idp.home}/credentials/sealer.kver
idp.sealer.storePassword= tanu123
idp.sealer.keyPassword= tanu123
idp.signing.key= %{idp.home}/credentials/idp-signing.key
idp.signing.cert= %{idp.home}/credentials/idp-signing.crt
idp.encryption.key= %{idp.home}/credentials/idp-encryption.key
idp.encryption.cert= %{idp.home}/credentials/idp-encryption.crt
idp.encryption.optional = false
idp.authn.flows= X509
idp.ui.fallbackLanguages= en,fr,de

Note that important property line idp.authn.flows = X509. Since i am going to use ssl mutual authentication handler, i have updated X509. Shibboleth also support many authentication handler

cat /app/shibboleth-idp/conf/authn/general-authn.xml |grep 'authn/'

<bean id="authn/IPAddress" parent="shibboleth.AuthenticationFlow"

<bean id="authn/SPNEGO" parent="shibboleth.AuthenticationFlow"

<bean id="authn/External" parent="shibboleth.AuthenticationFlow"

<bean id="authn/RemoteUser" parent="shibboleth.AuthenticationFlow"

<bean id="authn/X509" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Password" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Duo" parent="shibboleth.AuthenticationFlow"

<bean id="authn/MFA" parent="shibboleth.AuthenticationFlow"

We can also use multi factor authentication by updating like below, if user can't able to login with username and password credential it will redirect to another authentication handler

idp.authn.flows= Password|X509

And also change the idp.authn.LDAP.userFilter based on your ldap and authentication handler. In my poc environment post authentication employee id will be returned to session and my active directory configured with cn="employee id". but shibboleth default filter will be sAMAccountName.

Update the /app/shibboleth-idp/conf/c14n/x500-subject-c14n-config.xml file

<util:list id="shibboleth.c14n.x500.Transforms">

</util:list>

Above marked regular expression is for extract the employee id from the certificate.

After successfully authenticated shibboleth will extract the common name from the client certificate. in our environment all employees common name(CN) would be cn=tanu sathish (12345).

So using above expression i am replacing shibboleth resolutionContext.principal session variable with only employee id instead of whole common name,.

Update the /app/shibboleth-idp/conf/metadata-providers.xml file

<MetadataProvider id="clouderaManager"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/cm-metadata.xml"

metadataURL="https://cm.tanu.com:7183/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="hueserver1"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/HuelocalCopyFromXYZHTTP.xml"

metadataURL="http://hue.tanu.com:8888/saml2/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="clouderaNavigator"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/clouderaNavigatorlocalCopyFromXYZHTTP.xml"

metadataURL="http://nav.tanu.com:7187/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

Update the /app/shibboleth-idp/conf/attribute-resolver.xml file to retrieve the attributes from ldap/active directory

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

<DataConnector id="myLDAP" xsi:type="LDAPDirectory"

ldapURL="%{idp.attribute.resolver.LDAP.ldapURL}"

baseDN="%{idp.attribute.resolver.LDAP.baseDN}"

principal="%{idp.attribute.resolver.LDAP.bindDN}"

principalCredential="%{idp.attribute.resolver.LDAP.bindDNCredential}"

useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"

connectTimeout="%{idp.attribute.resolver.LDAP.connectTimeout}"

trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}"

responseTimeout="%{idp.attribute.resolver.LDAP.responseTimeout}">

<![CDATA[

%{idp.attribute.resolver.LDAP.searchFilter}

]]>

</FilterTemplate>

<ConnectionPool

minPoolSize="%{idp.pool.LDAP.minSize:3}"

maxPoolSize="%{idp.pool.LDAP.maxSize:10}"

blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"

validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"

validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"

expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"

failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />

</DataConnector>

</Attribute>

</DataConnector>

Note that i use static connector for attribute role which mean role attribute will be always user to all the authenticated users.

Update the /app/shibboleth-idp/conf/attribute-filter.xml file to release the attribute the respective SP entity.

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

Cloudera SSO/SAML /X.509 Login Handler/ Cloudera/Hue/navigator/Configuration /Part2

In my previous blog shibboleth IDP installation i posted how to install and setup shibboleth. In this post i am going to show how to configure cloudera manager/Hue/navgator Service provider.

Note that I am using cloudera manager 5.13.1

Prerequisites :

1) Create common folder in clodera manager and all the nodes like /opt/cloudera-manager/saml/

2) Download the shibboleth idp metdata xml from https://idp.tanu.com:8443/idp/shibboleth URL or copy from /app/test/shibboleth-idp/metadata/idp-metadata.xml to /opt/cloudera-manager/saml/ directory in all the servers.

3) shibboleth some time wont generate the metadata xml properly. so open the /opt/cloudera-manager/saml/idp-metadata.xml. update the idp correct url and port like below.

3)Setup ssl for clouder manager/Hue/Cloudera navigator, follow this link to implement pki for cloudera components https://hadoopguides.blogspot.com/2017/10/openssl-ca-authority-setup-with-san.html

4) Create additional SAML Keystore for CM/HUE/NAVIGATOR or you can use existing keystore files used for setting for SSL

Cloudera Manager SAML Configuration

1) go to https://cm.tanu.com:7183/cmf/settings

2) Update below settings

Authentication Backend Order Database then External

External Authentication Type SAML

Path to SAML IDP Metadata File

Path to SAML Keystore File

SAML Keystore Password

Alias of SAML Sign/Encrypt Private Key

SAML Sign/Encrypt Private Key Password

SAML Entity ID

SAML Entity Alias

SAML Response Binding

Source of User ID in SAML Response

SAML Attribute Identifier for User ID

SAML Role Assignment Mechanism

SAML Attribute Identifier for User Role

SAML Attribute Values for Roles

HUE SAML Configuration

go to Cluster/Hue/Configuration/Advanced/Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

[desktop]

redirect_whitelist="^\/.*$,^https:\/\/usls1005818.am.hedani.net:9443\/.*$"

[[auth]]

backend=desktop.auth.backend.AllowFirstUserDjangoBackend,libsaml.backend.SAML2Backend

[libsaml]

xmlsec_binary=/usr/bin/xmlsec1

metadata_file=/opt/cloudera-manager/saml/idp-metadata.xml

key_file=/opt/cloudera-manager/ssl/dummy/idp/huesaml.key

cert_file=/opt/cloudera-manager/ssl/dummy/idp/huemsalcert.pem

key_file_password="test123"

username_source=attributes

name_id_format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"

entity_id=hueserver1

create_users_on_login=true

authn_requests_signed=true

Navigator SAML Configuration

Go to Cloudera manager/Cloudera Management Service/Scope - Navigator Metadata Server/Category-External Authentication

Wednesday, December 13, 2017

Cloudera SSO/SAML /X.509 Login Handler/ shibboleth Installation /Part1

I was working on a requirement to setup password less/SSO using saml based authentication for cloudera applications particularly for cloudera manager/Hue/Cloudera navigator.

I opted shibboleth since it's open source and support federated identity and we can also use other identity management like ca siteminder, ping identity,oracle access manager if you have already in your infrastructure.

In this post i will show you how to install and configure shibboleth idp for cloudermanager/Hue/Cloudera navagator SP.

Shibboleth Installation :

Please note that, my tomcat instance is running on default port 8080 and 8443. So my shbbolenth final SAML EntityID URL will be https://idp.tanu.com:8443/idp/shibboleth..

If you setup tomcat instance in different port, change the port accordingly.

1) Download the shibboleth from this link https://shibboleth.net/downloads/identity-
provider/latest/shibboleth-identity-provider-3.3.2.tar.gz

2) Download Apache tomcat http://mirrors.gigenet.com/apache/tomcat/tomcat-7/v7.0.82/bin/apache-tomcat-7.0.82.tar.gz

3) Extract the tarball tar -zxvf shibboleth-identity-provider-3.3.2.tar.gz

4) cd shibboleth-identity-provider-3.3.2/bin

then run ./install.sh

Source (Distribution) Directory (press <enter> to accept default): [/app/test/shibboleth-identity-provider-3.3.2]

Installation Directory: [/opt/shibboleth-idp]
/app/test/shibboleth-idp
Hostname: [idp.tanu.com]

SAML EntityID: [https://idp.tanu.com/idp/shibboleth]
https://idp.tanu.com:8443/idp/shibboleth
Attribute Scope: [tanu.com]

Backchannel PKCS12 Password:
Re-enter password:
Cookie Encryption Key Password:
Re-enter password:
Warning: /app/test/shibboleth-idp/bin does not exist.
Warning: /app/test/shibboleth-idp/dist does not exist.
Warning: /app/test/shibboleth-idp/doc does not exist.
Warning: /app/test/shibboleth-idp/system does not exist.
Warning: /app/test/shibboleth-idp/webapp does not exist.
Generating Signing Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Encryption Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Backchannel keystore, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating cookie encryption key files...
...done
Rebuilding /app/test/shibboleth-idp/war/idp.war ...
...done

BUILD SUCCESSFUL
Total time: 52 seconds

5) copy the idp.war from /app/test/shibboleth-idp/war/idp.war to /app/apache-tomcat-7.0.65/webapps/

6) update the shibboleth home path in /app/apache-tomcat-7.0.65/bin/catalina.sh
JAVA_OPTS="$JAVA_OPTS -Didp.home=/app/test/shibboleth-idp"

7) shibboleth provide url to check the installation status but it need additional jar jstl-1.2.jar otherwise status url wont work.
Download http://central.maven.org/maven2/javax/servlet/jstl/1.2/jstl-1.2.jar jar and copy into /app/apache-tomcat-7.0.65/webapps/idp/WEB-INF/lib directory

8) Configure tomcat to support support SSL. I have already created jks key store and trust store for my tomcat instance

vi /app/apache-tomcat-7.0.65/conf/server.xml

<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="true" sslProtocol="TLS" keystoreFile="/app/apache-tomcat-7.0.65/ssl/idp-cert.jks" keystorePass="test123*"
truststoreFile="/app/apache-tomcat-7.0.65/ssl/ca_truststore.jks" truststorePass="test123" />


<Connector port="9009" protocol="AJP/1.3" redirectPort="8443" />

Please note that i have enabled client Auth = true since i am going to use X.509 Login Handler in shibboleth. Our client infrastructure use smartcard based authentication and all the internal sites are use ssl mutual authentication, so i also opted to use ssl mutual authentication for cloudera sites.. if you are going to use different shibboleth authentication handler, you can skip this entry.

9) Then start the tomcat instance.

10) verify the installation by accessing https://idp.tanu.com:8443/idp/status url

### Operating Environment Information
operating_system: Linux
operating_system_version: 2.6.32-642.15.1.el6.x86_64
operating_system_architecture: amd64
jdk_version: 1.8.0_131
available_cores: 2
used_memory: 225 MB
maximum_memory: 823 MB

### Identity Provider Information
idp_version: 3.3.2
start_time: 2017-12-13T10:57:51-05:00
current_time: 2017-12-13T10:57:52-05:00
uptime: 604 ms

service: shibboleth.LoggingService
last successful reload attempt: 2017-12-13T15:57:28Z
last reload attempt: 2017-12-13T15:57:28Z

service: shibboleth.ReloadableAccessControlService
last successful reload attempt: 2017-12-13T15:57:34Z
last reload attempt: 2017-12-13T15:57:34Z

service: shibboleth.MetadataResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z


service: shibboleth.RelyingPartyResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.NameIdentifierGenerationService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.AttributeResolverService
last successful reload attempt: 2017-12-13T15:57:32Z
last reload attempt: 2017-12-13T15:57:32Z

 DataConnector statiConnector: has never failed

 DataConnector myLDAP: has never failed

service: shibboleth.AttributeFilterService
last successful reload attempt: 2017-12-13T15:57:30Z
last reload attempt: 2017-12-13T15:57:30Z

Friday, December 1, 2017

convert p12 into pem and jks format

Convert p12 into pem

openssl pkcs12 -in certificate.p12 -nocerts -out key.pem
openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out cert.pem

convert p12 into jks

keytool -v -importkeystore -srckeystore ../certificate.p12 -srcstoretype PKCS12 -destkeystore keystore.ks -deststoretype JKS