Hadoop Knowledge base share

Thursday, December 14, 2017

Cloudera SSO/SAML /X.509 Login Handler/Shibboleth IDP/Configuration/Part3

In my Previous post i showed shibboleth installation and Cloudera SAML configuration. in this post i am going to show to how to configure shibboleth idp configuration for cloudera cm/hue/nav and release the attributes in the SAML assertion.

Update the following files

cat /app/shibboleth-idp/conf/ldap.properties|grep -v "^#"|sed "/^$/d"

idp.authn.LDAP.authenticator = adAuthenticator
idp.authn.LDAP.ldapURL = ldap://ad.tanu.com
idp.authn.LDAP.useStartTLS = false
idp.authn.LDAP.useSSL = true
idp.authn.LDAP.sslConfig = certificateTrust
idp.authn.LDAP.trustCertificates = /app/apache-tomcat-7.0.65/ssl/ca_certificate.pem
idp.authn.LDAP.returnAttributes = cn,mail,company
idp.authn.LDAP.baseDN = OU=Users,DC=tanu,DC=com
idp.authn.LDAP.subtreeSearch = true
idp.authn.LDAP.userFilter = (cn={0})
idp.authn.LDAP.bindDN = aduser@TANU
idp.authn.LDAP.bindDNCredential = tanu123
idp.authn.LDAP.dnFormat = %s@TANU
idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}
idp.attribute.resolver.LDAP.connectTimeout = %{idp.authn.LDAP.connectTimeout:PT3S}
idp.attribute.resolver.LDAP.responseTimeout = %{idp.authn.LDAP.responseTimeout:PT3S}
idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN:undefined}
idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN:undefined}
idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential:undefined}
idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}
idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates:undefined}

idp.attribute.resolver.LDAP.searchFilter = (cn=$resolutionContext.principal)

We need to update the ldap.properties file to get the users attributes like mail/ou..etc to include in saml assertion post authentication.

cat /app/shibboleth-idp/conf/idp.properties |grep -v "^#"|sed "/^$/d"

idp.additionalProperties= /conf/ldap.properties, /conf/saml-nameid.properties, /conf/services.properties, /conf/authn/duo.properties
idp.entityID= https://idp.tanu.com:8443/idp/shibboleth
idp.scope= am.hedani.net
idp.sealer.storeResource= %{idp.home}/credentials/sealer.jks
idp.sealer.versionResource= %{idp.home}/credentials/sealer.kver
idp.sealer.storePassword= tanu123
idp.sealer.keyPassword= tanu123
idp.signing.key= %{idp.home}/credentials/idp-signing.key
idp.signing.cert= %{idp.home}/credentials/idp-signing.crt
idp.encryption.key= %{idp.home}/credentials/idp-encryption.key
idp.encryption.cert= %{idp.home}/credentials/idp-encryption.crt
idp.encryption.optional = false
idp.authn.flows= X509
idp.ui.fallbackLanguages= en,fr,de

Note that important property line idp.authn.flows = X509. Since i am going to use ssl mutual authentication handler, i have updated X509. Shibboleth also support many authentication handler

cat /app/shibboleth-idp/conf/authn/general-authn.xml |grep 'authn/'

<bean id="authn/IPAddress" parent="shibboleth.AuthenticationFlow"

<bean id="authn/SPNEGO" parent="shibboleth.AuthenticationFlow"

<bean id="authn/External" parent="shibboleth.AuthenticationFlow"

<bean id="authn/RemoteUser" parent="shibboleth.AuthenticationFlow"

<bean id="authn/X509" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Password" parent="shibboleth.AuthenticationFlow"

<bean id="authn/Duo" parent="shibboleth.AuthenticationFlow"

<bean id="authn/MFA" parent="shibboleth.AuthenticationFlow"

We can also use multi factor authentication by updating like below, if user can't able to login with username and password credential it will redirect to another authentication handler

idp.authn.flows= Password|X509

And also change the idp.authn.LDAP.userFilter based on your ldap and authentication handler. In my poc environment post authentication employee id will be returned to session and my active directory configured with cn="employee id". but shibboleth default filter will be sAMAccountName.

Update the /app/shibboleth-idp/conf/c14n/x500-subject-c14n-config.xml file

<util:list id="shibboleth.c14n.x500.Transforms">

</util:list>

Above marked regular expression is for extract the employee id from the certificate.

After successfully authenticated shibboleth will extract the common name from the client certificate. in our environment all employees common name(CN) would be cn=tanu sathish (12345).

So using above expression i am replacing shibboleth resolutionContext.principal session variable with only employee id instead of whole common name,.

Update the /app/shibboleth-idp/conf/metadata-providers.xml file

<MetadataProvider id="clouderaManager"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/cm-metadata.xml"

metadataURL="https://cm.tanu.com:7183/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="hueserver1"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/HuelocalCopyFromXYZHTTP.xml"

metadataURL="http://hue.tanu.com:8888/saml2/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

<MetadataProvider id="clouderaNavigator"

xsi:type="FileBackedHTTPMetadataProvider"

backingFile="%{idp.home}/metadata/clouderaNavigatorlocalCopyFromXYZHTTP.xml"

metadataURL="http://nav.tanu.com:7187/saml/metadata">

<RetainedRole>md:SPSSODescriptor</RetainedRole>

</MetadataFilter>

</MetadataProvider>

Update the /app/shibboleth-idp/conf/attribute-resolver.xml file to retrieve the attributes from ldap/active directory

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

</AttributeDefinition>

<DataConnector id="myLDAP" xsi:type="LDAPDirectory"

ldapURL="%{idp.attribute.resolver.LDAP.ldapURL}"

baseDN="%{idp.attribute.resolver.LDAP.baseDN}"

principal="%{idp.attribute.resolver.LDAP.bindDN}"

principalCredential="%{idp.attribute.resolver.LDAP.bindDNCredential}"

useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"

connectTimeout="%{idp.attribute.resolver.LDAP.connectTimeout}"

trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}"

responseTimeout="%{idp.attribute.resolver.LDAP.responseTimeout}">

<![CDATA[

%{idp.attribute.resolver.LDAP.searchFilter}

]]>

</FilterTemplate>

<ConnectionPool

minPoolSize="%{idp.pool.LDAP.minSize:3}"

maxPoolSize="%{idp.pool.LDAP.maxSize:10}"

blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"

validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"

validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"

expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"

failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />

</DataConnector>

</Attribute>

</DataConnector>

Note that i use static connector for attribute role which mean role attribute will be always user to all the authenticated users.

Update the /app/shibboleth-idp/conf/attribute-filter.xml file to release the attribute the respective SP entity.

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

Cloudera SSO/SAML /X.509 Login Handler/ Cloudera/Hue/navigator/Configuration /Part2

In my previous blog shibboleth IDP installation i posted how to install and setup shibboleth. In this post i am going to show how to configure cloudera manager/Hue/navgator Service provider.

Note that I am using cloudera manager 5.13.1

Prerequisites :

1) Create common folder in clodera manager and all the nodes like /opt/cloudera-manager/saml/

2) Download the shibboleth idp metdata xml from https://idp.tanu.com:8443/idp/shibboleth URL or copy from /app/test/shibboleth-idp/metadata/idp-metadata.xml to /opt/cloudera-manager/saml/ directory in all the servers.

3) shibboleth some time wont generate the metadata xml properly. so open the /opt/cloudera-manager/saml/idp-metadata.xml. update the idp correct url and port like below.

3)Setup ssl for clouder manager/Hue/Cloudera navigator, follow this link to implement pki for cloudera components https://hadoopguides.blogspot.com/2017/10/openssl-ca-authority-setup-with-san.html

4) Create additional SAML Keystore for CM/HUE/NAVIGATOR or you can use existing keystore files used for setting for SSL

Cloudera Manager SAML Configuration

1) go to https://cm.tanu.com:7183/cmf/settings

2) Update below settings

Authentication Backend Order Database then External

External Authentication Type SAML

Path to SAML IDP Metadata File

Path to SAML Keystore File

SAML Keystore Password

Alias of SAML Sign/Encrypt Private Key

SAML Sign/Encrypt Private Key Password

SAML Entity ID

SAML Entity Alias

SAML Response Binding

Source of User ID in SAML Response

SAML Attribute Identifier for User ID

SAML Role Assignment Mechanism

SAML Attribute Identifier for User Role

SAML Attribute Values for Roles

HUE SAML Configuration

go to Cluster/Hue/Configuration/Advanced/Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

[desktop]

redirect_whitelist="^\/.*$,^https:\/\/usls1005818.am.hedani.net:9443\/.*$"

[[auth]]

backend=desktop.auth.backend.AllowFirstUserDjangoBackend,libsaml.backend.SAML2Backend

[libsaml]

xmlsec_binary=/usr/bin/xmlsec1

metadata_file=/opt/cloudera-manager/saml/idp-metadata.xml

key_file=/opt/cloudera-manager/ssl/dummy/idp/huesaml.key

cert_file=/opt/cloudera-manager/ssl/dummy/idp/huemsalcert.pem

key_file_password="test123"

username_source=attributes

name_id_format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"

entity_id=hueserver1

create_users_on_login=true

authn_requests_signed=true

Navigator SAML Configuration

Go to Cloudera manager/Cloudera Management Service/Scope - Navigator Metadata Server/Category-External Authentication

Wednesday, December 13, 2017

Cloudera SSO/SAML /X.509 Login Handler/ shibboleth Installation /Part1

I was working on a requirement to setup password less/SSO using saml based authentication for cloudera applications particularly for cloudera manager/Hue/Cloudera navigator.

I opted shibboleth since it's open source and support federated identity and we can also use other identity management like ca siteminder, ping identity,oracle access manager if you have already in your infrastructure.

In this post i will show you how to install and configure shibboleth idp for cloudermanager/Hue/Cloudera navagator SP.

Shibboleth Installation :

Please note that, my tomcat instance is running on default port 8080 and 8443. So my shbbolenth final SAML EntityID URL will be https://idp.tanu.com:8443/idp/shibboleth..

If you setup tomcat instance in different port, change the port accordingly.

1) Download the shibboleth from this link https://shibboleth.net/downloads/identity-
provider/latest/shibboleth-identity-provider-3.3.2.tar.gz

2) Download Apache tomcat http://mirrors.gigenet.com/apache/tomcat/tomcat-7/v7.0.82/bin/apache-tomcat-7.0.82.tar.gz

3) Extract the tarball tar -zxvf shibboleth-identity-provider-3.3.2.tar.gz

4) cd shibboleth-identity-provider-3.3.2/bin

then run ./install.sh

Source (Distribution) Directory (press <enter> to accept default): [/app/test/shibboleth-identity-provider-3.3.2]

Installation Directory: [/opt/shibboleth-idp]
/app/test/shibboleth-idp
Hostname: [idp.tanu.com]

SAML EntityID: [https://idp.tanu.com/idp/shibboleth]
https://idp.tanu.com:8443/idp/shibboleth
Attribute Scope: [tanu.com]

Backchannel PKCS12 Password:
Re-enter password:
Cookie Encryption Key Password:
Re-enter password:
Warning: /app/test/shibboleth-idp/bin does not exist.
Warning: /app/test/shibboleth-idp/dist does not exist.
Warning: /app/test/shibboleth-idp/doc does not exist.
Warning: /app/test/shibboleth-idp/system does not exist.
Warning: /app/test/shibboleth-idp/webapp does not exist.
Generating Signing Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Encryption Key, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating Backchannel keystore, CN = idp.tanu.com URI = https://idp.tanu.com:8443/idp/shibboleth ...
...done
Creating cookie encryption key files...
...done
Rebuilding /app/test/shibboleth-idp/war/idp.war ...
...done

BUILD SUCCESSFUL
Total time: 52 seconds

5) copy the idp.war from /app/test/shibboleth-idp/war/idp.war to /app/apache-tomcat-7.0.65/webapps/

6) update the shibboleth home path in /app/apache-tomcat-7.0.65/bin/catalina.sh
JAVA_OPTS="$JAVA_OPTS -Didp.home=/app/test/shibboleth-idp"

7) shibboleth provide url to check the installation status but it need additional jar jstl-1.2.jar otherwise status url wont work.
Download http://central.maven.org/maven2/javax/servlet/jstl/1.2/jstl-1.2.jar jar and copy into /app/apache-tomcat-7.0.65/webapps/idp/WEB-INF/lib directory

8) Configure tomcat to support support SSL. I have already created jks key store and trust store for my tomcat instance

vi /app/apache-tomcat-7.0.65/conf/server.xml

<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="true" sslProtocol="TLS" keystoreFile="/app/apache-tomcat-7.0.65/ssl/idp-cert.jks" keystorePass="test123*"
truststoreFile="/app/apache-tomcat-7.0.65/ssl/ca_truststore.jks" truststorePass="test123" />


<Connector port="9009" protocol="AJP/1.3" redirectPort="8443" />

Please note that i have enabled client Auth = true since i am going to use X.509 Login Handler in shibboleth. Our client infrastructure use smartcard based authentication and all the internal sites are use ssl mutual authentication, so i also opted to use ssl mutual authentication for cloudera sites.. if you are going to use different shibboleth authentication handler, you can skip this entry.

9) Then start the tomcat instance.

10) verify the installation by accessing https://idp.tanu.com:8443/idp/status url

### Operating Environment Information
operating_system: Linux
operating_system_version: 2.6.32-642.15.1.el6.x86_64
operating_system_architecture: amd64
jdk_version: 1.8.0_131
available_cores: 2
used_memory: 225 MB
maximum_memory: 823 MB

### Identity Provider Information
idp_version: 3.3.2
start_time: 2017-12-13T10:57:51-05:00
current_time: 2017-12-13T10:57:52-05:00
uptime: 604 ms

service: shibboleth.LoggingService
last successful reload attempt: 2017-12-13T15:57:28Z
last reload attempt: 2017-12-13T15:57:28Z

service: shibboleth.ReloadableAccessControlService
last successful reload attempt: 2017-12-13T15:57:34Z
last reload attempt: 2017-12-13T15:57:34Z

service: shibboleth.MetadataResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z


service: shibboleth.RelyingPartyResolverService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.NameIdentifierGenerationService
last successful reload attempt: 2017-12-13T15:57:33Z
last reload attempt: 2017-12-13T15:57:33Z

service: shibboleth.AttributeResolverService
last successful reload attempt: 2017-12-13T15:57:32Z
last reload attempt: 2017-12-13T15:57:32Z

 DataConnector statiConnector: has never failed

 DataConnector myLDAP: has never failed

service: shibboleth.AttributeFilterService
last successful reload attempt: 2017-12-13T15:57:30Z
last reload attempt: 2017-12-13T15:57:30Z

Friday, December 1, 2017

convert p12 into pem and jks format

Convert p12 into pem

openssl pkcs12 -in certificate.p12 -nocerts -out key.pem
openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out cert.pem

convert p12 into jks

keytool -v -importkeystore -srckeystore ../certificate.p12 -srcstoretype PKCS12 -destkeystore keystore.ks -deststoretype JKS

Friday, November 17, 2017

Cloudera SAML Integration with shibboleth/ Ldap Authentication Handler

IDP Configuration

Download and extract the shibboleth-identity-provider-3.3.2 file in following dir

[tanu@cloudera:[ET] /cs/app/shibboleth-idp]$ ls -lrt /cs/app/shibboleth-idp
total 292
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 old-20171116-1025
drwxr-x--- 6 tanu tanu 4096 Nov 16 10:26 dist
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 doc
drwxr-x--- 6 tanu tanu 4096 Nov 16 10:26 system
drwxr-x--- 7 tanu tanu 4096 Nov 16 10:26 webapp
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 credentials
drwxr-x--- 5 tanu tanu 4096 Nov 16 10:26 edit-webapp
drwxr-x--- 4 tanu tanu 4096 Nov 16 10:26 flows
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 messages
drwxr-x--- 4 tanu tanu 4096 Nov 16 10:26 views
drwxr-x--- 2 tanu tanu 4096 Nov 16 10:26 war
-rw-r----- 1 tanu tanu 235520 Nov 16 11:29 conf.org.tar
drwxr-x--- 3 tanu tanu 4096 Nov 16 14:12 bin
drwxr-x--- 2 tanu tanu 4096 Nov 17 12:09 metadata
drwxr-x--- 2 tanu tanu 4096 Nov 17 12:48 logs
drwxr-x--- 6 tanu tanu 4096 Nov 17 13:04 conf

############################################################################

Update the access-control.xml file to allow users to connect to IDP servers

p:allowedRanges="#{ {'127.0.0.1/32', '::1/128','179.30.0.0/25','179.0.0.0/25'} }" />

Edit the /cs/app/shibboleth-idp/conf/ldap.properties file and update the following maked lines.

And Make sure there are no space at the end of each line otherwise services wont start properly

Ex idp.authn.LDAP.useStartTLS = false

########################################################

idp.authn.LDAP.authenticator = adAuthenticator

idp.authn.LDAP.ldapURL = ldaps://ad.tanu.com

idp.authn.LDAP.useStartTLS = false

idp.authn.LDAP.useSSL = true

idp.authn.LDAP.sslConfig = certificateTrust

idp.authn.LDAP.trustCertificates = /var/tmp/ca_cert.pem

idp.authn.LDAP.returnAttributes = sAMAccountName,mail,company

idp.authn.LDAP.baseDN = DC=users,DC=tanu,DC=net

idp.authn.LDAP.subtreeSearch = true

idp.authn.LDAP.userFilter = (sAMAccountName={0})

idp.authn.LDAP.bindDN = adbinduser@TANU

idp.authn.LDAP.bindDNCredential = test123

idp.authn.LDAP.dnFormat = %s@TANU

idp.attribute.resolver.LDAP.ldapURL = %{idp.authn.LDAP.ldapURL}

idp.attribute.resolver.LDAP.connectTimeout = %{idp.authn.LDAP.connectTimeout:PT3S}

idp.attribute.resolver.LDAP.responseTimeout = %{idp.authn.LDAP.responseTimeout:PT3S}

idp.attribute.resolver.LDAP.baseDN = %{idp.authn.LDAP.baseDN:undefined}

idp.attribute.resolver.LDAP.bindDN = %{idp.authn.LDAP.bindDN:undefined}

idp.attribute.resolver.LDAP.bindDNCredential = %{idp.authn.LDAP.bindDNCredential:undefined}

idp.attribute.resolver.LDAP.useStartTLS = %{idp.authn.LDAP.useStartTLS:true}

idp.attribute.resolver.LDAP.trustCertificates = %{idp.authn.LDAP.trustCertificates:undefined}

idp.attribute.resolver.LDAP.searchFilter = (sAMAccountName=$resolutionContext.principal)

##############################################################

Move the attribute-resolver.xml attribute-resolver.xml_bkf

cp attribute-resolver-ldap.xml attribute-resolver.xml

</AttributeDefinition>

<!--

In the rest of the world, the email address is the standard identifier,

despite the problems with that practice. Consider making the EPPN value

the same as your official email addresses whenever possible.

-->

</AttributeDefinition>

</AttributeDefinition>

<DataConnector id="myLDAP" xsi:type="LDAPDirectory"
ldapURL="%{idp.attribute.resolver.LDAP.ldapURL}"
baseDN="%{idp.attribute.resolver.LDAP.baseDN}"
principal="%{idp.attribute.resolver.LDAP.bindDN}"
principalCredential="%{idp.attribute.resolver.LDAP.bindDNCredential}"
useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"
connectTimeout="%{idp.attribute.resolver.LDAP.connectTimeout}"
trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}"
responseTimeout="%{idp.attribute.resolver.LDAP.responseTimeout}">
<FilterTemplate>
<![CDATA[
%{idp.attribute.resolver.LDAP.searchFilter}
]]>
</FilterTemplate>
<ConnectionPool
minPoolSize="%{idp.pool.LDAP.minSize:3}"
maxPoolSize="%{idp.pool.LDAP.maxSize:10}"
blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"
validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"
validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"
expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"
failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />
</DataConnector>

############################################################################

Edit the attribute-filter.xml

</AttributeRule>

</AttributeRule>

</AttributeRule>

</AttributeFilterPolicy>

######################################################################

Update the metadata-providers.xml with cloudera manager SP metadata URL

<MetadataProvider id="clouderaManager"
xsi:type="FileBackedHTTPMetadataProvider"
backingFile="%{idp.home}/metadata/clouderalocalCopyFromXYZHTTP.xml"
metadataURL="http://cm.tanu.com:7180/saml/metadata">

<MetadataFilter xsi:type="EntityRoleWhiteList">
<RetainedRole>md:SPSSODescriptor</RetainedRole>
</MetadataFilter>
</MetadataProvider>

Cloudera SP Configuration

1) Create SP keystore file

keytool -genkeypair -keystore cm-sp.keystore -keyalg RSA -alias node1 -dname "CN=cm.tanu.com,O=Hadoop" -storepass changeme -keypass changeme -validity 365

2) /cs/app/shibboleth-idp/metadata/idp-metadata.xml file into cloudera manager custom path

3) Go to Cloudera Manager -> Administration --> Setting --> External Authentication -> select SAML

Then restart the Manager, it should redirect to IDP server for authentication.

Monday, November 6, 2017

spark-shell proxy setting for scala programing

Today we tried to parse xml url throguh the spark-shell but we endup with below error

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
java.net.UnknownHostException: static.klipfolio.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

Although unix server configured with below environment variable, didn't work for me.

export http_proxy="http://proxy.tanu.com:8080"

export https_proxy="http://proxy.tanu.com:8080"

Then i tried to pass the proxy conf via spark.driver.extraJavaOptions arguments.

spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.tanu.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.tanu.com:8080 -Dhttps.proxyPort=8080"

scala> import scala.xml.XML

import scala.xml.XML

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")

xml: scala.xml.Elem =

<root>

<qtd>

<area>

<name>Sweden</name>

<bookings_q1>323458</bookings_q1>

<bookings_q2>245684</bookings_q2>

<bookings_q3>260098</bookings_q3>

<bookings_q4>250840</bookings_q4>

<trend>/images/resources/indicators/small/ind-circle-green.png</trend>

<on_target>/images/resources/indicators/small/ind-check-green.png</on_target>

</area>

<area>

<name>Norway</name>

<bookings_q1>196845</bookings_q1>

<bookings_q2>185625</bookings_q2>

<bookings_q3>226300</bookings_q3>

<bookings_q4>241915</bookings_q4>

<trend>/images/resources/indicators/small/ind-diamond-yellow.png</trend>

<on_target>/images/resources/...

scala>

Hue notebook Session 'xxx' not found." (error 404)

After we setup livy server with hue notebook, developers were able to run and save pyspark codes through hue notebook, however couldn't able to rerun the codes after they re-login with new session.

while troubleshooting we found that, hue is deleting the sessions once users are logged-off

127.0.0.1 - - [06/Nov/2017:16:14:25 +0000] "GET /sessions/3 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:26 +0000] "GET /sessions/3 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "POST /sessions/3/statements HTTP/1.1" 201 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "GET /sessions/3/statements/0 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:14:31 +0000] "GET /sessions/3/log?from=0 HTTP/1.1" 200 -
127.0.0.1 - - [06/Nov/2017:16:25:46 +0000] "DELETE /sessions/3 HTTP/1.1" 200 -

So how to open the previously saved notebook sessions?

Solution is very simple:

Go to the notebook --> open the saved session --> then click on below highlighted

then click recreate, now you should be able to re-run the code.