Monday, November 6, 2017

spark-shell proxy setting for scala programing

Today we tried to parse xml url throguh the spark-shell but we endup with below error

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
java.net.UnknownHostException: static.klipfolio.com
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

Although  unix server configured with below environment variable, didn't work for me.

export http_proxy="http://proxy.tanu.com:8080"
export https_proxy="http://proxy.tanu.com:8080"

Then i tried to pass the proxy conf via spark.driver.extraJavaOptions arguments.

spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.tanu.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.tanu.com:8080 -Dhttps.proxyPort=8080"


scala> import scala.xml.XML
import scala.xml.XML

scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
xml: scala.xml.Elem =
<root>
        <qtd>
                <area>
                        <name>Sweden</name>
                        <bookings>1080180</bookings>
                        <bookings_q1>323458</bookings_q1>
                        <bookings_q2>245684</bookings_q2>
                        <bookings_q3>260098</bookings_q3>
                        <bookings_q4>250840</bookings_q4>
                        <weighted>1055232</weighted>
                        <trend>/images/resources/indicators/small/ind-circle-green.png</trend>
                        <on_target>/images/resources/indicators/small/ind-check-green.png</on_target>
                </area>
                <area>
                        <name>Norway</name>
                        <bookings>850685</bookings>
                        <bookings_q1>196845</bookings_q1>
                        <bookings_q2>185625</bookings_q2>
                        <bookings_q3>226300</bookings_q3>
                        <bookings_q4>241915</bookings_q4>
                        <weighted>1269685</weighted>
                        <trend>/images/resources/indicators/small/ind-diamond-yellow.png</trend>
                        <on_target>/images/resources/...
scala>

No comments:

Post a Comment