Today we tried to parse xml url throguh the spark-shell but we endup with below error
scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
java.net.UnknownHostException: static.klipfolio.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
java.net.UnknownHostException: static.klipfolio.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
Although unix server configured with below environment variable, didn't work for me.
export http_proxy="http://proxy.tanu.com:8080"
export https_proxy="http://proxy.tanu.com:8080"
Then i tried to pass the proxy conf via spark.driver.extraJavaOptions arguments.
spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.tanu.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.tanu.com:8080 -Dhttps.proxyPort=8080"
scala> import scala.xml.XML
import scala.xml.XML
scala> val xml = XML.load("http://static.klipfolio.com/static/klips/saas/example_data/sales.xml")
xml: scala.xml.Elem =
<root>
<qtd>
<area>
<name>Sweden</name>
<bookings>1080180</bookings>
<bookings_q1>323458</bookings_q1>
<bookings_q2>245684</bookings_q2>
<bookings_q3>260098</bookings_q3>
<bookings_q4>250840</bookings_q4>
<weighted>1055232</weighted>
<trend>/images/resources/indicators/small/ind-circle-green.png</trend>
<on_target>/images/resources/indicators/small/ind-check-green.png</on_target>
</area>
<area>
<name>Norway</name>
<bookings>850685</bookings>
<bookings_q1>196845</bookings_q1>
<bookings_q2>185625</bookings_q2>
<bookings_q3>226300</bookings_q3>
<bookings_q4>241915</bookings_q4>
<weighted>1269685</weighted>
<trend>/images/resources/indicators/small/ind-diamond-yellow.png</trend>
<on_target>/images/resources/...
scala>
No comments:
Post a Comment