hadoop - Apache Spark accessing the data in hdfs through cross cluster -


i running spark on amazon emr public dns is, lets say, 23.21.40.15.

now executing spark jar on cluster & want write output of spark job other amazon emr hdfs public dns 29.45.56.72.

i able access own cluster hdfs i.e. 23.21.40.15 not able write cluster 29.45.56.72.

  • what need spark job can access cross cluster hdfs??
  • if possible, can share sample code this??

when set output dir in spark job can set credentials access this:

hdfs://username:password@hostname:port/pathtofolder 

pd: shouldn't write ips of cluster in public question ;)


Comments

Popular posts from this blog

matlab - error with cyclic autocorrelation function -

django - (fields.E300) Field defines a relation with model 'AbstractEmailUser' which is either not installed, or is abstract -

c# - What is a good .Net RefEdit control to use with ExcelDna? -