hadoop - Spark-shell with 'yarn-client' tries to load config from wrong location -
i'm trying launch bin/spark-shell , bin/pyspark laptop, connecting yarn cluster in yarn-client mode, , same error warn scriptbasedmapping: exception running /etc/hadoop/conf.cloudera.yarn1/topology.py 10.0.240.71 java.io.ioexception: cannot run program "/etc/hadoop/conf.cloudera.yarn1/topology.py" (in directory "/users/eugenezhulenev/projects/cloudera/spark"): error=2, no such file or directory spark trying run /etc/hadoop/conf.cloudera.yarn1/topology.py on laptop, not on worker node in yarn. this problem appeared after update spark 1.2.0 1.3.0 (cdh 5.4.2) the following steps temporarily work-around issue on cdh 5.4.4 cd ~ mkdir -p test-spark/ cd test-spark/ then copy files /etc/hadoop/conf.clouder.yarn1 1 worker node above (local) directory. , run spark-shell ~/test-spark/