Front Page Forums Apache Spark How can we launch Spark application on YARN?

This topic contains 1 reply, has 1 voice, and was last updated by  dfbdteam5 2 months, 3 weeks ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #5904

    dfbdteam5
    Moderator

    What are the ways to launch Apache Spark over YARN?
    Explain the technique to launch Apache Spark over Hadoop YARN.

    #5906

    dfbdteam5
    Moderator

    One can launch a Spark Job over YARN by specifying the--master yarnoption as follows:

    spark-submit --class <main-class> --master yarn <jar file> [arguments]

    NOTE: Before running this job. You have to specify the path to Hadoop Configuration files as follows:

    For example: If the Hadoop Configuration files are in /usr/lib/hadoop/conf:
    export HADOOP_CONF_DIR=/usr/lib/hadoop/conf

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.