This topic contains 2 replies, has 1 voice, and was last updated by  dfbdteam5 10 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #5260

    dfbdteam5
    Moderator

    Explain API createOrReplaceTempView().

    #5264

    dfbdteam5
    Moderator
      <li style=”list-style-type: none”>
    • Its basic Dataset function.
    • Its under org.apache.spark.sql

     

    • def createOrReplaceTempView(viewName: String): Unit
    • Creates a temporary view using the given name.
    • The lifetime of this temporary view is tied to the SparkSession that was used to create this Dataset.

     

    <br />
    scala> val df = spark.read.csv("/home/hdadmin/titanic_data.txt")<br />
    df: org.apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields]</p>
    <p>scala> df.printSchema<br />
    root<br />
     |-- _c0: string (nullable = true)<br />
     |-- _c1: string (nullable = true)<br />
     |-- _c2: string (nullable = true)<br />
     |-- _c3: string (nullable = true)<br />
     |-- _c4: string (nullable = true)<br />
     |-- _c5: string (nullable = true)<br />
     |-- _c6: string (nullable = true)<br />
     |-- _c7: string (nullable = true)<br />
     |-- _c8: string (nullable = true)<br />
     |-- _c9: string (nullable = true)<br />
     |-- _c10: string (nullable = true)</p>
    <p>scala> df.show<br />
    +---+---+---+--------------------+-------+-----------+--------------------+-------+-----------------+-----+------+<br />
    |_c0|_c1|_c2|                 _c3|    _c4|        _c5|                 _c6|    _c7|              _c8|  _c9|  _c10|<br />
    +---+---+---+--------------------+-------+-----------+--------------------+-------+-----------------+-----+------+<br />
    |  1|1st|  1|Allen, Miss Elisa...|29.0000|Southampton|        St Louis, MO|    B-5|       24160 L221|    2|female|<br />
    |  2|1st|  0|Allison, Miss Hel...| 2.0000|Southampton|Montreal, PQ / Ch...|    C26|             null| null|female|<br />
    |  3|1st|  0|Allison, Mr Hudso...|30.0000|Southampton|Montreal, PQ / Ch...|    C26|             null|(135)|  male|<br />
    |  4|1st|  0|Allison, Mrs Huds...|25.0000|Southampton|Montreal, PQ / Ch...|    C26|             null| null|female|<br />
    |  5|1st|  1|Allison, Master H...| 0.9167|Southampton|Montreal, PQ / Ch...|    C22|             null|   11|  male|<br />
    |  6|1st|  1|  Anderson, Mr Harry|47.0000|Southampton|        New York, NY|   E-12|             null|    3|  male|<br />
    |  7|1st|  1|Andrews, Miss Kor...|63.0000|Southampton|          Hudson, NY|    D-7|        13502 L77|   10|female|<br />
    |  8|1st|  0|Andrews, Mr Thoma...|39.0000|Southampton|         Belfast, NI|   A-36|             null| null|  male|<br />
    |  9|1st|  1|Appleton, Mrs Edw...|58.0000|Southampton| Bayside, Queens, NY|  C-101|             null|    2|female|<br />
    | 10|1st|  0|Artagaveytia, Mr ...|71.0000|  Cherbourg| Montevideo, Uruguay|   null|             null| (22)|  male|<br />
    | 11|1st|  0|Astor, Colonel Jo...|47.0000|  Cherbourg|        New York, NY|   null|17754 L224 10s 6d|(124)|  male|<br />
    | 12|1st|  1|Astor, Mrs John J...|19.0000|  Cherbourg|        New York, NY|   null|17754 L224 10s 6d|    4|female|<br />
    | 13|1st|  1|Aubert, Mrs Leont...|     NA|  Cherbourg|       Paris, France|   B-35|     17477 L69 6s|    9|female|<br />
    | 14|1st|  1|Barkworth, Mr Alg...|     NA|Southampton|       Hessle, Yorks|   A-23|             null|    B|  male|<br />
    | 15|1st|  0| Baumann, Mr John D.|     NA|Southampton|        New York, NY|   null|             null| null|  male|<br />
    | 16|1st|  1|Baxter, Mrs James...|50.0000|  Cherbourg|        Montreal, PQ|B-58/60|             null|    6|female|<br />
    | 17|1st|  0|Baxter, Mr Quigg ...|24.0000|  Cherbourg|        Montreal, PQ|B-58/60|             null| null|  male|<br />
    | 18|1st|  0| Beattie, Mr Thomson|36.0000|  Cherbourg|        Winnipeg, MN|    C-6|             null| null|  male|<br />
    | 19|1st|  1|Beckwith, Mr Rich...|37.0000|Southampton|        New York, NY|   D-35|             null|    5|  male|<br />
    | 20|1st|  1|Beckwith, Mrs Ric...|47.0000|Southampton|        New York, NY|   D-35|             null|    5|female|<br />
    +---+---+---+--------------------+-------+-----------+--------------------+-------+-----------------+-----+------+<br />
    only showing top 20 rows</p>
    <p>scala> df.createOrReplaceTempView("titanicdata")</p>
    <p>
    #5266

    dfbdteam5
    Moderator

    NOTE : Please note that, this API is metioned as ‘Experimental’ in Spark API documentation

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.