Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32077

Support host-local shuffle data reading with external shuffle service disabled

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      After SPARK-27651, Spark can read host-local shuffle data directly from disk with external shuffle service enabled. To extend the future, we can also support it with external shuffle service disabled.

        Attachments

          Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27651 Avoid the network when block manager fetches shuffle blocks from the same host

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #28911 [Github] Pull Request #28911 (Ngone51)

          Pull request #28911 [Github] Pull Request #28911 (Ngone51)

            Activity

              People

              • Assignee:
                Ngone51 wuyi
                Reporter:
                Ngone51 wuyi
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: