[Feature] Introduce yarn queue setting to manage the queue resouce

[RocMarshal]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

Intention

Maybe we could introduce the queue-resource-manage feature. There will be some advantage:

• Keeping the available yarn-queues in streampark is helpful to keep the boundary between Streampark and yarn-cluster.
  • Prevent queue abuse.
  • Prevent failed submission caused by non-existent queue.
• Make it easier for users to perceive and to user the available yarn queues.
  • Support select queue when users using the yarn-application & yarn-session mode to deploy the jobs.

What

• The expected outline of the yarn queue management

• Support select queue when deploying on yarn.

deploy yarn session

deploy yarn-application jobs

How

• Introduce the yarn queue table to store the queue info
  fields:

  • id
  • team_id
  • queue_label
  • description
  • create_time
  • modify_time
    unique key unq_team_id_queue_label (team_id, queue_label)

• Need to note: Check the queue references( by yarn-session clusters & yarn-application jobs) when delete yarn queue.

SubTasks

• Completed [Feature][subtask] Add SQL DDL & DML for Yarn queue management feature #2415
• Completed [Feature][subtask] Add backend declaration & implementation for Yarn queue management feature. #2416
• Completed [Feature][subtask] Add FE pages for Yarn queue management feature. #2418
• Completed [Feature][subtask] Support selection on yarn queue when create/edit yarn per-job/application-mode application & yarn-session cluster. #2420
• Completed [Feature][subtask] Add the documentation page about yarn queue management. #2470

Usage Scenario

No response

Related issues

#2299

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[RocMarshal]

The interface preview for FE

@Slf4j
@Validated
@RestController
@RequestMapping("flink/yarnQueue")
public class YarnQueueController {

  @Autowired private YarnQueueService yarnQueueService;

  /** Pageful request the queues in the teamId */
  @PostMapping("list")
  public RestResponse list(RestRequest request, Long teamId) {
      return RestResponse.success(IPage<YarnQueue>);
  }

  /**
   * Check for the yarn queue if exists in the team or the queueLabel format is invalid.
   * status      msg
   * 0           Success
   * 1           The queue already existed in the team
   * 2           The queueLabel format in invalid.
   */
  @PostMapping("check")
  public RestResponse check(YarnQueue yarnQueue) {
      ResponseResult checkResult;
      return RestResponse.success(checkResult);
  }

  @PostMapping("create")
  @RequiresPermissions("yarnQueue:create")
  public RestResponse create(YarnQueue yarnQueue) {
      return RestResponse.success(true/false);
  }

  @PostMapping("update")
  @RequiresPermissions("yarnQueue:update")
  public RestResponse update(YarnQueue yarnQueue) {
      return RestResponse.success();
  }

  @PostMapping("get")
  public RestResponse get(Long queueId) throws InternalException {
      YarnQueue yarnQueue;
      return RestResponse.success(yarnQueue);
  }

  @PostMapping("delete")
  @RequiresPermissions("yarnQueue:delete")
  public RestResponse delete(YarnQueue yarnQueue) {
        return RestResponse.success();
    }
  }

}

Hi, @wangsizhu0504 I make the draft of interfaces you would use. Please take a look if you have free time. Thank you~
CC @wolfboys

The definition of YarnQueue is here

[RocMarshal]

Discussion result: For compatibility issues, we only verify the queue label permission of the hostParams attribute, and ignore the queue label parameter verification at the dynamic parameter level

[RocMarshal]

#2467