Separating the Archive+MS suites for clarity and less execution time

[viduship]

Description

Apart from fixing the TFA https://issues.redhat.com/browse/RHCEPHQE-16857
This PR would address the below items:

1. Reducing the execution time for the MS + Archive zone suite ( removed the tests that do not require archive zone but only need MS and added them to a new test suite.)
2. Add a new test suite that has HAproxy support for MS tests
3. Reduce the nodes and storage space required for deploying the archive and multisite. (before we were using 18 nodes, with this we will use only 12 nodes.)
4. Add EC pool support for archive zone tests as well.
5. The new multisite suite and conf file would require only 8 nodes ( 4 nodes per cluster.)
6. Add support for a product bug in 8.0 Bug 2325018, where the test "test M buckets multipart uploads on haproxy node " will fail in the MS + archive zone suite.

Passed Logs :

1. archive + multisite suite: http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-36CEDK/ ( failure is a known bug) test M buckets multipart uploads on haproxy node (Known issue in 8.0,Bug 2325018)

2. Only MS test suite: http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-DKLNQ9

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: viduship

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ckulal]

LGTM,
Since new suites are added here, please sure required suites are added in metadata file.