Review Flow Adjustments - introducing asynchronous submissions and bi-weekly evaluations

[kaefergeneral]

TLDR:
We plan to move to asynchronous contributions evaluations, optionally allowing guilds to submit on each individual contribution up to every two weeks instead of one bulk submission every six weeks. The framework will be set up in a way that still accommodates comprehensive submissions every six weeks the way Guilds are used to, so people can take the time they need to adapt to the new system. The goal is to reduce organizational overhead, minimize the impact of deadlines and submission cut-offs, while at the same time improving communication, and increasing transparency and predictability of scores.

Over the past years, we have seen a significant increase in products developed by our guilds. With most guilds owning and maintaining more than one product or engagement, the reality is that a lot of the time, not everything is worked on simultaneously.
The current structure of one report every six weeks leaves little flexibility in the reporting, and sometimes, just missing a submission window because a release or update is delayed by a couple of days means waiting another six weeks to get a proper evaluation.

At the same time, the increased number of submissions creates crunch times for the OIG, leaving little room for coordinated communication with the Guilds during evaluations.

With version 6 of the Guidelines, we propose moving away from the 6-week cycle bulk submissions and instead to a more flexible bi-weekly submission where every product can be submitted individually. Guilds have a chance of submitting updates as they happen without waiting multiple weeks to submit a bulk submission covering all efforts. This should, however, be limited to one submission per contribution per two weeks.
Every two weeks, all submissions from the past two weeks will be rolled into one evaluation that is then performed over four days. Guilds can still bundle submissions into one more extensive PR if it's easier for them, but they can also submit individual PRs with updates within one two-week timeframe. It's important to note that Guilds will NOT be required to submit reports every two weeks. It's merely an option. If no updates are submitted for a product or engagement, we will still be revisiting each effort at regular intervals. To make it easy to track, each contribution will receive a date telling guilds when we would roll it into the following evaluation if no updates are submitted. This will, at most, happen in line with the current 6-week cycle, and any submission covering the contribution will reset the target date. Evaluations will only occur more often than now if a guild decides to submit additional reports. Metrics can be submitted in a bundled form at any time, without triggering an evaluation of all contributions reported upon. When an evaluation takes place, the OIG will simply default to the latest metrics. Metrics the OIG can easily collect themselves through services like waxmarketcap or dappradar do NOT have to be submitted as we always collect them during our evaluations anyway, no matter if they were submitted or not.

In addition to the current decay rate and target system, we will add a date depicting when the decay rate would be first applied if no updates are submitted. This will make scores more predictable, consistent, and transparent while allowing Guilds to plan their efforts more accordingly. With every evaluation decay rate, target, and date will be revised. For this it does not matter if the evaluation was triggered by a submission or if it as rolled into the evaluation automatically. This way Guilds will always know ahead of time if they are about to lose scores. The only exception to this are instances where a contribution or service is shut down, in which case it is flagged as concluded or abandoned accordingly, and the score is set to zero with immediate effect.

By allowing Guilds to submit individual reports on their contributions, we aim to be able to give more direct feedback on each individual submission beyond just assigning a score.

This change would result in a weekly update of scores happening every friday, as the evaluation of contributions and infrastructure would be alternating. To bring this in line, the evaluation of infrastructure would be shifted by two days, with snapshots being taken at the end of Tuesday, instead of Sunday.

Key Changes:

1. Bi-weekly Evaluations:
   • Guilds will be able to submit individual contributions for evaluation on an ongoing basis.
     The OIG will roll the submissions of two weeks into one bi-weekly evaluation.
   • Technical evaluations will remain on a two-week schedule, alternating with the review of contributions.
   • This change aims to allow the OIG to spend more time on each contribution and communicate more ****with guilds throughout the review process, giving direct feedback ****and increasing transparency.
2. Submission of Reports:
   • Report submission remains optional, not mandatory. Guilds are encouraged to submit reports only when their products or contributions undergo new updates or changes.
   • Guilds may submit a report at any time; we encourage guilds to submit reports on a per-contribution basis within the established ‘latest.md’ through Git Hub.
   • A 6-week evaluation cycle will remain in place as a shell structure. This means that products will be rolled into every third evaluation if no update is submitted in between. This will be captured by a new timestamp on every contribution, displaying when we will re-evaluate a specific contribution at the latest If no update is provided.
3. Decay
   • In addition to the current ‘Decay Rate’ and ‘Decay Target’, each contribution will receive a timestamp determining when the decay rate will be applied for the first time if no significant updates are provided or related requirements are not met.
   • ‘Decay Rate’, ‘Decay Target’, and the date will be revised with every evaluation.
4. Criteria Updates:
   • New criteria and guideline iterations will not be released on a fixed schedule but when deemed necessary.
   • Any changes to the guidelines will be proposed in advance, providing ample time for Guilds to review and give feedback before a potential addition is made.
   • New criteria will always include a sufficiently delayed effective date to allow Guilds time to acknowledge the changes and make the necessary adjustments.
5. Holiday and Vacation Exceptions:
   • There will be exceptions to the regular evaluation cadence during major holidays, specifically during Christmas and summer holidays.
   • The exact dates of these breaks will be announced in advance to allow Guilds to plan accordingly.

Evaluation Week Schedule:

1. Contributions Evaluation Week:

   • Monday to Thursday,

     Initial evaluation period by the OIG.

   • Thursday, 12:00 UTC to Friday, 12:00 UTC:

     Appeals window for guilds to contest evaluations.

   • Friday evening:

     Final results for the week will be released.

2. Technical Evaluation Week

   • Monday and Tuesday

     Timeslots for product introduction and guild update calls.

   • Wednesdays, 00:00 UTC:

     Snapshots of technical performance are taken.

   • Wednesdays, 00:00 UTC to Friday, 12:00 UTC

     Review of technical infrastructure takes place.

   • Friday evening:

     Final results for the week will be released.

As with all framework updates, this is a mere proposal and not final. We want to hear your ideas and concerns and will update or discard the proposal based on your feedback.

[TarasCL]

Looks fine for CryptoLions.
Thank you