Each evaluation starts with the oracle script determining the primary reviewer of each guild. The orcale is executed on Github: https://github.com/wax-office-of-inspector-general/office/actions/workflows/guild-lottery.yml
The outcome is manually forwarded to the ‘monthly’ channel on Keybase. Infrastructure evaluations taking place during contributions evaluations apply the set of primary reviewers determined for the ongoing evaluation.
IN_PROGRESS > Pending > Reviewed > Prepared > T21 || STB || Failed
Before any reviewer starts their evaluation a template for the ongoing evaluation has to be created to ensure all IGs follow the same naming scheme.
When beginning the evalution of a Guild, the status tag is set to ‘IN_PROGRESS’ Scores should not be reset at this point.
The template for the new evaluatiuon will be inserted below the applicable Contributions Evalution, above the previous infrastructure evaluation. This mirrors the order of the “Total Score” table and ensures the most up to date Contributions and Infrastructure Evaluation are always together at the beginning of the Guild Page, and can be accessed without much scrolling.
When inserting a table (1) will be added to the title. This should be removed.
If applicable, the “Requirements” tables at the top have to be updated to mirror the guideline version applied.
An initial review of the guilds takes place.
Once the review is concluded, the guilds status is marked as ‘Pending’ peer review.
‘Pending’ guilds are then to be peer reviewed. The peer reviewer verifies the evaluation and updates the “Total Score” table, NOT the score “Total” at the top, and updates the Status to ‘Reviewed’. Special attention should be put to infrastructure not previously evaluated, as the addition of .e.g., ‘IPFS’ or ‘Testnet History’ has regularly been missed for consecutive evaluations.
Once all guilds are tagged as ’Reviewed’, the ‘Total’ scores of each guild are updated and the Status set to ‘Prepared’. At this point the IG handling this should ensure the version number in the ‘Total Scores’ table has been updated and doublcheck the score in this table is identical to the according infrastructure evaluation.
Only when all guilds are ‘Prepared’, the their status is set according to their position. “T21”, “STB” or “Failed’.
Abiding by this flow should minimize oversight errors, ensure all guilds are properly peer-reviewed, and allow us to keep the guild reports open, as it minimizes the times when scores do not reflect the current guild roster.