- make it simple enough so that people actually use it
- provide a certain level of customization
- must be mapped on XStudio's data-model which is basically:
Now, here is what I’m thinking of:
Business Requirements:
For each requirement, the user will have to provide:
a) a "probability/frequency" P value (between 1 and 5)
b) a "consequence/cost" C value (between 1 and 5)
Then each requirement have a computed "business risk" Br=P*C (between 1 and 25)
Technical Specifications:
For each specification, the user will have to provide:
a) a "technical risk/complexity" R value (between 1 and 10)
Tests:
The system will parse the graph linking each test to its related specifications then its related requirements and will compute a "criticality" Cr value based on some calculation:
Not sure what is the best formula yet though.
(A and B are user-customizable constants.)
Formula#1: no requirement duplicate
Formula#2: all path
We could also recursively parse the graph and authorize duplicate requirements:
Output for the user:
This will allow providing many metrics to the user including 2 major ones:
1) a graph showing the distribution of tests (risk / number of test cases)
This is useful to check the correlation between risk and nb of test cases implemented per test (you do want that the more risky tests are tested the more in details so have the more test cases).
2) more importantly, some "risk-weighted-coverage" statistics
The normal coverage metrics give basic information about how much the requirements are covered but it does consider any requirement/specification with the same “weigth”. the "risk-weighted-coverage" would introduce in the calculation the criticality notion. If a highly critical test is succeeding and a low-criticity test is failing, the risk-weighted-coverage will be much better that the opposite way even though there are only 2 test.
Presentation will be as the standard coverage currently displayed.
Many other metrics might be deliverable to the user (need to think more about it).
I often find that giving many values in such fields is a bit redundant - as most people would only use ~3 values (High/Mid/Low, or Hard/Mid/Easy) and neglect any further refinement.
ReplyDeleteWhen not giving similar fields to the tests themselves, it means that all those who do not fully use requirements and traceability within the XStudio ALM DB, might not be able to make any use of these important tools, so you might want to consider two calculation methods - Req & Spec based vs. Tests based.
Same applies to complexity - which would be available only to those using Specs.
I am wondering if in addition to complexity of the feature, the complexity of the test itself should be considered (or at least easy/hard to run, or requires special configuration, or manual vs. automatic)
RiskBased Filtration of tests while adding to Campaigns might also be a very practical usage method - where one could select to create a campaign for running the top 15% riskiest items 1st (as sanity), then complete with the rest.
(That might also be a common regression decision)
After all - metrics are nice, but practical usage precedes :-)
These values may also be used in later stages, to give more power to traceability measurements - as one can assume a more risky/complicated Req might require more coverage in tests.
First of all, this feature is NOT implemented (yet). This is just a design we are thinking of.
ReplyDeletewe might polish this for a while before we actually start implementing it (as it's pretty impacting and complex).
Now, here are some answers to your questions:
1) if the user does not provide any P or C or R then the behaviour should be exactly the same as of Today.
2) Yes, we should think more about the cases where people use:
- only tests
- only specs and tests.
3) Yes again, using the risk factor to filter which test to include in a campaign is a very good idea.
ReplyDeleteOffice Setup
www.office.com/setup
Office Com Setup
Printer Tech Support
www.norton.com/setup