XStudio 1.5: New settings to better control notifications

In XStudio 1.5, new settings will appear to better control email notification mechanisms:

Send additional notification per failed test case:
When you're running non-regression campaigns, this option can be useful as it will send you a specific notification as soon as one test case is failing (regression). The notification is sent immediately while the session is still running. One notification per failed test case is sent.

Send bug notification only on status change:
This option will limit the email notification to the case where the status of the bug has been changed.


Also, for the integrated bug-tracking database, a new butto allows sending a reminder to all people involved in the bug:

XStudio 1.5: User-groups, projects and access-rights

XStudio already has a great set of options to authorize users to execute certain actions or not depending on their profile. This is what we call "Actions rights".

With XStudio 1.5, a new feature will allow restricting users to view only a subset of the objects in the Data-Model. For instance, you will want some people to work on a specific project and you want them to be isolated from the rest (you don't want them to see what the others are working on).

To do so, you will be able to create groups of users. Each user group will be granted to access certain folders in ANY XStudio tree. This is what we call "Access rights". If a user is member of several groups, all the rights will be cumulated and given to the user.

Here are a few screenshots:

Members of a user group

Selection of Access Rights for a user group

User group members

XStudio 1.5: Microsoft Word document parser

Before using a real Test Management system, people often use Microsoft Word to write their requirements or specifications.

With XStudio 1.5, a tool allowing to read all your Word documents and generate an XML file will be delivered. Hence, you will be able to import very simply all your requirements/specifications in XStudio:

• Install and run Export from Word v2.0

• Indicate a type of export (Flat or Deep)
  
• Indicate the root folder containing all the Word documents
• Press "Generate XML"
• Open XStudio and import the generated file

1) the "Deep" mode reads the Word document and will interpret any "leaf" tree node as a requirement. All its parent chapters will be imported as folders in the requirement tree
2) the "Flat" mode reads on the summary of the Word document and import any entry (inluding the head numbering)

XStudio 1.5: Import requirements from Word documents

We are working at the moment on a Word exporter.
The goal is to be able to read/parse some Word document to export requirements in an XML file that XStudio can import.

2 modes are available:
1) the "deep" mode reads the Word document and will interpret any "leaf" tree node as a requirement. All its parent chapters will be imported as folders in the requirement tree
2) the "flat" mode reads on the summary of the Word document and import any entry (inluding the head numbering)

The programs takes 2 arguments:
ExportFromWord.exe inputFolder outputXML

All the Word documents in the input folder are parsed but only one single output XML is generated.

Example:
ExportFromWord.exe "C:\Documents and Settings\egavaldo\Desktop" "C:\exportFromWord.xml"

Study: Risk-based design

I'm seeing many ways to "implement" risk-based concepts but I have the following requirements and constraints:

- make it simple enough so that people actually use it

- provide a certain level of customization

- must be mapped on XStudio's data-model which is basically:

Now, here is what I’m thinking of:

Business Requirements:

For each requirement, the user will have to provide:

a) a "probability/frequency" P value (between 1 and 5)

b) a "consequence/cost" C value (between 1 and 5)

Then each requirement have a computed "business risk" Br=P*C (between 1 and 25)

Technical Specifications:

For each specification, the user will have to provide:

a) a "technical risk/complexity" R value (between 1 and 10)

Tests:

The system will parse the graph linking each test to its related specifications then its related requirements and will compute a "criticality" Cr value based on some calculation:

Not sure what is the best formula yet though.

(A and B are user-customizable constants.)

Formula#1: no requirement duplicate

This is the simple formula where there is no duplicate.

Formula#2: all path

We could also recursively parse the graph and authorize duplicate requirements:

Output for the user:

This will allow providing many metrics to the user including 2 major ones:

1) a graph showing the distribution of tests (risk / number of test cases)

This is useful to check the correlation between risk and nb of test cases implemented per test (you do want that the more risky tests are tested the more in details so have the more test cases).

2) more importantly, some "risk-weighted-coverage" statistics

The normal coverage metrics give basic information about how much the requirements are covered but it does consider any requirement/specification with the same “weigth”. the "risk-weighted-coverage" would introduce in the calculation the criticality notion. If a highly critical test is succeeding and a low-criticity test is failing, the risk-weighted-coverage will be much better that the opposite way even though there are only 2 test.

Presentation will be as the standard coverage currently displayed.

Many other metrics might be deliverable to the user (need to think more about it).

Release of XStudio 1.4sp5

XStudio 1.4sp5 has been released today.
This is a maintenane version fixing a couple of bugs releated to JIRA and TRAC connection.

Website: User's Manual being converted to online version

Good news! Manon started working on the online documentation. This will avoid including a large PDF (more than 100 pages and a few MBytes) in the distribution package. But more importantly, this will allow keeping the documentation maintained/updated much easily and often!
The complete User's Manual should be available in 3 weeks.

Release of XStudio 1.3b2

Beta 2 of XStudio 1.3 has been released today !

Release of XStudio 1.3b1

Beta 1 of XStudio 1.3 has been released today !

XStudio 1.3: New test dependencies graph

In XStudio 1.3, a new test dependencies graph will be released. This is using a new hierarchical layout much clearer than the former circular layout:


When some cycles do exist they are indicated in red/orange so that you can fix them. And again the new layout does a tremendous job:

XStudio 1.3: New manual launcher

One comment that many people made about the 2 manual launchers already available was: "It's nice but how can I execute some tests, then stop, close XStudio and continue the same session the day after?".
Another question was: "what if I want to run again one individual test already executed a long time ago in a campaign session?".

In XStudio 1.3, we would like to introduce a new manual launcher and a new feature to answer these 2 questions.

1) A third manual launcher
This launcher will be aimed at giving the more flexibility as possible to the operator so that:

• he can execute the tests in the order he wants
• he can re-run the same tests several times
• it's trivial and fast to run the tests
  

The goal being: "as simple as executing an Excel/Word test plan writing down the result manually with a pen"

2 options:

Option1:
Here is a mock-up of what could be the design of the new launcher:

This solution is pretty elegant and provides a good overview of the current status to the test operator but it's not exactly as simple as running test cases "on paper": The test operator has to:

• select a test in the test tree (A)
• select a test case in the sub tree (B)
• description of the test and test case are updated
• click on Succeeded or Failed buttons
• use the Previous test case and Next test case buttons
• optionally submit a bug or link to an already existing bug, post a comment on the fly

At any time the test operator can select a specific test (A) and test case (B) to execute it (even if already executed). The clock and progress bar are still triggered only if the test has the timeout attribute (no change from the current launchers).

Option2:
The launcher would just display tests and test cases in a table.
Columns would be:

• Id
• Path
• Name
• Priority
• Result
• Comment

The Result column of test case rows would contain a combo-box to manually select the results. This is much simpler but the test operator would have to rely on a printed copy of the test plan. All the results would be updated when the test operator clicks on the Submit button.

Option3:
Maybe something in the middle?

I'm interested in your views about it...



2) An option to re-execute stopped campaign session.
This is a delicate point as cheating must be avoided. Indeed, who never got to the following situation:
A tester runs a very time consuming test campaign (several days) on a product, getting all tests succeeding... except one. Really unfortunate isn't it? Hopefully the day after the tester receives a new version of the SUT and (because he's running out of time) re-runs only the failed test overwriting a failure with a success (hence ignoring all potential risk of regression included in the new version o the SUT) :( It is very common unfortunately and should be authorized only in very specific cases.

So, a new right will be introduced so that the users who are granted with this right are the only ones having the ability to re-run a stopped campaign session.

In a next version, the ability the select a specific version of the test to run will be added. This will be part of the flagging system.

XStudio 1.3: Accelerated startup

In XStudio 1.3, a new innovative system is implemented to pre-fetch icons and compute (using the Java 2D API) derived icons (based on the following algorithms: concatenation, HSB filter, opacity, grey-out, overlay, rescale etc.) much faster (3 times faster than downloading them individually).

This will drastically increase start-up time (especially in Java Web Start mode).

XStudio 1.3: Proxy launcher

One big issue in testing distributed environment is how to publish scripts execution on each component.

Let's imagine you have to test a VoIP framework that comprises 3 different machines: a Client, a Gateway and a Server. To test the data flow, you will need to deploy for each test (at least) 3 scripts on each of those and run them at the same time. And that's were Test Management systems (even very expensive ones!) generally fails.

XStudio 1.3 includes a new launcher called the proxy launcher and a new kind of agent called the XSubAgent. An XSubAgent is basically a kind of XAgent instance embedding an XML-RPC server and which communicates only through this media.

Here is the flow:

• You create your test campaign from XStudio and start it
• The tests in this campaign are associated to the proxy launcher which can be running on any host
  
• The proxy launcher is configured with a list of individual XSubAgents, launchers and configurations
• The proxy launcher will just contact individually all the XSubAgents and will forward the requests to execute extracted sub-tests
  
• Each XSubAgent will execute its specific script and will return results to the proxy
• The proxy consolidates results and logs and returns it to XStudio.

Hence, one single test is split by the proxy launcher in different scripts that are executed simultaneously on different XSubAgents located on the Client, the Gateway and the Server of our VoIP Framework.

XStudio 1.3: Tests and test cases versioning

One feature requested for long was the test and test cases versioning. Indeed, it was possible to track the progression and regression of test campaigns in time but there were not yet any way to manage the versions of the tests/test cases in parallel. Hence, if you are looking at an old campaign session and see that the test A did fail, you were not able to know what was the test A at this time since it may have changed afterward.

Starting in XStudio 1.3, all changes in tests and test cases are tracked and versioned so that you can get their states at any point in time.

Information versioned are:

Tests:

• Description
• Priority
• Canonical path
• Prerequisites
• Additional information

Test cases:

• Procedure tree
• Description
• Use description as testplan flag
• Additional information

Currently, the system associate to each revision a date of submission and an revision (an automatically incremented number). Here is how the GUI looks like:


In the future, a flagging system will be added so that the user can flag ALL test at a T time with a personalized label. It will also be possible to run a campaign session with the tests as they were defined in the past.

XStudio 1.3: TRAC integrated

In XStudio 1.3, a new third-party bug-tracking database is integrated: the very popular TRAC.

This integration has been done using TRAC's XML-RPC API.

Many other bug-tracking system provide XML-RPC interface (i.e. JIRA) so this will ease very much of those in the future as well.