This development will be done in 4 phases:
The test environment should be stable to be able to compare the results between differents versions of Sigmah.
It will be composed of 2 virtual machines:
The client and server virtual machines should be 2 differents machines to measure the impact of network usage.
A scenario is a set of functional actions.
A functional action is something the user feel as being an unique action but that may send multiple requests to the server (like opening a project or saving the values of a phase).
The duration of each functional action is summed up to obtain the duration a scenario. The time spent during actions (like the time spent moving the mouse cursor to the next button or the time spent writing a value in text area) is not taken in account.
The robot that will be used to run scenarios is Selenium.
Selenium can mimic a user and follow instructions to perform one or multiple scenarios. It can be used to run hundreds or thousands times the scenarios to calculate the average durations.
To allow the placement of probes by the community, a documentation of the API will be written on this wiki.
Probing will record the duration of each scenario when using Sigmah. To avoid slowing down users in production, a parameter will be added in the administration to enable or disable probing.
Results will be stored client-side in a dedicated IndexedDB. This database will be shared between every users of a same browser. Storing the traces client-side minimize the performance impact of probing. Using a database also eases the exploitation of the traces.
When probing is enabled, a button will be added in Sigmah to send a report containing informations about each scenario.
When an user chooses to send its performance, a report will be generated in Markdown format. Using this format will make publications to wikis easier.
The reports will be anonymous and will contains the following informations:
Clicking on the
send button will create a report and automatically send it to the email address of the user.
Selenium can't be used to measure performances. It can not detect when an ajax call is finished. Using probes to record and store performances allow the recording to include only the duration of the actions.
JMeter is great to run load tests on a simple request but it can't measure the time taken by a set of functional actions. It does not reflect the user experience.
Since probes are activables in production, users of Sigmah will be able to send reports and share their performances.