March 12th 2008 - Allied Testing have designed a new Behavioural Testing Framework.

Allied Testing Design Behavioural Testing Framework

Behavioural Testing Framework (BTF) is a set of models and algorithms designed to analyse the quality of models built into other systems: complex market forecasting, decision-making systems and others. In addition to normal QA, systems like these should also go through, comprehensive behavioural testing.

The idea of behavioural testing is to run the system on data which is altered, simulated or loaded from a database according to a set of parameters and system settings that form a "scenario". BTF provides the means to manage different scenarios, automating the process of running the system in these scenarios and analysing the results. As the number of runs (or ‘experiments’) – grows, the complexity of processing and interpreting the results increases and the Experiment Explorer (EE) tool goes into action. The experiments support storing information about the system being tested, which allows several versions or instances of the system to be compared. This simplifies model regression and performance testing.

EE has been developed by Allied Testing and the concept originates from the need to have an easy-to-use and robust instrument to analyse the results of any kind of experiment: scientific, behavioural testing QA test suites, EE provides users with algorithms and visual tools that reduce significantly the complexity inherent in handling large amounts of experiment data, allowing users to browse through experiment sets and visual representations or comparisons of experiment results and, in the end, making it possible to perform a qualitative analysis of the system. EE is as much an ‘experiment browsing’ tool as a tool for displaying comparative results. Its key feature is its ability to select a comparable subset of experiments, in a visual and intuitive way. EE is essentially a client application that allows the building of distributed behavioural testing environments that are free from OS, platform and geographical limitations, thus allowing experiments to be accessible from anywhere. It makes it possible to run all the calculations run on Linux-based servers in one office and to analyse the results on Windows-based desktops in another office. Moreover EE exploits a unified data storage interface which allows the use of almost any DMBS, such as MySQL server, PostgreSQL, MS SQL.

BTF is a distributed back-testing environment that comprises an experiment result generator (a wrapper around the system under test), a database for storing test results and EE for browsing and comparing test results.

Being a general framework BTF usually doesn't require customisation for any particular system. The current version of BTF is applicable to systems designed in Matlab/MySQL and this makes sense because Matlab is a widely accepted modelling language and MySQL is a popular open source DBMS. However, the system can be redesigned in other languages and can be integrated with other DBMSs.

“Behavioural testing is necessary because it is the only methodology that can track changes in model logic. For simple systems, ordinary QA suffices but, for complex systems with non-trivial internal logic, just writing unit tests is not enough” Comments Peter Gagarinov, Head of Research at Allied Testing. “With regard to trading systems it allows system performance to be tracked and the points where system logic breaks to be identified. Also, it simplifies the parameterisation of trading strategies with numerous parameters or settings which is difficult to control for a human being. Finally it simplifies the process of improving system performance.”

To date, BTF has been used for behavioural testing of an electricity portfolio management system, which Allied Testing developed for one of its US clients. The system was used to compose an optimal portfolio of futures on energy and fuel commodities, in medium term time horizon, by means of a complex directional forecasting system and by tracking the performance of the portfolios under a given trader’s management. 

• Copyright © Algorithmic Media Ltd 2009

• Popular Searches
• Privacy Policy
• Sitemap
• Designed by Vibrant Web Design Brighton
• Hosted by Enhancion