All investment banks are different in terms of attitude, culture, technologies, programme management and approach to quality assurance; as such the recommendations discussed here need to be viewed alongside these other considerations.
The current environment
Following the 2008 liquidity and credit crisis, investment banking regulators across the world began a process of introducing unprecedented restrictions on existing capital adequacy requirements and associated liquidity risk management practices.
Different regulators and regulations apply to operations in different parts of the world. Most major investment banks have international operations operating in accord with 24x7 markets. Changes to the regulatory framework are in conflict with the unprecedented speed at which financial institutions are seeking to maximise their profits, cut the cost of their business operations and embrace the opportunities presented by evolving technology.
These early successes made a material change to flow business by scaling the execution process to reach new customers, demanding internal straight through processing (STP) to deal with the increase in flow. A huge appetite was created at banks for increased electronic distribution and the tail-end of the first Dot.com boom witnessed the creation of a number of multi dealer platforms (MDPs) in both Foreign Exchange (FX) and Fixed Income (FI) to serve this. New venues were established that presented multiple dealer inventories for price discovery, where customers could use request for quotations (RFQs) to put dealers into competition on the order.
From the board level and trading floors, through to the compliance and audit departments, the implications of these new regulations must be addressed. The fast-paced and demanding trading room environment requires quick fixes be made to circumvent the limitations of the existing technology, whilst ensuring minimal client impact. In contrast, down-stream business functions require more structured and thorough approaches to ensure that the banks do not become exposed to malpractice and unwanted attention from the authorities.
Interpreting the regulations and mandates
Certain Basel III requirements, such as the Expected Positive Exposure mandate will dictate software or system changes. The implications of the software changes upon applications will need to be investigated and applied across all the impacted systems. The nature of the new regulatory requirements is such that they require a complete understanding of both the technology infrastructure and internal trade processing capabilities to ensure that the changes are fully understood. The key to a successful implementation of this nature is to deploy specialist project and testing teams that can demonstrate extensive domain knowledge that complements the internal system capabilities of the bank.
One key issue faced by investment bankers is the apparent conflict between regulations as yet unresolved by case law. For example, the Data Protection Act 1998, the Freedom of Information Act 2000, and the Data Retention (EC Directive) Regulations 2009, each have different requirements on data retention. Similarly Sarbanes-Oxley imposes different retention rules for data used in the US.
System architects and technology teams may not be aware of such regulatory requirements; their focus is more likely to be on testing and optimising the system performance of real-time trading systems. High latency levels of an application at one institution may cause an open trade offer to be taken by a competitor with a better designed and faster system. However, testers of such applications will also need to take decisions on compliance, and make provisional and conditional assessments of fitness for purpose, often pending the advice from internal compliance and audit departments who may be unable to provide such advice in the required timescale.
Achieving testing objectives in a rapid delivery development
It is an accepted fact that the majority of systemic changes within today's marketplace are both complex and challenging, and that meeting the demands of the various business areas places unwelcome pressure on both project and testing teams. The nature of the trading room environment tends to attract high profile individuals that are demanding of themselves, their colleagues and the supporting areas of the financial organisation. Of all the challenges that are experienced today, one of the most common is related in some form to the variety of the different personalities and associated characteristics of the people that are employed by the banks.
The majority of change programmes originate from one or maybe two business departments rather than an entire financial organisation. The trend is that the originating department is likely to benefit from the change itself, and therefore has a direct interest in pressing for the Time to Market (implementation timescales) to be as short as possible, to ensure that the cost of change is kept to a minimum. Other departments tend to have an indirect interest in the change programme, but are likely to view additional project work as a burden and an unnecessary overhead in terms of both resource utilisation and workload.
Whilst traders and front-office staff expect rapid delivery, the reality is that the development and testing of projects requires detailed planning and scheduling of resources. In achieving this, it is necessary to engage business managers with sufficient knowledge to understand the importance of testing, and with sufficient authority to ensure that it is correctly undertaken.
Such engagement means that testing should be at the forefront of any change programme to ensure that the requirements are clear and concise, that impacted business areas are identified and engaged at the project inception, and ultimately to ensure that testing is part of the complete development lifecycle.
As most investment banking programme managers favour an agile methodology to software development, this should mean that continuous testing is performed throughout. Experience shows however, that true test driven development is not fully understood, and that testing often takes place towards the end of a project, when deadlines are fast approaching and timescales are cut short. As a result, insufficient time is made available for a comprehensive phase of testing, and as testers find defects in the software, subsequent delays are attributed to the testing teams rather than the overall project management.
Front office fluidity versus regulatory constraints
Traders and sales personnel do not typically view testing as part of their day-to-day responsibilities and will often require coaching or at least guidance on testing best practice. Operations personnel tend to take a far more pragmatic approach that embraces the need for a comprehensive testing plan. These different views can result in a conflict of effort and timescales depending on the importance placed by the individual on the requirements for testing, from the minimal an extended approach throughout the lifecycle of a project.
Whilst the majority of business areas that support the trading systems prefer a risk mitigation policy, the requirement for rapid system change to support business demand, can result in conflicts that potentially have damaging implications to both the project and the testing teams. An example scenario is where a new trade type is implemented as an exceptional item, bypassing most internal audit procedures and without proper notification to ancillary back office accounting and transparency requirements. It will then need to be integrated again as formal due diligence processes sanction the changes.
In this scenario, the solution is to deploy a strong test manager with both technical knowledge and a good understanding of testing best practice. However the most important quality in these circumstances is for the test manager to possess strong stakeholder management techniques. It is always important to engage all stakeholders at project inception, rather those that are directly impacted by the system or software change.
Technology constraints and considerations
In recent years, the demand for technological innovation has increased, particularly with the advent of rich internet applications (RIAs) that dramatically improve usability and access to bespoke user-data. As a result, technology change is no longer the preserve of the technology experts, with increasingly technically astute end-users now demanding more from the applications which populate their order books and provide them with market data.
The user experience encountered elsewhere, such as on a tablet device, a smartphone, or a social networking site, plus the associated applications are now setting the bar for user interaction. The testing implications of this are immense, with new RIAs often providing multi-layered complexity that requires thorough testing, such as trading websites that employ Adobe Flex technology to provide complex solutions for both trading and information distribution.
The implications of utilising these emerging technologies means that testing needs to remain one step ahead and be aligned with rapid development practices, in order to accommodate consumer driven technologies. As a result, the following testing challenges need to be considered:
- Not to be distracted by the 'rich internet' of a RIA; the front end is important, but so is the way in which these new applications hook into legacy systems
- Ensuring the new approaches are technically robust enough to cope with:
- Security challenges
- Changes to the back-end
- Architecture changes
- Requirements to test handheld devices (and the associated security implications)
The development of tools to automate the functional testing of new technology lags behind the speed of change of the technology itself. This is attributable to the conflict between the new technology taking market share, and the longer-term development of tools that evolve to thoroughly test its functionally, especially in an enterprise environment. This conflict is evidenced within investment banking, whereby the development and use of automated testing tools for Flex from Adobe falls far behind the take-up and use of Flex itself. Whilst potentially opening new opportunities for the developers of automated testing tools for RIAs based on Flex, their adoption within investment banks is slow.
Much has been written about the migration to 'cloud computing' and its suitability for testing. Such technology is rarely used within investment banking, particularly as a result of strict regulation regarding the confidentiality of banking information. Whilst some banks are adopting a 'private cloud' approach to some information and services, this emerging technology is still in its infancy. Test analysts must be aware of the sensitivities of bankers to the risk of data loss, and it is a credit to investment bankers that sensitive information appears more likely to be lost by public services than by banks. However, in circumstances where restrictions are not imposed, individual traders are able to use cloud services without sanction; thereby risking the breach of data protection and other regulations.
Testing across multiple platforms and technologies does not lend itself to a traditional end-to-end test strategy, as quite often new technologies tend to conflict with legacy systems. For example, where new 'publish and subscribe message technologies' are employed using specific 'peer-to-peer authentication', it may be very difficult to identify which services consume internally broadcast messages or transactions.
The considered solution in this situation is to assume a three phased approach:
1) Integrate
Systems integration testing is a complex task in many investment banks, where transactions and data are shared across many disparate systems. The initial phase of testing is to ensure that information flows correctly in each direction of the test environment. Normal methods would include an initial 'smoke test', which can be executed without directly affecting any of the data stores. These tests are often automated, with a series of parameters allowing the smoke tests to be conducted from the same test framework as the later functional testing. Such smoke tests can be employed as a quality check to ensure that the correct instances of systems and applications are communicating properly with one another.
The goal is to be assured that neighbouring systems are actually able to communicate correctly. Whilst being no different to standard integration testing in other domains, there are very particular demands made in investment banking to ensure that all the affected systems form part of the tests, and that test execution always produces a permanent trail to show where data is modified as part of the testing.
2) Differentiate
The test approach should be built or chosen for its ability to demonstrate differences in method. Test automation tools employed in the approach must differentiate between the key layers and technologies. Some will address transactions at a message level, testing key functionality at the transport layer. Others will address data storage, speed of access and the ability to trace specific transactions from capture to their effect on the bank's balance sheet, profit and loss accounts, risk, plus applications alerting those responsible for disclosure to the appearance of unusual customer transactions. From the outset, the tools and testing methods must be designed to accommodate the various needs of different (and sometimes opposing) parts of the business operation, providing complete coverage. In addition, the tools must differentiate between the specific features of the different technologies that have been used.
3) Mitigate
In order to mitigate associated risk, there are numerous avenues for risk-based testing. When considering the investment banking domain, we tend to consider the implications of defects from a number of risk mitigation aspects, including; financial risk, operational risk and reputational risk. With the very large financial value often involved in trades, risk mitigation can take a number of forms in business operations, where speed of response is a key differentiator of margin. Regulations have slowly adapted to require more and more liquidity to cover open positions, measured by the risk of counterparty failure. These are all factors to be mitigated in business, which must be mirrored in testing.
Changes to business processes and procedures
In addition to software and system changes, the introduction of new regulation under Basel III will demand a number of enhancements to business processes and procedures, including most notably the mechanics that the banks will need to adopt in order to conform to the changes in liquidity ratio regulations. This will entail the monitoring of the bank's exposures to different counterparties and clients.
The consequences of this type of change are more in line with how investment banks embrace change. Whereas system modifications ultimately enforce a conscious change, amendments to procedures or regulation require indirect changes that the bank will have to monitor internally.
Conclusion
A natural conclusion to the majority of the challenges discussed in this paper may be to adopt a heavy-handed management approach to ensure that mandatory changes are enforced within investment banks. Whilst a number of financial organisations are likely to take such an approach, this does not tend to embrace best practice, or the requirement to integrate testing efforts across the entire organisation.
The recent experience of City Index in being fined £490,000 (GBP) for failing to keep adequate records of transactions demonstrates that, whilst their technology was fit for purpose at a trading level, the associated back-end data retention fell short of the required standards. This was a clear failure of both IT and the business in conforming to explicit regulations.
Testing should be viewed as the critical measure of quality and project success in any IT project. This is more likely to be successfully achieved by ensuring that from the outset, the testing function is seen as the pivotal project component that brings IT and the business together.
At Rule Financial we understand the importance of utilising testing best practice, and continue to help our investment banking clients update their testing business practices, enabling them to benefit from efficiencies in project implementation and the underlying legislation.
_________________________________________________________________________________________To To receive this article as a pdf, simply enter your email address in the 'Send as PDF' box at the top of this page.
