Test Engine
Although Transaction Simulator is handy while you are developing or demonstrating your product, when you want to verify the wallet’s behavior you normally want to submit many tests in a short time. Test Engine allows you to do that, while monitoring the card’s balance changes and the wallet responses.
Overview
Nautilus allows you to create automated tests, with each test containing multiple test cases, and each test case comprising numerous transactions. The Test Engine executes tests automatically, recording execution results and full wallet communication for detailed analysis if needed. For each executed transaction, the Test Engine ensures that the response code returned by the wallet matches the response code defined in the test case. Additionally, the Test Engine can validate changes in the card’s balance after a transaction or test case, offering thorough validation of the wallet system.
If a test case fails during execution (i.e., returns an unexpected response code or a balance different from the expected balance), the Test Engine stops the execution of that specific test case while continuing to execute all other test cases within the test.
To use the Test Engine, you should first create test cases and tests.
Test cases are created on the transaction details page, where you can save a transaction chain as a test case. For example, a simple test case like 'ATM money withdrawal' involves executing a money withdrawal transaction on an ATM simulator (with settlement included). Afterward, you open the details for the created transactions and click on the 'Save as a test case' button. You can create more complex test cases that include more than two transactions, such as base authorization, incremental authorization, and settlement for the full amount.
Once you have test cases, on the test listing page, you can create a new test and select the test cases you want to include. Once created, the test can be executed. Test execution only requires the ID of the card used for transactions. The system will execute all transactions on the provided card, ensuring that other transaction details remain the same as in the test cases. Note that if the test contains multiple test cases, multiple transaction requests might be sent in parallel to the wallet. However, if the test requires balance verification, the transactions will be executed sequentially to prevent one transaction from interfering with the card’s balance while other transactions are executing.
While executing the test, the system records the results of execution. For each transaction, it records whether the related constraints are satisfied or not. The request sent to the wallet is recorded, along with the wallet's reply. If balance verification is required, a balance inquiry transaction is executed before and after each test transaction or test case, and these results are also recorded. This detailed recording allows you to inspect the wallet's behavior thoroughly. Furthermore, you can export test execution results into a CSV file for processing by other tools if needed.
Validation of wallet’s balance keeping
The most crucial function to test and verify is the wallet's balance-keeping. Incorrect balance management can lead to significant problems for the company. Nautilus verifies this by allowing the test engine to check balance changes in parallel with test execution. For each test created, you can choose whether you want balance verification and to what extent (after every transaction or only after completing the entire test case). After the test case completes execution, you can check if the transactions passed balance verification and inspect balance data in detail.
If you put all these test cases into one big test, you can end up with a single test verifying the complete behavior of the system. After each change to the system, you can execute this test, to be confident that improvements didn’t break something that used to work before.
Regression testing as described is a great way to reduce stress of the team before major releases, and to increase confidence in the product you are building.
Regression tests
The purpose of regression testing is to spot regressions (i.e., something used to work but doesn’t work anymore) in the system’s behavior. Ideally, while you are building the system, for each functionality that you implement, you should immediately create one or more test cases that verify that it works as it should. This way, simply by executing these test cases, you can easily, at any time in the future, verify that the wallet performs as it should.
If you put all these test cases into one big test, you can end up with a single test verifying the complete behavior of the system. After each change to the system, you can execute this test to be confident that improvements didn’t break something that used to work before.
Regression testing as described is a great way to reduce stress of the team before major releases and to increase confidence in the product you are building.
Performance tests
Performance tests are essential for evaluating the system's performance, including aspects like authorization processing speed and system throughput. The Test Engine makes it easy to conduct such tests by efficiently sending a large number of transactions to the wallet in a short time. Nautilus records and presents the time taken by the wallet to execute each transaction within a test, allowing you to identify transactions with extended processing times.
By combining this with the Authorization Processing Time report, which provides a list of the slowest authorizations, you have all the tools needed to pinpoint and address authorizations that require more time to process within the wallet.