You have many choices when you select data for your system testing. You should therefore exercise great care in choosing your test data because they have a big influence on test results.
Choose your data vendor with care, since data vendors differ in accuracy, depth of coverage, and reporting conventions. For example, there can be differences in the opening price of a contract between data vendors. Another policy difference is how errors are detected and corrected. You want a vendor with "clean" data all around.
If you are testing futures contracts, use a continuous contract or the actual contracts with rollovers. Unlike a cash market index, futures data are not continuous: contracts are dormant, become active, and expire. Hence, to produce a continuous, albeit "synthetic," data stream, different contracts must be combined in some consistent manner. You have two choices when you combine futures contracts. You can preserve the prices or price levels, at the cost of correctly preserving the amplitude of price movements, or, alternately, you can preserve the amplitude of price movements at the cost of adjusting prices. Continuous contracts that preserve amplitudes are preferred for testing. Each type of continuous contract will give slightly different numerical results.
Data! Handle with Care! 65
A good choice is using actual data with rollovers if the software allows it. Results from continuous contracts come closest to data with rollovers, but they tend to underestimate the number of trades and hence the losses due to slippage and commissions. They also underestimate the difficulty with placing stops on rollover trades. For example, after rollovers, you can get stopped out on a real trade, while the system trading the continuous contract continues to hold its position.
Other than type of data, you must also choose the amount of data. In general, the more data you can use, the better, because you can then test a model over a wide variety of market conditions. There is also a statistical requirement, usually quoted as sufficient data to produce 30 or more trades over the test period. The idea here is simply that the more trades you have in a given sample, the better your estimate of the average properties for the entire universe. Hence, you can use the average properties (such as average trade) to estimate how the system will do in the future.
Although 30 trades may not seem like a large number, you may have difficulty obtaining 30 trades with weekly data using certain trend-following models. Thirty trades is probably too small a number to fully eliminate any effects of the data used for the tests. Hence, more than 100 tests is preferable, if possible, and you can obtain sufficient numbers by combining tests over multiple markets and multiple time frames. Later in the book you will learn how to generate synthetic data to generate a large sample of trades.
The point of this discussion is that different input data will give you different results. Besides, you cannot be certain that your trading will ever achieve those results because you do not actually trade the synthetic data. We now look at the results of testing a dual-moving average crossover system using 5-day and 50-day simple moving averages, an initial money management stop of $1,500, and $100 for slippage and commissions.
Table 3.12 was created by Continuous Contractor™ software from Tech Tools, Inc. to show different types of continuous contracts for the Swiss franc. Over the same period, model performance was calculated using actual contracts with automatic rollover on the twenty-first day of the month prior to expiration. The cautionary tale here is that test results vary widely with the type of data used.
Focus on the number of trades for the same entry and exit rules. There are 111 trades with the actual data, almost 37 percent more than the average number of trades created using continuous contracts. The extra trades result primarily from rolling over existing positions into the next active contract.
66 Foundations of System Design
Table 3.12 Comparison of test results with different types of continuous contracts