Strategy Optimization: Avoiding Overfitting and Curve-Fitting

Optimization means adjusting a strategy's parameters (a moving average length, an RSI threshold, a stop-loss distance) to find the combination that performed best on historical data. Used carefully, it's a legitimate way to refine a strategy that has a genuine underlying edge. Used carelessly, it produces curve-fitting: parameters so finely tuned to one specific stretch of historical data that they capture that data's noise, not any real, repeatable market behavior.

A telltale sign of overfitting is a strategy with an unusually large number of parameters, each tuned to a suspiciously specific value (a 37-period moving average, rather than a round 20 or 50), producing an almost perfect historical equity curve. Real market edges tend to be somewhat robust to small parameter changes -- a genuinely good strategy usually still works reasonably well with a 35-period or 40-period average instead of exactly 37. If small changes to a parameter cause performance to collapse, that parameter was likely fit to noise rather than signal.

Out-of-sample testing is the standard defense: splitting historical data into a period used for optimization and a separate, later period never touched during optimization, then checking whether the optimized parameters still perform reasonably on that untouched data. A strategy that performs well in-sample but poorly out-of-sample is a strong sign of curve-fitting, regardless of how impressive the original backtest looked.

The practical rule professional strategy developers follow: prefer fewer parameters, prefer round and theoretically justifiable values over precisely tuned ones, and always validate on out-of-sample data and then forward-tested demo data before trusting an optimized result with real capital.