SO

Share Optimiser

See News at the bottom of this page.

This project aims to create a sharemarket (or stockmarket) optimisation software system which:

• aims to forecast not only index and share prices, but also the uncertainty associated with those predictions;
• automatically suggests changes to an existing portfolio as new information arrives;
• maximises returns based on a specified risk (of portfolio loss);
• acknowledges that brokerage costs will limit the optimal portfolio change rate.

Scope Elements

The important elements (and associated techniques) are:

• time-series modelling/prediction/forecasting (via the following candidate techniques)
  • linear time-series models with variable interval
  • non-linear time-series models using feedforward neural networks
  • (Extended) Kalman Filtering
• integration of stock indicators (P/E etc.) with time-series to improve prediction
  • (long-term feature, since downloadable data sources for indicators seem scarce)
• uncertainty estimation and aggregation
• Principal Component Analysis (PCA) for reducing dimensions of index set
• constrained optimisation for maximising returns at a risk constraint

Software Tools

The tools are based around existing open source programming and mathematical software.

• python - as the basic programming language

• SciPy - for the mathematical algorithms (including numpy, matplotlib) - also see the SciPy page on the Process Control part of this wiki

• NeuroLib - a fast C implementation of feedforward Neural Networks written by PeterMills

• Leo - A Literate Programming and Outliner tool written in Python
  • quite a reasonable IDE
  • can serve as development and deployment user interface (buttons and commands structure as well as displaying text information)
  • can hold lots of configuration information and output reports
  • highly scriptable with internal python code
  • very configurable

• [no longer maintained]iTrade - a stockmarket portfolio management, technical analysis and plotting tool being developed by GillesDumortier

  • provides basic infrastructure, conventional technical analysis, and GUI for SO system
  • also uses python, numpy and matplotlib

  • needs plugin modifications for Australian Stock Exchange data (asx.zip) - done

Status

Done

• create the basic classes (Model, Invest, Market, Portfolio)
• integrate multiple time-series model types for prediction (linear, NN)
• integrate the Neural Network library

In progress

• tidy duplication of class code
• store both model normalisations and coefficients/weights data together
• properly split available data into train and test sets to enable realistic performance testing
  • use a date ('2010-06-23') for the split criteria to allow for historical simulation, rather than index)

• allow lag stepsize >1 for fast end of log lags

• resolve how to predict relative change from a noisy "present value" of short duration for log lags
  • this is only a problem for chosen "delta" formulation, non-delta form can't represent the different scales of multiple stock.

To Do

• re-test prediction performance (including simulation over historical data)
• refine model setup (linear/log, 5day/25day, etc.)
• set up long-term history files (from existing data) and automatic updating/merging system
• begin multivariable-modelling for a wide ranging portfolio of indices (as a back-bone for forecasting individual share prices)
• set up scanning for all stock codes to find candidate stocks with best return/risk measures vs worst return/risk measures in existing portfolio

• use optimisation to calculate an "optimal" change to the portfolio (accounting for broking costs etc)

Ideas

• Design of iTrade predictive indicator
  • Output in the form of predicted (1 week) price change divided by calculated prediction uncertainty (i.e. like return/risk). This automatically normalises for price etc and helps confine the indicator to a reasonable range of values.
  • NN predictor:
    • a single trained network for prediction of all stocks. Although not tailored to the nuances of an individual stocks, this has the advantage that much larger data-sets can be used for training, making the network more generally applicable and less biased due to the current resources boom.
    • trained on most significant stocks in the market (e.g. top 200 capitalised companies, or total volume from entire stock history (to preferentially capture long historied stocks)). The stocks used for training may need to be from the target market if the markets are significantly different in stock type (e.g. NASDAQ). For a user, the best data to use for training would be from:
      • specific sector stocks for which the preditions will be used (e.g. NASDAQ, or Resource stocks only). This requires the preparation of a selection of representative stock price history files for training, by the user.
      • representative stocks from a specific market, e.g. Australian ASX stock-exchange. This requires the preparation of a selection of representative stock price history files for training, by the user.
      • ASX general stock history. Although based stocks only from the ASX, it should be reasonably representative of world-wide stockmarket dynamics. The advantage of this option is that the user can just use a network prediction model which is distributed by the author.
    • incorporate: high, low, close, volume, day-of-week, day-of-year
    • Use an Non-linear AR model with increasing time-step interval for older AR terms. The terms represent average price/vol behaviour over the interval, rather than a sample at the time-step instant.
    • If NN package can handle it, use clustered inputs for each interval (high, low, close, vol).
    • Need to normalise the history data to allow multiple company price histories to be combined into a single training set. Normalise by taking logs, differencing, divide by 2 standard deviations (calculated from zero?).
    • Consider normalising high and low (and open) to be relative to (closing) price value. How to aggregate the high and low values in larger intervals of AR model? But if the price time-series is differenced, then "price" will be represented by a rate-of-change type of value which won't be comparable to the high and low values. So high and low values should be an average relative to the price (normalised by log-price), even though price will not appear in the NN inputs (only average differenced price).
    • Prediction uncertainty output is trained using a second stage of training which evaluates the absolute value of the prediction error for each pattern. Then this error is used (normalisation similar to price diff) as the training target for the second output. This error is used to normalise the prediction value to form the indicator.
    • Also consider multiple prediction outputs for different intervals, e.g. 1 day, 1 week, 1 month (each with their own error).
• Use iTrade's "Investment Period" and "Risk Level" settings for directing SO portfolio optimisation.

  • GD: starting with the r619 trunk, these settings are for a whole portfolio and can be edited on the Portfolio properties page. See term() and risk() methods in the Portfolio object.

• Add a new Tab in "Money Management" to calculate and display risk/return.
• Use -i flag on iTrade to do a command-line, scheduled history-update for a SO candidate "portfolio". Then use this history to generate models, predictions and risks for each stock; then optimise portfolio change recommendation. All scheduled to be calculated at close of market, by command-line, ready for after-hours review.
• A chart to illustrate chart of predicted Risk vs Return for stocks in a (proposed) portfolio:

• Test using wiki page as the ShareOptimiser/Transactions input (is this easy for others to use?)

References

• http://groups.google.com/group/misc.invest.financial-plan/msg/3b9d13f3d399050c?dmode=source

• http://www.departments.bucknell.edu/management/apfa/Dundee%20Papers/27Schwartz.pdf - how to optimise a portfolio that beats the S&P500.

• http://www.caliopywork.org/main/page_screen_shot.html - an engineering application designed with the Leo outliner as the user interface (a starting point for interface ideas)

Other software sources

• ffnet for NNs (uses numpy, but also needs compiled fortran, http://ffnet.sourceforge.net )

• bpnn.py for NNs (pure-python (portable), moderately fast but doesn't use numpy arrays and isn't vectorised

• check http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/ann/ for developments in SciPy

• pyKF for Kalman filter

• http://www.radlogic.com/releases/radwikiedit.py can be used to update or add a new page to a moinmoin wiki - may be useful as an output mechanism

Weblog

---

CategoryActive