Some kind of a road map.
My goal is to consolidate financial data from varying sources and find a way to deduct hopefully wise trading decisions. I obviously will need data, store them somewhere. As I am the graphical type of guy, generate some charts.
The data:
about 1 million financal instruments trade resulting in some 300 million rows of data per day. Uncompressed 18 giga bytes a day. http://asus-p4.lan/lvlamb.com/project/market.html
Woops! There is some selection to be made as this amount of data needs some T1 lines.
So, either I'll have to take a drill down approach, for instance follow some widely available indexes or subsets of instruments, or I would need to concentrate on specific intruments or sectors and forget about the rest.
This is a first trading decision and I have no idea on how to start.
Real time data vs. delayed data.
Some exchanges even prohibit diffusion of delayed data.
As a rule of thumb, take $30 per family of instruments you want to follow (currencies, grains, metals, energy,...). A correct budget in exchange fees alone can amont $2000 a month in no time, double it for the service fee you would pay to a data provider.
Yeesh! Gotta make a second trading decision here, really forget some, limit real time data to a couple of instruments, gather "free" delayed data where I can, consolidate cheapo real time with delayed....
Many questions to be answered for that second decision.
And, do I really need real time?
Yes. Real time is in my opinion the only way to observe a market: the prices react in sync with CNN, CBS or CNBC and can lead to interpretations on the health of a specific market.
Can I trade based on delayed data?
Yes. As, my opinion only, I believe that it is not difficult to statistically weight some techical analysis signals as since fast internet and online data offering, an increasing number of private traders will all react the same time on the same signal. Hence, short term, you would take an opposite position to the crowd, joining the deep pockets guys.
To the contrary, there are some indicators one could follow as the deep pockets all use the same basic algorythms and will react in concert. So, the trick would be to find indicators which are not acted upon by the short term arbitrageurs, neither traded upon by deep pockets position traders. Say that I can miss the 30% moves of the arbitrageurs, miss the 30% move of the deep pockets position traders, try to keep the 40% between.
So, I would have 3 sorts of indicators to rank: the short term ones (mostly generated by real time data), the long ones (deep pockets moving the market to trap the crowd), in the remaining those which could work.
Let's face it, equities markets are a market for paper: more and more funds and mutual funds buy an index of shares and the supply is limited. And there are funds of funds. Most of the market for shares currently is bidding up a limited amount of paper (which trend to price higher), arbitrageurs and deep pockets do not care about the fundamentals but can to a certain point manage to make profits by controlling te price movements. Up to a certain point as, with all man-made sytems, something unexpected can happen. A disrupting event can trigger a cascade of trading programs before a human would interfere.
This means that we are at the mercy of harsh corrections we cannot predict, even with sophisticaded trading signals, and could lose all our positions. Here again, real time data are a necessity.
So, there is a middle point somewhere: some sort of real time data is needed. For the remaining, a good database of end of day data, eventually completed with more or less delayed current data should suffice.
One could subscribe for about $12 a month for real time US indexes data. These data (streaming) usually will not be easily grabbed by an alien sytem to the available application. But some index sites do have real time data which can be grabbed from a webpage. Hence, not too difficult to get (not quite real time but) snapshots.
As there is a choice to be made of the universe of instruments we want to follow, and a diversity of sources, I feel the need of a database system versatile enough to handle a variety of sources.
What I want: a couple of T1 lines with a Reuters or a Bloomberg terminal ranging $4000 a month.
All the other systems are a subset.
Hence, I can spend a good deal of $4000 a month in time writing my own system, or expense what I really need, can't handle myself or find cheaper than my own time.
And in any case make cross-referencing of data possible, which cannot even on a $4000/mo. system: there always will be data you gather on your own. As an own index for instance: the only way to beat a widely diffused index is to design your own, better performing, index.