🇳🇱
If you arent using a Cooley-tukey Fast fourier Transform to analyze your stock data can you really say you know how to buy/sell stocks? :)
Integrating fourier analysis into my Algorithmic Trading that I've been doing. I really should have done this much sooner.
@freemo Hi there Freemo, if I may make a suggestion, I would recommend trying to incorporate wavelet analysis rather than Fourier analysis for this particular application, as the wavelet transform allows for some temporal specificity on signals that vary over time, while still allowing you to access the frequency data you need. It's incredibly useful, and you may even be able to use a matrix factorization approach with the resultant data to pull out key points of interest and recommend stocks based on the behaviors of other stocks, assuming you have the processing power and access to relevant data. Here is a link to a paper I recently read on the topic regarding this application in brain networks, as well as the Wikipedia page to wavelet transforms, and a video on the application to financial time series analysis. Feel free to ask any questions you may have and I'll try to respond soon :)
https://en.wikipedia.org/wiki/Wavelet_transform
@johnabs I've worked a lot with wavelets and generally speaking I dont disagree that stocks are better suited to wavlets than a fourier transform if we arent talking about specifics. But in truth both types of analysis have their place it just depends on what your doing and how you apply it.
FYI I wrote a wavelet based Machine Learning algorithm over a decade ago you may wish to check out (Genetic Wavelets Algorithm)
So that said, first thing to keep in mind is a fourier transform is also temporal in nature, just in a different way. Generally speaking when you do fourier analysis you use a sliding window with some type of overlap and a window function. In this way you dont have a single FT but rather a FT specific to each moment in time. So it very much becomes temporal.
The way it differs is that wavelet analysis allows you to work with different size "windows" (not the correct term but basically the idea as a wavelet has a limited bandwidth where it is high enough magnitude to have significance) at the same time across different frequencies without needing to compare windows in the way you would with FFT.
So with a FFT approach you can still say things like "this particular frequency was present and dominate in this time range here"
@freemo I'm with you (I've published some work on this as well, particularly in application to NMF), but in this case of specifically looking at (what I'm assuming to be) synchronous time-series data, wouldn't you prefer an approach that allows you to work with continuous data as new information "reaches your detector" so to speak, rather than needing to recompute and compare your FT series? I guess my question would be why do it the FT way instead of the wavelet way when the latter should be more suited to this use case in general. Additionally, wouldn't it be easier to perform auto-correlation of your various data traces using wavelets in comparison to FT, which would make suggestions simpler? The real advantage of FT I would see would be ease of implementation and computational speed, which is definitely a justification I would understand, but I'm just curious precisely what your contention is with the technique.
Also, I searched for your algorithm but couldn't find it on google scholar or your Github, would you mind linking it?
@johnabs Here is a thesis someone did evaluating my algorithm on an earlier version of it (the final version had some significant improvements that came after this paper):
http://korykirk.com/thesis.pdf
As for the FT vs wavelets question there are a few considerations. for starters the FT approach is just a standin for a model i have in my head that I havent flushed out yet, so its unlikely to be the final iteration, but in this form it is still useful. There is also the fact that I have access to a wider range of libraries that handle FFT than I would for wavelets. Moreover, the FT stuff I'm doing right now isnt really as complex as the scenario you suggest.
Basically at the current moment I'm using FFT analysis as little more than a bandpass filter and improved moving average (really a low pass filter). In this way I can apply the FFT split the low-pass and high pass out, and then convert it back to time domain, the low-pass will show the actual trends with any "noise" and fluctuations removed, and the high pass are effectively just the noise isolated. The noise essentially represents the volatility and will integrate to approximately 0, so represents short term signals that should average out over time.
From there I know if im looking to trade on time scales on the order of a month then I set my cutoff appropriately and look at what sort of volatility is to be expected and therefore know that as long as the fluctiations are within the norm for noise there is no need to close my position. In other words I can set a trailing stop limit where the amount of the trail is equal to the maximum amplitude of the high-pass filter. That way i can be relatively certain that if a stock deviates by more than the peak volatility that it represents a mis-trend rather than normal noise that will recover and thus is an optimal set point for a trailing stop.
Obviously wavelets could do this in a more dynamic way (responding to suffen changes in volatility). Just not needed yet.
@freemo Thank you so much for the information and detailed breakdown of your problem! I was initially under the impression you were looking potentially variable time-scale trading (e.g. adjusting your trading frequency based on predicted trends and volatility) rather than the more static, though significantly more safe and stable, approach you're currently taking. I think it's really interesting, and I'd love to hear more about your project as it develops more; that is, if you don't mind keeping me in the loop! :D
@johnabs the core algorithm has already been wildly successful for me and been using it for months, but it lacks the elements we discussed here. Sadly the main algo is also top secret since it is just too valuable to let it get out in the wild right now. These are all enhancements im working on and i can at least discuss some aspects outside fo the core algorithm.
Right now im mostly just porting my algo to python for better automation as until now ive been using it to signal my trades but I make them manually, and already making far more money than anyone should be able to off the stock market.
@freemo Oh okay, I understand. My research is in a similar boat of secret, since I'd rather not be scooped XD Well if you don't mind me asking, what trader are you currently using? I do have a pre-existing API handler for my bank to execute automatic trades, but it was written in Julia rather than python, and it's for Ally bank; so if you're not using it, my code would be useless to you xD
@johnabs if its just a stubbed out API probably not much value there that I can extract just yet, but still feel free to share the link and I dont mind taking a quick look and see if anything stands out as worth looking at closer
@freemo Like I said, just a handler, nothing fancy, just a few functions to query and post various lists of stocks of interests, get prices for options, get time-series data for stocks of interest with a default resolution of 5-minutes, current balances, execute basic trades, which can be wrapped in more complexity as time goes on. I'll have to upload the code to github and make the repo public, but I can do that super quick if any of that sounds interesting :)
@johnabs it sounds like its mostly an API to talk to your broker without much logic beyond that. In which case it likely wouldnt be too helpful for me.
@freemo I figured as much xD But if I get any other work behind it (unlikely at the moment), I'll send what I find your way. I'm particularly interested in the NLP side of things, especially since my bank provides headlines based on ticker that can be scraped using the API, but it will be a while until I get anything else done on this, unfortunately.
@johnabs We could always join forces on such a project, though it would be difficult to figure out a way to work together and still amange the trust aspects around my algo. So not sure it would be workable. But would be nice to work along side someone with a good base knowledge to compliment my own. I have a friend who i am pretty close with and trust who is helping with the project a bit so i can focus more on the strategy/algo part. He is valuable and really the only one I could have invited onto the project as I'd trust him enough. But otherwise it would be great to bring in someone who has some decent algo and math backgroudn to work with.
@freemo I'm flattered by the offer and would love to collaborate (though it may be in sparse bursts rather than continuous output due to preexisting commitments). If you'd like to discuss the details regarding this, I'll send you a PM with my email as well as my PGP public key if you'd prefer to keep things encrypted. I can preface this with I am a very trustworthy and ethical individual, considering I had to abandon my first PhD due to unethical practices by the two advisors I tried working under. Fortunately, I've found another advisor in a different department who has taken me in and actually values me, my well-being, and doing things the right way: to advance science and human understanding, rather than his own ego/pocketbook. Unfortunately, I had to throw out my candidacy in the process of switching departments, and effectively start over, but I think my job prospects in this department will be much better anyways.
@johnabs We can discuss preliminaries in any unsecured channel that is convenient. I wont be giving away anything secret early on anyway and we can navigate the waters from there. so for now email or here or anywhere else works to keep the conversation going. With the friend i am working with we usually just use facebook voice chat but even in his case we havent discussed the algorithm itself in too much detail yet.
@freemo At the moment it's literally JUST a barebones API handler to exchange data between some (reinforcement, for example) algorithm X and my bank's trading API. With that said, if you're interested in it, I can link it xD