@WestCoastJustin

I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly.

Can you give some thoughts with your experimentation on the following from a Go perspective.

1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)

2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB

3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)

Why not switch to Mojo lang for this? It's Python-compatible with Golang like performance from what I hear.

@eshnil (& others familiar with Mojo)

I had not looked at Mojo... thanks for the pointer. In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc).

Know if Mojo directly uses existing Python ecosystem?

I've been using Polars, etc.

> Know if Mojo directly uses existing Python ecosystem?

They say they do, including libs with C bindings.

It's not available yet outside of hosted environments provided by Modular.

Haven't looked. Honestly, I'm so invested in this now it would be a pain to change anything.

I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...

Hilarious how you suggest someone use a technology that isn't even publicly available, showing how extremely shallow your knowledge is.

> Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)

I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing.

> Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB

Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly.

> If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)

Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.

I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.

Yeah, I'm heavily testing everything it writes. So, I'm very sure it's correct or get it close enough and then code it. I've seen ChatGPT add imports, functions, etc that don't even exist in Go. What I've found so useful though, is not even coding, but asking it how to solve problems, and then having it code things up. With Google, you need to pretty much already know how to solve the problem then go looking for answers. With ChatGPT you can ask it how it would solve this problem. For algorithms and stuff it have been unreal. Thanks for the warning though. You're totally right.

Not a complete answer but I quite liked https://github.com/markcheno/go-talib for technical indicators.

Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks.

Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system.

Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?

I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anything I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet.

For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time. Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this.

This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.

Fascinating. Have you considered open sourcing this without your algorithms ( empty buy/ sell loops)?

I’d like to play with something like this but I couldn’t be bothered to build it all from scratch.

Yeah, I can do that. Ping me at justin @ sysadmincasts . com and I'll send you a note when I've done it.

Count me in the group of solo algo trading developers (Go and Python and Tradingview Pinescript and other sruff). Planning to send you an email but maybe we can form a group/discord or something online.

sign me up as well! I would def be interested too.

Do count me in as well! Literally in the middle of building something like this myself.

I've also got a few questions regarding the way you manage memory, so I might ask you a couple !

Do you have an email? I've been building a similar system for awhile except in F#, would love to connect.

justin @ sysadmincasts . com. Happy to connect.

You may be interested in this: https://www.techtrader.ai/#wall

Excerpt from his site:

"Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist.

Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "

Thanks, I'll check it out!

No broker API change in 10 years? No bugs?

I call bullshit. Reads more like a scam for trading signals.

I don't run that site myself, but a quick search on sec.gov shows it's a registered hedge fund:

https://www.sec.gov/Archives/edgar/data/1653903/000165390318...

I can't say any more about its validity. Ask your financial planner if techtrader is right for you :)

Is your system consistently profitable enough for you to live off of? How long did it take you to get to this point?

No way. Not even close. That would be the dream though. Taxes are insane in Canada. Like 50% insane. Housing and kids just eat money. Maybe if I was renting and didn't have a family but that's not happening.

Curious if you mind sharing your sharpe/annual return? I understand if you don't want to though!

You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all?

I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?)

Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?

Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing!

[1] https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Fe...

How do you relate with tradingview?

I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO.

I've used wonderbit, zigz, and some of the others.

Cool project.

I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well.

I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in.

I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!