How the failover service works
From Triage Documentation
How does failover service help?
The RainMaker failover service will switch your trading to a new server if your primary trading server encounters unrecoverable problems. Within minutes, an orderly process of recovery of your trading on a new server occurs without any human intervention. The entire failover process is automatic and keeps your servers trading continuously through all manners of technical difficulties in your trading environment or connection.
Supported trading platforms
Currently, RainMaker failover service is available for TradeStation, MultiCharts and NinjaTrader.
We will soon be adding support for MetaTrader, eSignal, Interactive Brokers, AmiBroker, SmartQuant, NeoTicker, RightEdge, MarketCetera, Worden, OEC Trader and Sierra Chart. For more information, contact us at: http://triagetrading.com/about-us/contact-us
Monitoring your trading computers
RainMaker monitors your trading servers to make sure your operating system, trading platform processes, data/trade server connections and internet connection are running properly. The RainMaker monitoring panel in your RainMaker web interface shows your trading server and platform health.
You choose which monitored information is watched by the failover service. Once there is a failure alert triggered, RainMaker starts a countdown to switching your trading from your Master trading server to your backup failover server.
Once reaching the countdown timeout and your backup failover server is ready, RainMaker takes your Master trading server offline so it doesn't interfere with the backup failover server taking over live trading.
What is a master server?
A master server is your original trading server on which you run your trading. All backups and copies are created from this original server, no matter if the server is a cloud computer or off-cloud computer. An off-cloud computer is any computer that is not a virtual cloud computer at a cloud service provider, such as Amazon AWS.
What is a failover server?
A failover server is a backup copy of your master trading server. It is created at the time you start the master server and failover service. This failover server "copy" will be used to take over trading if your master server should fail.
Failure warnings and alerts
RainMaker watches the operating system, trading platform, processes, trade/data server connections and internet connection. Any one of these can trigger a failure warning or alert.
Whenever there is a problem, it is updated at the RainMaker web interface, and a message is sent to you via email, sms or phone.
And if an amount of time passes (called the failover time-out) without returning back to a normal operating state, a switch to a failover backup server starts.
Self-recovery of trading server
As soon as the trading server encounters problems, a RainMaker agent running on your master server will try to recover the operating system, trading platform or processes. Only when it cannot do this within the failover time-out period does the RainMaker central control starts your failover backup server and switches trading over to it.
Time-out since failure alert
The switch of trading to your failover server only occurs after some period of time after a trading failure alert is received at Triage central monitoring. This is so your trading platform and server has time to recover on its own, and prevent false positive readings from triggering a failover process.
The failover time-out is usually set at 1 to 2 minutes.
After a failover server is started, RainMaker automatically recovers the trading platform and sets it to live trading. However, there are times when things continue to go wrong. For these cases, we must limit the number of failovers the system tries.
You will specify the maximum number of times you want failover to occur from the original master server. This is to prevent the failover process from executing an endless loop of failover attempts. This could be costly if unattended. This is default set to 3 failovers.
You will also specify the time between failovers. Because once a failover occurs to a new server, it may take a period of time before all systems are restored. You don't want to trigger yet another failover before your trading systems have a chance to fully recover. This is default set to 5 minutes.
These settings will make sure that you have failovers occur only when you are sure the failure alert is accurate, and to prevent too many failovers to continue to happen if your live trading should turn out to be completely unrecoverable.
Hot, warm and cold cloud trading servers
You can failover your primary trading computer to hot, warm or cold cloud trading servers.
Hot servers are cloud trading servers that are run as soon as you start your primary trading servers and can be synchronized in real-time. These hot trading servers are like active standby servers ready to take over immediately. They continuously run, and provide for rapid trading application recovery.
Warm servers are cloud trading servers that are started up, but suspended until needed for trading, thus saving you money by not running a live server all the time. The warm servers are synchronized after they are restarted from suspended mode. Resuming the warm servers takes usually under a minute, but is dependent on what software you installed and the configuration your operating system. Warm mode is similar to when you suspend your Windows desktop and resume it again.
Cold servers are cloud trading servers that are started up from a stored server image created at the time you start your primary trading server. They are the least expensive solution, and are not synchronized in real-time. Instead, they can be a clean image of your trading platform prior to your primary trading platform booting up. Boot-up usually takes one to three minutes. However, your wait times may depend on your cloud service provider's capacity to boot up new virtual servers.
Boot or resume cloud servers
When RainMaker decides to failover to your trading from your Master trading server to your failover server, warm and cold servers are resumed or booted. Soon after the operating system and normal application processes are available, RainMaker gets your system ready for the trading platform to takeover.
Auto run and recover trading at trading platforms
After a failover server boots or resumes operation, our RainMaker software kicks off automatic processes that will start your trading platforms. RainMaker then monitors if your trading platform is active and running correctly.
Store dead server and register new master trading server
We never destroy your master server even after it fails. We stop and store the original master trading server that you first booted up for you to review. You may fix the technical issues and reuse this original master trading server.
Get ready new failover server
After your dead master trading server is stored away, and you new trading server is running, we setup a new failover server. This new failover server is automatically configured to run should another failure event occur.
You can set in your failover configuration how many times you want to allow a failover before RainMaker quits trying to failover. For example, after trying three different failover servers you still experience trading failure, you can decide that is enough to put stop trying to failover to a new server.