Monitoring a System

January 24, 2010
55 Views
I’ve been using Python with IbPy, rather than Matlab. The code I post will most likely be in Python going forward. I like it because it’s more natural, ex. default and variable arguments are concise, no semicolons, dictionaries; it’s open source so there are more libraries and it’s free (Matlab requires a “license server” for instances running in parallel, for example); and it’s easier to organize code with better package management, simple classes, and files are more flexible.

Anyway, I like to know what’s going on with my system when I’m away. The system writes out a log of errors, trades, finished processes, etc, and also emails me important events like Trader Workstation shutting down, and executed trades. These go to my Blackberry wherever I am. To send emails from/to a Gmail account…

I’ve been using Python with IbPy, rather than Matlab. The code I post will most likely be in Python going forward. I like it because it’s more natural, ex. default and variable arguments are concise, no semicolons, dictionaries; it’s open source so there are more libraries and it’s free (Matlab requires a “license server” for instances running in parallel, for example); and it’s easier to organize code with better package management, simple classes, and files are more flexible.

Anyway, I like to know what’s going on with my system when I’m away. The system writes out a log of errors, trades, finished processes, etc, and also emails me important events like Trader Workstation shutting down, and executed trades. These go to my Blackberry wherever I am. To send emails from/to a Gmail account, which I like to use because it has a lot of storage and can be searched easily, I had to write a little script (code at the bottom) to add to the Python logging library since Gmail uses a special kind of authentication not supported by the standard library.

Here’s how I use it:
import logging
logging.config.fileConfig(“/yourdirectorystructure/logging.conf“)
logging.debug(‘Live trading online’)
logging.critical(strategy_name+’ ‘+signal+’ ‘+position_size+’ ‘+etf)
What that says is: load Python’s logging library (which does most of the work for me); then configure it according to my preferences in the file logging.conf; then as an example send a debug-level message ‘Live trading loop starting’; and then a critical-level message with the strategy name, buy/sell signal, position size, and ETF name. These two messages are taken out of context from one of my strategies, so the variables don’t reference anything here. The first is called when I turn on the system, the second is called whenever a trade is executed.
It looks like this in the inbox:
logging.debug(msg) sends a debug(low)-level message to be logged, in my case it prints to a console window and writes to a file. logging.critical(msg) sends a critical(high)-level message, which goes to the console and file, and also gets sent to my email address.
It used to be frustrating to wake up in the morning and see that my system had crashed at some unknown point in the night for some unknown reason, but now I know what’s going on, when it’s happening, and why.
Python’s logging library isn’t documented very well, especially the email handler and file configuration, so I thought sharing these two parts would be useful. Here’s my configuration file, from above (modify it to set the directory where you want to store logs), and the custom Gmail email handler (modify it for your own username/password). The overall logging system works well and adds the minimum amount of superfluous code.
I’m interested in knowing how others have approached the problem of monitoring and logging.

You may be interested

How SAP Hana is Driving Big Data Startups
Big Data
298 shares3,060 views
Big Data
298 shares3,060 views

How SAP Hana is Driving Big Data Startups

Ryan Kh - July 20, 2017

The first version of SAP Hana was released in 2010, before Hadoop and other big data extraction tools were introduced.…

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion
Data Management
60 views
Data Management
60 views

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion

Manish Bhickta - July 20, 2017

Physical Data destruction techniques are efficient enough to destroy data, but they can never be considered eco-friendly. On the other…

10 Simple Rules for Creating a Good Data Management Plan
Data Management
69 shares664 views
Data Management
69 shares664 views

10 Simple Rules for Creating a Good Data Management Plan

GloriaKopp - July 20, 2017

Part of business planning is arranging how data will be used in the development of a project. This is why…