Generating Multi-Plot Real-Time Plots with Python

In an earlier post the real-time plotting capabilities were demonstrated, we're extending on this by showing how to generate multiple plots simultaneously.  A couple noteworthy observations, in the past post the X and Y scaling was automatically scaled after each element addition.  While you can still do this, typically for multiplots we would prefer maintaining a shared X range.  While somewhat unnecessary, I've elected to maintain a uniform Y range.

#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test02():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(311);
  ax2=fig.add_subplot(312);
  ax3=fig.add_subplot(313);
  l1,=ax1.plot(100,100,'r-');
  l2,=ax2.plot(100,100,'r-');
  l3,=ax3.plot(100,100,'r-');
  time.sleep(3);

  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i),cos(i),cos(i*2)));
    T1=[x[0] for x in D];
    L1=[x[1] for x in D];
    L2=[x[2] for x in D];
    L3=[x[3] for x in D];

    l1.set_xdata(T1);
    l1.set_ydata(L1);

    l2.set_xdata(T1);
    l2.set_ydata(L2);

    l3.set_xdata(T1);
    l3.set_ydata(L3);

    ax1.set_xlim([0,50]);
    ax2.set_xlim([0,50]);
    ax3.set_xlim([0,50]);
    ax1.set_ylim([-1.5,1.5]);
    ax2.set_ylim([-1.5,1.5]);
    ax3.set_ylim([-1.5,1.5]);

    plt.draw();
    i+=0.10;
  show(block=True);

#---main---
log("main process initializing");
test02();
log("main process terminating");

Easy Peasy;

Real-Time Data Analysis From Debug Logs

Debug logs contain a plethora of business and system information.  Most commonly, the logs are cached and post processed, but occasionally it's valuable to process the logs as they are being appended to.  For instance, monitoring the current system state from logs requires a means to read the logs as they are being updated, extract log events and calculating system metrics from the events.

Let's use a simple example, one that monitors (or follows) a 'top.dat' file which is generated by piping 'top' to the file.  The python monitoring utility will process the 'top.dat' file log events as they are written.  Since we wish to monitor the log file(s) and display metrics along the way, we need at least two threads of control: one to monitor and extract information from the 'top.dat' file, the other to display info to the user.  This example calculates the cpu load by extracting the idle percentage from top.

     1 #!/usr/bin/python
     2 import logging
     3 import threading
     4 import time
     5 import glob;
     6 import re;
     7 import datetime;
     8
     9 class Display:
    10   def __init__(self):
    11     self.lock_ = threading.Lock();
    12     self.map_=dict();
    13     pass;
    14   
    15   def update(self,key,val):
    16     self.lock_.acquire();
    17     self.map_[key]=val;
    18     self.lock_.release();
    19   
    20   def display(self):
    21     self.lock_.acquire();
    22     print "---------------"
    23     print datetime.datetime.now();
    24     for k in self.map_.keys():
    25       print "%s : %s"%(k,self.map_[k]);
    26     print "\n"
    27     self.lock_.release();
    28
    29 display=Display();
    30
    31 class fileProcessor:
    32   def __init__(self, fileName): 
    33     self.fileName_=fileName;
    34     logging.info("following %s"%(fileName));
    35     self.follow(open(self.fileName_,'r'));
    36
    37   def follow(self, fp):
    38     fp.seek(0,2);
    39     while True:
    40         line = fp.readline();
    41         if not line:
    42             time.sleep(0.1);
    43             continue;
    44         self.handle(line);
    45
    46   def handle(self, line):
    47 #   logging.info("processing line %s"%(line));
    48     m=re.match('.+, (.+) id,.*',line);
    49     if (m):
    50 #     print m.group(0);
    51       cpuLoad=100.0-float(m.group(1));
    52       print "cpuLoad: %s"%(cpuLoad);
    53       display.update('CpuLoad',cpuLoad);
    54
    55 def thread_function(name):
    56   logging.info("running %s"%(name));
    57   obj=fileProcessor(name);
    58
    59 if __name__ == "__main__":
    60   format = "%(asctime)s: %(message)s";
    61   logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S");
    62
    63   logging.info("main process initializing");
    64   t1=threading.Thread(target=thread_function, args=('./top.dat',));
    65   t1.start();
    66   logging.info("main process terminating");

Not terribly interesting, and you could readily accomplish the same thing on the main thread, there is only one file to monitor.

But suppose you wish to monitor X files, then you really need threading.

     1 #!/usr/bin/python

     2 import logging

     3 import threading

     4 import time

     5 import glob;

     6 import re;

     7 import datetime;

     8

     9 class Display:

    10   def __init__(self):

    11     self.lock_ = threading.Lock();

    12     self.map_=dict();

    13     pass;

    14   

    15   def update(self,key,val):

    16     self.lock_.acquire();

    17     self.map_[key]=val;

    18     self.lock_.release();

    19   

    20   def display(self):

    21     self.lock_.acquire();

    22     print "---------------"

    23     print datetime.datetime.now();

    24     for k in self.map_.keys():

    25       print "%s : %s"%(k,self.map_[k]);

    26     print "\n"

    27     self.lock_.release();

    28

    29 display=Display();

    30

    31 class fileProcessor:

    32   def __init__(self, fileName): 

    33     self.fileName_=fileName;

    34     logging.info("following %s"%(fileName));

    35     self.follow(open(self.fileName_,'r'));

    36

    37   def follow(self, fp):

    38     fp.seek(0,2);

    39     while True:

    40         line = fp.readline();

    41         if not line:

    42             time.sleep(0.1);

    43             continue;

    44         self.handle(line);

    45

    46   def handle(self, line):

    47 #   logging.info("processing line %s"%(line));

    48     m=re.match('.+, (.+) id,.*',line);

    49     if (m):

    50 #     print m.group(0);

    51       cpuLoad=100.0-float(m.group(1));

    52       print "cpuLoad: %s"%(cpuLoad);

    53       display.update('CpuLoad',cpuLoad);

    54

    55 def thread_function(name):

    56   logging.info("running %s"%(name));

    57   obj=fileProcessor(name);

    58

    59 if __name__ == "__main__":

    60   format = "%(asctime)s: %(message)s";

    61   logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S");

    62

    63   logging.info("main process initializing");

    64   D=dict();

    65   while(1):

    66     fList=glob.glob("./*.dat");

    67     logging.info("fList: %s"%(str(fList)));

    68      for e in fList:

    69       if not e in D.keys():

    70         logging.info("starting thread for '%s'"%(e));

    71         D[e]=threading.Thread(target=thread_function, args=(e,));

    72         D[e].start();

    73     display.display();

    74     time.sleep(3);

    75   logging.info("main process terminating");

If you had 100 '*.dat' files the above would spawn 100 threads, each following a dedicated file, each updating the display which is rendered every 3 seconds.

Just recently put something similar together to monitor dozens of user logs and tracking uptime and job submission events. Works real well, hope you have equal success.

Cheers.

Real-Time Plots with Python

In my previous post we described plotting data using MatplotLib utilities and Python.  While this may be valuable, it becomes notably more valuable when you can generate 'live' plots during run-time.  In a past employment I worked with a series of controls engineers that utilized real-time data plots to debug and develop a highly complex multi-axis weapons system and it was the first time I understood how a real-time plot of sequence of steps simplified the development effort.

Let's get started.
Unlike the previous post, let's create the data and plot it as it is generated.

$ cat rtPlot 
#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test01():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(111);
  l1,=ax1.plot(100,100,'r-');

  time.sleep(2.0);
  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i)));
    T=[x[0] for x in D];
    L=[x[1] for x in D];
    l1.set_xdata(T);
    l1.set_ydata(L);
    ax1.relim();
    ax1.autoscale_view();
    plt.draw();
    i+=0.10;
    plt.pause(1/10.0);
  show(block=True);

#---main---
log("main process initializing");
test01();

log("main process terminating");

The result is a dynamically generated plot that resembles the following:

Tie this plotting routine to a system providing run-time information via a socket, or perhaps monitoring network traffic via pcapture libraries and you've got yourself the foundation of a real-time data monitoring system.

Cheers.

Real-Time Plots with Python

In my previous post we described plotting data using MatplotLib utilities and Python.  While this may be valuable, it becomes notably more valuable when you can generate 'live' plots during run-time.  In a past employment I worked with a series of controls engineers that utilized real-time data plots to debug and develop a highly complex multi-axis weapons system and it was the first time I understood how a real-time plot of sequence of steps simplified the development effort.

Let's get started.
Unlike the previous post, let's create the data and plot it as it is generated.

$ cat rtPlot 
#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test01():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(111);
  l1,=ax1.plot(100,100,'r-');

  time.sleep(2.0);
  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i)));
    T=[x[0] for x in D];
    L=[x[1] for x in D];
    l1.set_xdata(T);
    l1.set_ydata(L);
    ax1.relim();
    ax1.autoscale_view();
    plt.draw();
    i+=0.10;
    time.sleep(1/10.0);
  show(block=True);

#---main---
log("main process initializing");
test01();

log("main process terminating");

The result is a dynamically generated plot that resembles the following:

Tie this plotting routine to a system providing run-time information via a socket, or perhaps monitoring network traffic via pcapture libraries and you've got yourself the foundation of a real-time data monitoring system.

Cheers.