Component Implementation - Lesson 5 - Statistics

In most cases you’ll want to gather some run-time statistics from your component to see how it works, detect possible performance issues or congestion problems. All server statistics are exposed and are accessible via XMPP with ad-hoc commands, HTTP, JMX and some selected statistics are also available via SNMP. As a component developer you don’t have to do anything to expose your statistic via any of those protocols, you just have to provide your statistics and the admin will be able to access them any way he wants.

This lesson will teach you how to add your own statistics and how to make sure that the statistics generation doesn’t affect application performance.

spam statitics small

Your component from the very beginning generates some statistics by classes it inherits. Let’s add a few statistics to our spam filtering component:

@Override
public void getStatistics(StatisticsList list) {
  super.getStatistics(list);
  list.add(getName(), "Spam messages found", totalSpamCounter, Level.INFO);
  list.add(getName(), "All messages processed", messagesCounter, Level.FINER);
  if (list.checkLevel(Level.FINEST)) {
    // Some very expensive statistics generation code...
  }
}

The code should be pretty much self-explanatory.

You have to call super.getStatistics(…​) to update stats of the parent class. StatisticsList is a collection which keeps all the statistics in a way which is easy to update, search, and retrieve them. You actually don’t need to know all the implementation details but if you are interested please refer to the source code and JavaDoc documentation.

The first parameter of the add(…​) method is the component name. All the statistics are grouped by the component names to make it easier to look at particular component data. Next is a description of the element. The third parameter is the element value which can be any number or string.

The last parameter is probably the most interesting. The idea has been borrowed from the logging framework. Each statistic item has importance level. Levels are exactly the same as for logging methods with SEVERE the most critical and FINEST the least important. This parameter has been added to improve performance and statistics retrieval. When the StatisticsList object is created it gets assigned a level requested by the user. If the add(…​) method is called with lower priority level then the element is not even added to the list. This saves network bandwidth, improves statistics retrieving speed and is also more clear to present to the end-user.

One thing which may be a bit confusing at first is that, if there is a numerical element added to statistics with 0 value then the Level is always forced to FINEST. The assumption is that the administrator is normally not interested zero-value statistics, therefore unless he intentionally request the lowest level statistics he won’t see elements with zeros.

The if statement requires some explanation too. Normally adding a new statistics element is not a very expensive operation so passing it with add(…​) method at an appropriate level is enough. Sometimes, however preparing statistics data may be quite expensive, like reading/counting some records from database. Statistics can be collected quite frequently therefore it doesn’t make sense to collect the statistics at all if there not going to be used as the current level is higher then the item we pass anyway. In such a case it is recommended to test whether the element level will be accepted by the collection and if not skip the whole processing altogether.

As you can see, the API for generating and presenting component statistics is very simple and straightforward. Just one method to overwrite and a simple way to pass your own counters. Below is the whole code of the example component:

Example component code. 

public class TestComponent extends AbstractKernelBasedComponent {

  private static final Logger log = Logger.getLogger(TestComponent.class.getName());

  @Inject
  private TestModule testModule;

  @Override
  public synchronized void everyMinute() {
    super.everyMinute();
    testModule.everyMinute();
  }

  @Override
  public String getComponentVersion() {
  String version = this.getClass().getPackage().getImplementationVersion();
    return version == null ? "0.0.0" : version;
  }

  @Override
  public String getDiscoDescription() {
    return "Spam filtering";
  }

  @Override
  public String getDiscoCategoryType() {
      return "spam";
  }

  @Override
  public int hashCodeForPacket(Packet packet) {
    if (packet.getElemTo() != null) {
      return packet.getElemTo().hashCode();
    }
    // This should not happen, every packet must have a destination
    // address, but maybe our SPAM checker is used for checking
    // strange kind of packets too....
    if (packet.getElemFrom() != null) {
      return packet.getElemFrom().hashCode();
    }
    // If this really happens on your system you should look carefully
    // at packets arriving to your component and decide a better way
    // to calculate hashCode
    return 1;
  }

  @Override
  public boolean isDiscoNonAdmin() {
    return false;
  }

  @Override
  public int processingInThreads() {
    return Runtime.getRuntime().availableProcessors();
  }

  @Override
  public int processingOutThreads() {
    return Runtime.getRuntime().availableProcessors();
  }

  @Override
  protected void registerModules(Kernel kernel) {
    // here we need to register modules responsible for processing packets
  }

  @Override
  public void getStatistics(StatisticsList list) {
    super.getStatistics(list);
    list.add(getName(), "Spam messages found", testModule.getTotalSpamCounter(), Level.INFO);
    list.add(getName(), "All messages processed", testModule.getMessagesCounter(), Level.FINE);
    if (list.checkLevel(Level.FINEST)) {
      // Some very expensive statistics generation code...
    }
  }

}

Example module code. 

@Bean(name = "test-module", parent = TestComponent.class, active = true)
public static class TestModule extends AbstractModule {

  private static final Logger log = Logger.getLogger(TestModule.class.getCanonicalName());

  private Criteria CRITERIA = ElementCriteria.name("message");
  private String[] FEATURES = { "tigase:x:spam-filter", "tigase:x:spam-reporting" };

  @ConfigField(desc = "Bad words", alias = "bad-words")
  private String[] badWords = {"word1", "word2", "word3"};
  @ConfigField(desc = "White listed addresses", alias = "white-list")
  private String[] whiteList = {"admin@localhost"};
  @ConfigField(desc = "Logged packet types", alias = "packet-types")
  private String[] packetTypes = {"message", "presence", "iq"};
  @ConfigField(desc = "Prefix", alias = "log-prepend")
  private String prependText = "Spam detected: ";
  @ConfigField(desc = "Secure logging", alias = "secure-logging")
  private boolean secureLogging = false;
  @ConfigField(desc = "Abuse notification address", alias = "abuse-address")
  private JID abuseAddress = JID.jidInstanceNS("abuse@locahost");
  @ConfigField(desc = "Frequency of notification", alias = "notification-frequency")
  private int notificationFrequency = 10;
  private int delayCounter = 0;
  private long spamCounter = 0;
  private long totalSpamCounter = 0;
  private long messagesCounter = 0;


  @Inject
  private TestComponent component;

  public void everyMinute() {
    if ((++delayCounter) >= notificationFrequency) {
      write(Message.getMessage(abuseAddress, component.getComponentId(), StanzaType.chat,
                               "Detected spam messages: " + spamCounter, "Spam counter", null,
                               component.newPacketId("spam-")));
      delayCounter = 0;
      spamCounter = 0;
    }
  }

  @Override
  public String[] getFeatures() {
    return FEATURES;
  }

  @Override
  public Criteria getModuleCriteria() {
    return CRITERIA;
  }

  public long getMessagesCounter() {
    return messagesCounter;
  }

  public long getTotalSpamCounter() {
    return totalSpamCounter;
  }

  public void setPacketTypes(String[] packetTypes) {
    this.packetTypes = packetTypes;
    Criteria crit = new Or();
    for (String packetType : packetTypes) {
      crit.add(ElementCriteria.name(packetType));
    }
    CRITERIA = crit;
  }

  @Override
  public void process(Packet packet) throws ComponentException, TigaseStringprepException {
    // Is this packet a message?
    if ("message" == packet.getElemName()) {
      component.updateServiceDiscoveryItem(component.getName(), "messages",
                                           "Messages processed: [" + (++messagesCounter) + "]", true);
      String from = packet.getStanzaFrom().toString();
      // Is sender on the whitelist?
      if (Arrays.binarySearch(whiteList, from) < 0) {
        // The sender is not on whitelist so let's check the content
        String body = packet.getElemCDataStaticStr(Message.MESSAGE_BODY_PATH);
        if (body != null && !body.isEmpty()) {
          body = body.toLowerCase();
          for (String word : badWords) {
            if (body.contains(word)) {
              log.finest(prependText + packet.toString(secureLogging));
              ++spamCounter;
              component.updateServiceDiscoveryItem(component.getName(), "spam", "Spam caught: [" +
                                                   (++totalSpamCounter) + "]", true);
              return;
            }
          }
        }
      }
    }
    // Not a SPAM, return it for further processing
    Packet result = packet.swapFromTo();
    write(result);
  }
}