EPL Memory Profiler

EPL Memory profiling is about investigating and analyzing an Apama application’s behavior to aid in optimizations for better memory usage and identifying any leaks. Apama’s capability to profile the EPL memory usage does provide a handy mechanism to identify any listener or object leaks used in the application within the correlator. Identifying such leaks could otherwise be a daunting task for any developer.

Note : Usage of the memory profiler is recommended during the development or debugging of an EPL application but, not when in production since it impacts the performance of the application.

The EPL memory profiler is invoked using the -r (or --dorequest) option of the engine_management tool, followed by a request. Several requests are available for the EPL memory profiler, which are described below. The information that is returned for a request can be viewed directly (for example, in the Apama Command Prompt), or it can be written to a comma-separated values (CSV) file which can easily be viewed in tabular form using a tool such as Microsoft Excel.

Commands

The commands applicable for the EPL memory profile are explained briefly below. Details of the profiler can be seen in the documentation (Using the EPL memory profiler).

eplMemoryProfileOverview – Returns information on all the monitors in the correlator. Example:

eplMemoryProfileMonitorInstanceDetail – Returns information for all EPL types across the monitor instances of a specific monitor in the correlator. The monitor-name is expected to be the fully qualified name of the monitor. Example:

eplMemoryProfileMonitorDetail – Command is similar to eplMemoryProfileMonitorInstanceDetail, except that it aggregates the object count and size from each monitor instance, displaying data per monitor rather than per monitor instance.  Example:

 

Example

We will use a simple EPL application which gets a Request event and processes it to return a response to demonstrate identifying leaks in an EPL application using the EPL memory profiler.

Below are the EPL files for the example.

Event Definitions :

Request Handler monitor

Request and Response Generator which basically generates 100 Request events and each Request generates either a Response or ResponseError event and about 25% of the responses will be errors.

Now Launch the Correlator and let us use the EPL Memory Profiler to understand the application memory profile. Once the Correlator is started and the EPL files injected open the “Apama Command Prompt“.

From the above EPL application the expectation is that once all Request events are handled there should be no pending requests in the dictionary. Fire the below command in the Apama Command Prompt.

MonitorMonitor instancesEPL objectsListenersBytesOverhead bytes
com.apamablog.leaks.DemoGeneratorMonitor111312760
com.apamablog.leaks.DemoLeaksMonitor198251784015688

The above table is the output we get in the csv format. Notice that the DemoGeneratorMonitor has only one EPL object and one listener which is on expected lines and the memory usage is also acceptable. The DemoLeaksMonitor has 98 EPL objects and 25 active listeners with an unexpectedly high memory consumption. So to understand the further details on the DemoLeaksMonitor we can use the other commands available i.e. “eplMemoryProfileMonitorDetail” or the “eplMemoryProfileMonitorInstanceDetail“. As seen above the monitors have only once instance each so the instance detail would not be of great value in this case so we can probe further by using the “eplMemoryProfileMonitorDetail” command.

The “eplMemoryProfileMonitorDetail” command expects the fully qualified name of the monitor being probed so we can execute the below command in the command line.

MonitorPersistentEPL typeEPL objectsBytes
com.apamablog.leaks.DemoLeaksMonitorfalsecom.apamablog.leaks.DemoLeaksMonitor.Global Variables1144
com.apamablog.leaks.DemoLeaksMonitorfalsecom.apamablog.leaks.DummyHolder243120
com.apamablog.leaks.DemoLeaksMonitorfalsecom.apamablog.leaks.Request243232
com.apamablog.leaks.DemoLeaksMonitorfalse“dictionary<string,com.apamablog.leaks.Request>”11304
com.apamablog.leaks.DemoLeaksMonitorfalseListener – C:\Users\pad\workspace912\Demo\monitors\DemoLeaks.mon:11244992
com.apamablog.leaks.DemoLeaksMonitorfalseListener – C:\Users\pad\workspace912\Demo\monitors\DemoLeaks.mon:91200
com.apamablog.leaks.DemoLeaksMonitorfalsesequence<integer>243072
com.apamablog.leaks.DemoLeaksMonitorfalsestring241776

As seen in the above table there are unexpected 24 Request event objects which has led to a similar number of DummyHolder event, sequence<integer>, string objects since they are contained within the Request event, also there exists 24 Listener objects at line 11 i.e.

In this application we were expecting all the Request events to be handled and the holding dictionary of the pending request events to be empty which does not seem to be the case. We yet do have 24 Request objects being held in the monitor and as the only place we are storing the Request object is within the dictionary, so we can infer that the dictionary is not being cleaned up. As the Request event can have responses of Response and ResponseError we have not handled the ResponseError event thus leading the Request objects to be held up in the dictionary. The monitor generates for about 25% of the Request events a ResponseError event. To prevent this leak let us handle the ResponseError event as a result of a Request.

As seen below in the DemoLeaks.mon adds a listener for the ResponseError event and removes it from the dictionary. See listener at line 21 below.

Restart the correlator inject the EPL files and run the eplMemoryProfileOverview command in the Apama Command Prompt.

MonitorMonitor instancesEPL objectsListenersBytesOverhead bytes
com.apamablog.leaks.DemoGeneratorMonitor111312760
com.apamablog.leaks.DemoLeaksMonitor121012204019072

As seen in the above table the DemoGeneratorMonitor is the same as expected since we made no changes to it but the DemoLeaksMonitor has 101 Listeners which is quite unexpected since we expect only one listener for the Request event and no other listeners. As explained above the  “eplMemoryProfileMonitorDetail” command can be used to probe further.

MonitorPersistentEPL typeEPL objectsBytes
com.apamablog.leaks.DemoLeaksMonitorfalsecom.apamablog.leaks.DemoLeaksMonitor.Global Variables1144
com.apamablog.leaks.DemoLeaksMonitorfalse“dictionary<string,com.apamablog.leaks.Request>”1344
com.apamablog.leaks.DemoLeaksMonitorfalseListener – C:\Users\pad\workspace912\Demo\monitors\DemoLeaks.mon:11316448
com.apamablog.leaks.DemoLeaksMonitorfalseListener – C:\Users\pad\workspace912\Demo\monitors\DemoLeaks.mon:216914904
com.apamablog.leaks.DemoLeaksMonitorfalseListener – C:\Users\pad\workspace912\Demo\monitors\DemoLeaks.mon:91200

As seen in the above table the listeners on line 11 and 21 are unexpected while there are no more Request objects as we have cleared the holding dictionary correctly.

As seen in the EPL we have two listeners for every requestId but we get only one response i.e. either Response or ResponseError we need to terminate one listener when the other fires. We can fix this by using the ‘and not’ pattern – adding ‘and not OtherEvent(..)’  will terminate the listener without firing it if OtherEvent is received, thus fixing the listener leak.

Restart the correlator inject the EPL files and run the eplMemoryProfileOverview command in the Apama Command Prompt.

MonitorMonitor instancesEPL objectsListenersBytesOverhead bytes
com.apamablog.leaks.DemoGeneratorMonitor111312760
com.apamablog.leaks.DemoLeaksMonitor121688264

As seen in the above table now we do not have any unexpected EPL object or listener count and the memory for the monitor also seems acceptable so now there seems to be no leaks in the application.

Attached are the above artefacts for reference.

DemoLeak1

DemoLeak2

DemoLeak3

– Pavan

Disclaimer:
Utilities and samples shown here are not official parts of the Software AG products. These utilities and samples are not eligible for technical support through Software AG Customer Care. Software AG makes no guarantees pertaining to the functionality, scalability, robustness, or degree of testing of these utilities and samples. Customers are strongly advised to consider these utilities and samples as “working examples” from which they should build and test their own solutions