The ARM (Application Response Measurement) standard has been around for many years; first as a standard of the Computer Measurement Group (CMG) [CMG1], then as a standard of the Open Group [OPEN1]. Although it is currently in its fourth major revision, adoption by the industry has been limited at best.
One of the key limiting factors in the past has been the burden on the programmers to write code to the standard. This introduced a level of overhead that many coders found unacceptable, both in terms of coding cost and effort, and in terms of run time overheads. It's certainly possible to run programs coded to the standard without the overhead of measurement, but this involves either relinking against an empty library, or reconfiguring the collection agents.
The latest trend in the industry is to embed the instrumentation in the application containers, such as the web server [APACHE1,CART2004], the application server, or key system components such as the database server[MYARM1]. These efforts have proven much more successful and are starting to become the norm. Coders don't have to worry about a new API, and measurements can be taken at the discretion of the systems administrator or the capacity manager. In fact, they lead to a new class of tools that allow management software to adjust system resources to meet changing systems requirements [EWLM04].
Like all engineering efforts, fixing one issue usually shifts the problem to a different area. The successful development of new technologies that will effect the measurement of application response times still depends on the availability of a data collector. Many of the tools currently developed have depended on the developers also producing an ARM agent. For many, this is an undesirable exercise that is secondary to their primary development goal. Similarly, many in the open source community who've depended on inexpensive tools to continue their development have been without an adequate tool to collect ARM measurements. The ARM4.org project was instigated to address this glaring need.
Project Goals
The primary goal of this project is to provide an open source implementation of the ARM 4.0 [ARMC04,ARMJ04] and ARM 4.1[ARMC07,ARMJ07] standards. As the standard evolves, the project would continue to evolve along with it.
The target audience can be broken down into 2 main groups:
Developers – these are the people developing new tools and technologies, or those who wish to code applications that interface with the agent directly.
Enterprise Administrators – these are the people who just wish to use the data collector within their organization to manage their applications.
In its pre-production phases (pre 1.0), the tool will primarily be usable by the developers as architectures are fleshed out and management tools developed.
Portability is a primary design goal. Since the final product is intended to be used in production environments, the tool should be able to run on as many platforms as possible.
The tool should be as non-invasive as possible. Running the tool shouldn't consume so much of the systems resources that the tool becomes too cumbersome for wide scale deployment. Nor should it have such a significant impact on the instrumented applications that it becomes an unacceptable overhead.
Project Overview
Early development efforts were geared more towards building a solid and stable application architecture than to building a standards compliant collector. Different technologies were examined including message queuing and shared memory technologies, before the project finally settled on a hybrid architecture that used the best elements of each. Through all of this, portability was tested across a variety of platforms.
At this point, portability is limited to UNIX style platforms. It has been testes on Solaris 10, various flavors of Linux, and Mac OS X. With updates to the build system, it is likely to run on HP/UX, AIX, and other UNIX variants with minimal porting issues, but this has not been tested. Porting to Windows based platforms will follow once basic stability and performance issues are addressed.
At the time of this writing, version 0.6 of the agent is being prepared for release. This version will be of primary use to application developers, and isn't yet ready for enterprise deployment.
Architecture
The piece the application developer sees is the libarm4 library. This is a shared library that gets linked against the application that manages communications between the application and the agent. It's also capable of detecting when no agent is active. This has the effect of insulating the instrumented application from the agent in cases where problems with the agent might impact the application. Similarly, it also means the agent can be administered independently of the application.
Similarly, the piece the systems administrator sees is the arm4_daemon program, which is the actual collector.
Communications between the two elements is controlled by the shared memory region. Messages are passed from libarm4 to arm4_daemon using standard UNIX message queues, and the results sent back to the agent via data structures in the shared memory region. Synchronization objects are also contained in the shared memory region.
The key to operation is the shared memory region. Information from the agent is returned to the instrumented application by way of the shared memory. Inter-process communications are synchronized using semaphores contained in this region.
The data store keeps a permanent record of the measured data. While future versions will support multiple data stores, the current version uses the Berkeley DB data store. This is a fast, robust data store that allows for concurrent access from multiple threads and processes, and is available on many platforms.
There are also a number of support utilities that control the operation of the agent, and provide some rudimentary reporting. These are minimal at present and are designed to provide only the most basic functions for program control, debugging, and development.
Multi-Thread Support Issues
The system as designed is inherently multi-threaded. In the case of the arm4_daemon program, this improves concurrency and reduces latency. For libarm4, it becomes necessary to coordinate access to the shared memory region. No effort has been made to explicitly support single threaded applications. While this may cause some conflicts with single-threaded libraries on some systems, most applications should be able to link against the library with little difficulty.
Multi-Agent Support
The agent is designed to support multiple independent instances on a single system. Each instance gets its own shared memory region, and data store instance. In the case of the instrumented application, the instance is controlled by setting the ARM4_INSTANCE environment variable. The agent selects the instance through the instance parameter in its configuration file.
Access is controlled by the operating system. Both the instrumented application and the agent need access to the shared memory region. On the UNIX operating system, this is normally done through the use of group permissions. It is also possible to run the collector and the instrumented application in user space. This means a user can run a development environment without having systems administration privileges.
Program Control
The arm4_control program allows the systems administrator to control data collection without having to restart the agent or the application.
Some of the commands available are:
archive – create a dated archive of the data store. In addition to providing a data backup, it also allows the agent to age the data and prevent the data store from growing indefinitely.
collect – turn data collection on or off. This gives the administrator some discretion in terms of when data is collected.
remove – removes the shared memory region and synchronization objects leaving the database intact. This may be necessary when upgrading from one program version to another.
wipe – similar to remove, but also removes the database from the system.
status – verify the run state of a collector instance
stop – shut down the collector.
3.5 Reporting and Debugging
At this point in development, reporting is minimal. This will be expanded as the project progresses.
The arm4_report program provides two types of reports. The detailed report provides details on all transactions recorded by the system, while the summary report gives some overall statistics for all transaction instances.
The arm4_dump program gives a raw and unfiltered look at the internal databases. It is intended for debugging only.
Current Standard Support
The collector currently implements the C version of the 4.0 standard. While it should be possible to collect Java measurements using available Java libraries, this hasn't been tested. Java support, and 4.1 standard support, will be added in future releases.
Future Development
This project is still in its infancy and needs a number of features to make it an enterprise quality application. Development priorities in the order of importance are:
Operational stability – the agent should run consistently, stably, and securely with minimal impact on the instrumented applications, and minimal resource requirements.
Standards compliance – full implementation of the ARM 4.0 and 4.1 standards. This includes full Java and C language support. This may also include support for ARM 2[ARMC02] and ARM 3[ARMJ02] collection as well.
Reporting – the ability to generate rich and complete reports on the applications across multiple tiers.
Troubleshooting – generate alerts when an application fails to meet performance goals.
While the priorities can be summed up in these four simple points, the number of tasks left to accomplish is much larger. Details about the current project road map and development milestones can be obtained from the ARM4.org website[ARMORG1].
Summary
The ARM4.org project is well on its way to developing an ARM tool that developers and systems administrators can enjoy. While not yet an enterprise class application, it can provide the basis for the development of other ARM based tools. Of course, as an open source project, the price is right.
At this point, the field is wide open for a variety of projects based on ARM. Open source projects such as the MySQL database server, or the Geronimo application server could be instrumented. Instrumentation libraries for web languages such as PHP, or scripting languages such as PERL or Python would add a whole new dimension to performance management. Developers can now rest assured that the ARM tools they need will be available to them and to the users of their software.
References
[ARMC02] ARM 2.0 Technical Standard, available from http://www.opengroup.org/management/arm.htm/
[ARMC04] ARM Issue 4.0 Version 2 - C Binding, available from http://www.opengroup.org/management/arm.htm/
[ARMC07] ARM Issue 4.1 Version 1 - C Binding, available from http://www.opengroup.org/management/arm.htm/
[ARMJ02] ARM 3.0 Java Binding Technical Standard, available from http://www.opengroup.org/management/arm.htm/
[ARMJ04] ARM Issue 4.0 Version 2 - Java Binding, available from http://www.opengroup.org/management/arm.htm/
[ARMJ07] ARM Issue 4.1 Version 1 - Java Binding, available from http://www.opengroup.org/management/arm.htm/
[ARMORG1] ARM4.org Project Web Site, http://www.arm4.org/
[APACHE1] mod_arm4
https://svn.apache.org/repos/asf/httpd/mod_arm4/trunk/
[CART2004] D. Carter, “ARMing Apache”, CMG Proceedings (2004)
[CMG1] CMG. ARM Working Group
http://regions.cmg.org/regions/cmgarmw/index.html
[EWLM04] Enterprise Workload Manager, available from http://publib.boulder.ibm.com/infocenter/eserver/v1r1/en_US/index.htm?info/ewlminfo/kickoff.htm
[JOHN2004] M. Johnson, “Monitoring and Diagnosing Applications with ARM 4.0”, CMG Proceedings (2004)
[MYARM1] MySQL Instrumentation, http://myarm.de/solutions/mysql.html
[OPEN1] The Open Group. Application Response Measurement
http://www.opengroup.org/management/arm.htm/
Posts
