Running Benchmark Applications: CMIS

[derek]

Table of Contents

• 1 What it Does


• 2 Prerequisites


• 3 Deploying
  
  
  
  • 3.1 Local Deployment
  
  
  • 3.2 Remote Deployment
  
  


• 4 Create and Start a Test Run


• 5 Handling failures


• 6 Extra Information

What it Does

• Exercises basic CRUD operations using the Java OpenCMIS client
  
• Provides a template for extending operations and scenarios to cover use cases specific to individual applications
  
• Record, in detail, an errors that occurred during individual CMIS operations
  
• Record individual CMIS operation times
  

---

Prerequisites

Use the Benchmark Testing with Alfresco page for version compatibility.

• Java 1.7.0_51 or later
  
• MongoDB 2.6.3 or later installed and running on port 27017 on some server: <mongo-host>
  
• A compatible version of the Benchmark Server running on a Tomcat7 at port 9080: <bmserver-host>
  
• Alfresco with /alfresco available: <alfresco-host>.  Any CMIS server can be used this document will only deal with Alfresco as the server.  The target server requires a folder with full write access to ALL users e.g. /Share/load1 with Collaborator rights for GROUP_EVERYONE.
  
• A user data mirror reflecting the list of users that can be used by the test.  See the Alfresco Sign Up test on how to create users in Alfresco.
  
• Validate the CMIS server availability using the Apache CMIS Workbench and check the shared folder (e.g. /Share/load1) permissions.
  

---

Deploying

Local Deployment

• Check out source code of ent-signup test.
  
• Check out the required tag:
  

/benchmark-cmis>git checkout V1.0

• Build and start a local Tomcat7 instance
  

/benchmark-cmis>mvn tomcat7:run -Dmongo.config.host=<mongo-host>

...
[INFO] Using existing Tomcat server configuration at c:\work\projects\bm-cmis\target\tomcat
...
10:49:10,944 [localhost-startStop-1] [ INFO] [                 MongoClientFactory: 124] - New MongoDB client created using URL: mongodb://mongo-host/?conne...
...
10:49:11,294 [localhost-startStop-1] [DEBUG] [                LifecycleController: 174] - Started components: appLifeCycleController
Nov 03, 2014 10:49:11 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ['http-bio-9090']

Remote Deployment

• Set up a Tomcat7 load driver instance with the manager application listening on port 9080 and configure your Maven settings with the manager application credentials.
  
• Deploy the application directly into the load driver:
  

mvn tomcat7:redeploy -DskipTests  -Dbm.tomcat.ip=<bmdriver-host> -Dbm.tomcat.port=9080 -Dbm.tomcat.server=bm-remote

• Connect to the driver server and check that the application was successfully deployed.
  

 http://bmdriver-host:9080/manager

• You can start/stop the driver applications as required
  

---

Create and Start a Test Run

Properties set for the test are inherited by each test run, which can then override any values as required.  The choice about where to change properties (at the test level or at the test run level) depends on how the test is used.

1. When setting up a test for comparative benchmarking (comparing one machine's or release's performance against another's) all parameters required are set at the test level except for the target server IP address.  Alfresco uses this method when performing benchmark testing against different versions of the product; the only property that needs to change for each test run is the IP address of the server being tested.  The results from these test runs are seldom discarded, being extremely useful for historic comparisons.
   
2. When performing load tests for discovery purposes e.g. to diagnose a performance issue, the server IP address is set at the test level and inherited by each test run, which will only contain load-related changes such as the time between the triggering of new CMIS sessions.  Once the diagnostics are complete, the entire test is often discarded.
   

The example given here will assume the latter case i.e. several test runs are going to be made against a single server for investigative purposes.

• Connect to the Benchmark Server
  

http://bmserver-host:9080/alfresco-benchmark-server

• Create a new test, CMIS_10 using alfresco-benchmark-tests-cmis-1.0-schema: 0 or whichever version you deployed
  
• The property edit page is displayed, but can also be accessed using the gears icon.
  
• Click on the Driver Details box, which will show details of all compatible driver(s) connected to the same configuration database.
  
• The following properties should be set for all new tests:
  

Section	Property	Description
MongoDB Connection	mongo.test.host	The hostname of a MongoDB server where the test results and general working data will be stored.  This should be an IP address that is visible to all load drivers and _may_ be the same MongoDB instance as the configuration database.  This value must always be set for new tests, as there is no working default.  A value of 'localhost' is dangerous and should not be used except for local testing.
Test Controls	Test Duration	Time in seconds for the test to run.  This acts as an emergency shut off if the test is unable to complete the desired operations e.g. if the server-in-test fails to respond to new session requests and each session only times out after 5s.
CMIS Session Details	Test Folder Path	A folder within the server that ALL users can write to.  This may need to be configured on the server before use.  For the Alfresco test, log into Share using the admin user and create a folder called load1 in the Shared Files section.  The setting to apply here (look at the WebDAV path for the folder) is /Shared/load1.
	CMIS Binding URL	The location of the CMIS atom or browser definition.  For a standard Alfresco 5.0 instance, set this to http://${cmis.host}:${cmis.port}/alfresco/api/-default-/public/cmis/versions/1.1/browser.  The host and port values will be substituted by the test application.
	CMIS Binding Type	Either atompub or browser.  For this example, use browser.
	CMIS Server Port	Use 8080 for a default Alfresco server install.
	CMIS Server Hostname	The server host name.  It is important to use the same host name as used by the Alfresco Sign Up test so that the user data mirror name matches.  As mentioned before, this value can be set once at the test level when performing repeated runs against the same server.
Load Control	CMIS Session Count	The number of CMIS sessions to start.  Each session will choose a scenario to run through.  The default of 20 is low and can be increased to get a longer-running test.

• Click back up to the CMIS_10 test.  You should be presented with 'No test runs found ...'.
  
• Create a test run named TRIAL_01.
  
• The list of test runs is displayed.  Click on the cog (properties editor) of the test run.
  
• Notice that the properties you set in the test are inherited by the test run.  This example assumes that all properties remain the same except for the load parameters:
  

Section	Property	Description
Load Control	CMIS Session Count	Defaults to 20.  Increase to 60.
	CMIS Session Delay	The default load is 10 sessions per second (100ms delay between sessions).  Change this to 1000 so that we have one session per second.  Combined with the 60 sessions, we should have a test run of just over a minute.

• Check the other sections to ensure that the inherited properties are correct.  If not, go back up to the test properties and set them there.
  
• Click back up to the TRIAL_01 test run.
  
• Click the run button.
  

The application console or log file for the CMIS test application should display the following:

...
2:42:51,472 [TestRunPing-alfresco-benchmark-tests-cmis-1.0-0] [ INFO] [                            TestRun: 398] - Test run application context starting:
  Run ID:       5457776c3881731bc643bfb6
  Test Name:    CMIS_10
  Run Name:     TRIAL_01
  Driver ID:    54575da73881ff768c85fec1
  Release:      alfresco-benchmark-tests-cmis-1.0
  Schema:       0
  Test Run Properties:
     files.ftp.path=/sites/www.linuxfromscratch.org/images
...
12:42:51,727 [TestRunPing-alfresco-benchmark-tests-cmis-1.0-0] [ INFO] [                 MongoClientFactory: 124] - New MongoDB client created using URL: mongodb://localhost/?safe=true&
...
12:42:52,255 [CMIS_10.TRIAL_01-Controller] [ INFO] [                    EventController: 249] -     Event processing started: CMIS_10.TRIAL_01 (64 events per second using 16 threads)
12:42:52,255 [TestRunPing-alfresco-benchmark-tests-cmis-1.0-0] [DEBUG] [                LifecycleController: 174] - Started components: testLifeCycleController
12:42:52,266 [TestRunPing-alfresco-benchmark-tests-cmis-1.0-0] [ INFO] [                            TestRun: 227] - Transitioned test run to STARTED: 5457776c3881731bc643bfb6

---

Handling failures

If the test fails to progress from SCHEDULED to STARTED, it could be because:

1. mongo.test.host has not been set.  Currently the only way to see the error is to locate the tests log file and check that the value following New MongoDB client created using URL is correct.
   
2. the CMIS atompub or browser URL has been set correctly.  Query the failures in the MongoDB database.
   

If the test starts but experiences a high number of failures, the failures can be accessed directly in the MongoDB results.  The video, Following up on Failures shows how this can be done.  It is especially useful when the server under load starts to produce errors or fails to respond correctly.  From the MongoDB console, it would look something like this:

 mongo <mongo-data-host>
 use bm20-data
 db.CMIS_10.TRIAL_01.find({success:false}).pretty();

---

Extra Information

• A video showing the creation of users and the user data mirror.
  
• Videos from Alfresco Summit 2014.