Using a Clusterable Cache
Learn how to increase performance on a multiple Network Director deployment using a clusterable cache.
Supported Platforms: 7.2
Table of Contents
In previous Policy Manager versions, a cluster of Network Director (ND) instances worked independently. This means if the policy only allowed 10 requests per minute each node (instance) in an ND cluster would allow 10 requests per minute. Data was not synchronized across clusters (e.g., ND1 processed 2 requests, so there are 8 counts left, etc.).
Beginning with Policy Manager version 7.2, the processing of requests can now be synchronized across a cluster of multiple ND instances within a single deployment by creating a clusterable cache. A clusterable cache is memory that different processes can share.
For example, if you want the entire cluster to collectively only allow 10 requests per minute you can configure the NDs to share memory (i.e., a clusterable cache) which will hold information using counters. Then when one node receives a message and increments a counter, it's the same counter the other nodes are using.
A clusterable cache can be used with the following policies:
- Throughput Quota Policy - Allows you to monitor a policy by specifying a throughput limit (i.e., quota) and queue size.
- Bandwidth Quota Policy - Allows you to monitor a policy by specifying a bandwidth cap that can be specified as kilobytes or megabytes per second.
- WS-Security Policies - Using a clusterable cache, you can share nonces to prevent getting duplicate messages across all NDs in a cluster. Supported WS-Security policies include WS-Security Message, WS-Security Asymmetric Binding, and WS-Security Transport policies.
You configure a clusterable cache by updating a series of configuration categories in the SOA Software Administration Console > Configuration section. The process involves setting up the clusterable cache using the com.soa.grid property, and then configuring properties for the Throughput Quota Policy, Bandwidth Quota Policy, and WS-Security Policies to use the clustered cache.
How do I create a clusterable cache and use it in a policy?
Perform the following steps for each Network Director instance:
- Step 1: Create Clusterable Cache
- Step 2: Use Clustered Cache in Throughput Quota, Bandwidth Quota, and WS-Security Asymmetric Binding Policies
- Step 3: Restart Network Director Instances
- Step 4: Synchronize Clocks
Step 1: Create Clusterable Cache
- Select com.soa.grid.
- Modify/verify the following settings, as shown below (Note that the Multicast option (grid.network.config.enableMulticast) is currently not supported):
- Set grid.network.config.enableMulticast to false.
- Set grid.network.config.enableTcp to true.
- Set grid.network.config.localPort so that the value is a unique port number for the server.
Note: If you don't specify a port value, the platform assumes the default of 47500 but will also try other ports by incrementing the value by 1 and retrying. It's best to always specify the localPort value as well as the IP address.
- In grid.network.config.peerAddresses, specify the peer address of each Network Director instance in your deployment. Omit the peer address of the current Network Director instance in the list. The peerAddress value is a comma-separated list of all the other systems within the cluster (grid). Valid addresses can be:
- Hostname. For example: machine101,machine102
- IP address with port number. For example: 18.104.22.168:47501,22.214.171.124:47502,126.96.36.199:47503
- IP address without port number. For example: 188.8.131.52,184.108.40.206,220.127.116.11
- Range of IP addresses without port number. For example: 18.104.22.168-7
Note: Do not include a comma (,) at the end of the list as the platform would interpret it as an empty member and return an exception.
- Click Apply Changes.
Step 2: Use Clustered Cache in Throughput Quota, Bandwidth Quota, and WS-Security Asymmetric Binding Policies
- Launch the SOA Software Administration Console and navigate to the Configuration tab.
- In the Configuration Categories section, you can select properties for the Throughput Quota Policy (com.soa.policyhandle.quota.throughput) or Bandwidth Quota Policy (com.soa.policy.handle.quota.bandwidth).
- Change use.clustered.cache to True and click Apply Changes.
Above: Throughput Quota Policy (com.soa.policyhandle.quota.throughput)
Above: Bandwidth Quota Policy (com.soa.policy.handle.quota.bandwidth)
- For the WS-Security Policy (com.soa.policy.handle.wssp.noncecache), ND will automatically use the clustered cache if you enabled the Grid and TCP options in Step 1, otherwise the local cache will be used.
Above: WS-Security Policy (com.soa.policy.handle.wssp.noncecache)
Step 3: Restart Network Director Instances
- Before you can successfully use the clusterable cache, you must restart all Network Director instances after completing the configuration category changes outlined in the previous steps.
Step 4: Synchronize Clocks
- The Policy Manager clusterable caching logic is based on the system clock. To ensure there is not a lag in one of the Network Director instances while processing requests, you must synchronize all system clocks so they match (i.e., by milliseconds and seconds).
Step 5: Test Policy Enforcement (Example)
Throughput Quota Policy
- Configure a Throughput Quota Policy to process 10 requests per minute.
- Virtualize a service.
- Attach the policy to the virtual service.
- Call the virtual service from the SOAP UI.
- Call the virtual service 10-20 times.
- In approximately one minute, ten requests will be success and the remainder will fail.
- Review the alerts. All requests failed past the 10 requests per minute will show a quota exceeded alert.
Bandwidth Quota Policy
- Configure a Bandwidth Quota Policy with 10 kbps for the request.
- Virtualize a service.
- Attach the policy to the virtual service.
- In SOAP UI, run a single test step (e.g, Echo-Request1) in your case and make sure that the response is received successfully.
- Click on raw tab and note the content length of the request. Let's assume that content length is 1129 bytes.
- Configure your load test's limit as 60 seconds or whatever number of seconds you want.
- After running the test for 60 seconds let's say you see a count of 616 and an error count of 10. In this case the net count is 616-10=606. Ideally you would want to see the error count as 0. Some of these errors might be caused because a backend physical service took too long to return the response and then in this example the errors should have been included in the net count as ND allowed this request and passed it to physical service. Some of these errors might be caused because all of the ND threads were busy due to high load and in this example errors should not be included in net count.
- So let's assume that the net count is 606. Total number of bytes allowed by ND in 60 seconds-> 606*1129= 684174.
- Total number of bytes allowed per second 684174/60= 11402.9 bytes ~ 10kbps.
WS-Security Asymmetric Binding Policy
For an example illustration of how to use clusterable caching with this WS-Security policy type, refer to Using a Clusterable Cache with WS-Security Asymmetric Binding Policy.