Using a Clusterable Cache
Learn how to increase performance on a multiple Network Director deployment using a clusterable cache.
Table of Contents
- How do I create a clusterable cache and use it in a policy?
The processing of requests can be synchronized across a cluster of multiple ND instances within a single deployment by creating a clusterable cache. A clusterable cache is memory that different processes can share.
For example, if you want the entire cluster to collectively only allow 10 requests per minute you can configure the NDs to share memory (a clusterable cache) which will hold information using counters. Then, when one node receives a message and increments a counter, it's the same counter that the other nodes are using.
A clusterable cache can be used with the following policies:
- Throughput Quota Policy - Allows you to monitor a policy by specifying a throughput limit (quota) and queue size.
- Bandwidth Quota Policy - Allows you to monitor a policy by specifying a bandwidth cap that can be specified as kilobytes or megabytes per second.
- WS-Security Policies - Using a clusterable cache, you can share nonces to prevent getting duplicate messages across all NDs in a cluster. Supported WS-Security policies include WS-Security Message, WS-Security Asymmetric Binding, and WS-Security Transport policies.
You configure a clusterable cache by updating a series of configuration categories in the Akana Administration Console > Configuration section. The process involves setting up the clusterable cache using the com.soa.grid property, and then configuring properties for the Throughput Quota Policy, Bandwidth Quota Policy, and WS-Security Policy to use the clustered cache.
How do I create a clusterable cache and use it in a policy?
Perform the following steps for each Network Director instance:
- Step 1: Create Clusterable Cache
- Step 2: Use Clustered Cache in Throughput Quota, Bandwidth Quota, and WS-Security Asymmetric Binding Policies
- Step 3: Restart Network Director Instances
- Step 4: Synchronize Clocks
Step 1: Create Clusterable Cache
- Select com.soa.grid.
- Modify/verify the following settings, as shown below. Note that the Multicast option (grid.network.config.enableMulticast) is currently not supported:
- Set grid.network.config.enableMulticast to false.
- Set grid.network.config.enableTcp to true.
- Set grid.network.config.localPort so that the value is a unique port number for the server.
Note: It's best to always specify a unique localPort value—for example, 48500—as well as specifying the IP address. Port 47500 and subsequent numbers could be reserved, so it's best not to use those.
- In grid.network.config.peerAddresses, specify the peer address of each Network Director instance in your deployment. Omit the peer address of the current Network Director instance in the list. The peerAddress value is a comma-separated list of all the other systems within the cluster (grid).
Important: the cluster must be local network only. All addresses must reside in the same subnet on a local network.
Valid addresses can be:
- Hostname. For example: machine101,machine102
- IP address with port number. For example: 220.127.116.11:48501,18.104.22.168:48502,22.214.171.124:48503
- IP address without port number. For example: 126.96.36.199,188.8.131.52,184.108.40.206
- Range of IP addresses without port number. For example: 220.127.116.11-7
Note: Do not include a comma (,) at the end of the list as the platform would interpret it as an empty member and return an exception.
- Click Apply Changes.
Step 2: Use Clustered Cache in Throughput Quota, Bandwidth Quota, and WS-Security Asymmetric Binding Policies
- Launch the Akana Administration Console and navigate to the Configuration tab.
- In the Configuration Categories section, select properties for the Throughput Quota Policy (com.soa.policyhandle.quota.throughput) or Bandwidth Quota Policy (com.soa.policy.handle.quota.bandwidth).
- Change use.clustered.cache to True and click Apply Changes.
Above: Throughput Quota Policy (com.soa.policyhandle.quota.throughput)
Above: Bandwidth Quota Policy (com.soa.policy.handle.quota.bandwidth)
- For the WS-Security Policy (com.soa.policy.handle.wssp.noncecache), ND will automatically use the clustered cache if you enabled the Grid and TCP options in Step 1, otherwise the local cache will be used.
Above: WS-Security Policy (com.soa.policy.handle.wssp.noncecache)
Step 3: Restart Network Director Instances
- Before you can successfully use the clusterable cache, you must restart all Network Director instances after completing the configuration category changes outlined in the previous steps.
Step 4: Synchronize Clocks
- The Policy Manager clusterable caching logic is based on the system clock. To ensure there is not a lag in one of the Network Director instances while processing requests, you must synchronize all system clocks so they match (i.e., by milliseconds and seconds).
Step 5: Test Policy Enforcement (Example)
Throughput Quota Policy
- Configure a Throughput Quota Policy to process 10 requests per minute.
- Virtualize a service.
- Attach the policy to the virtual service.
- Call the virtual service from the SOAP UI.
- Call the virtual service 10-20 times.
- In approximately one minute, ten requests will be success and the remainder will fail.
- Review the alerts. All requests failed past the 10 requests per minute will show a quota exceeded alert.
Bandwidth Quota Policy
- Configure a Bandwidth Quota Policy with 10 kbps for the request.
- Virtualize a service.
- Attach the policy to the virtual service.
- In SOAP UI, run a single test step (for example, Echo-Request1) in your case and make sure that the response is received successfully.
- Click on raw tab and note the content length of the request. Let's assume that content length is 1129 bytes.
- Configure your load test's limit as 60 seconds or whatever number of seconds you want.
- After running the test for 60 seconds let's say you see a count of 616 and an error count of 10. In this case the net count is 616-10=606. Ideally you would want to see the error count as 0. Some of these errors might be caused because a backend physical service took too long to return the response and then in this example the errors should have been included in the net count as ND allowed this request and passed it to physical service. Some of these errors might be caused because all of the ND threads were busy due to high load and in this example errors should not be included in net count.
- So let's assume that the net count is 606. Total number of bytes allowed by ND in 60 seconds-> 606*1129= 684174.
- Total number of bytes allowed per second 684174/60= 11402.9 bytes ~ 10kbps.
WS-Security Asymmetric Binding Policy
For an example illustration of how to use clusterable caching with this WS-Security policy type, refer to Use Case: Using a Clusterable Cache with WS-Security Asymmetric Binding Policy.