TouchedExpiryPolicy/AccessedExpiryPolicy performance overhead

[ravaelamanov]

Hey there. We've been integrating Apache Ignite in embedded mode as a replacement for in-memory on-heap caches based on Caffeine. Without a second thought we set cache expiration policy to TouchedExpiryPolicy. Load testing resulted in almost 2x CPU consumption compared to Caffeine. Spent a couple of days diagnosing the problem and was able to find the cause only through profiling the application using asprof - it appears that TouchedExpiryPolicy/AccessedExpiryPolicy updates TTL on each access, which is logical, but also synchronizes it with other Ignite cluster nodes. This behavior is a bit implicit and hard to figure out, especially for those who come from the world of non-distributed caches.

My suggestion would be:

1. Explicitly warning users about potential intensive resource consumption in the javadoc of TouchedExpiryPolicy/AccessedExpiryPolicy
2. Adding a WARN log on node startup if TouchedExpiryPolicy/AccessedExpiryPolicy are chosen as expiry policy