Bug Report: Payara 6.2024.6, @Clustered @Singleton not work as expected.

[cidylong]

Brief Summary

Payara 6.2024.6 community version, @clustered & @singleton annotations Timer EJB not work as expected.

Expected Outcome

@clustered and @singleton Time EJB should instanced on only one node in cluster.

Current Outcome

Actually, there are two Timer EJB instances created in a four members cluster.

Reproducer

Create a four members cluster, server instances as: DAS, AppServerLocal, AppServerRemote, MigAppServer on separated machines.

DAS and AppServerLocal were deployed on 10.0.1.103 (physical Linux Rocky 9);
AppServerRemote and MigAppServer were deployed on 10.0.1.104 (physical Linux Rocky 9);

Developed a set of applications (include service EJB and Web applications) deployed on different server instance.

Developed a clustered & singleton timer EJB to manage business process (have to be Cluster wise singleton to make suer only one EJB instance work at the moment.)

Code as:

@Clustered(callPostConstructOnAttach = true, callPreDestroyOnDetach = true, keyName = "clusteredSingletonTimer", lock  = DistributedLockType.LOCK)
@Singleton
@Startup
public class ClusteredSingletonTimer implements Serializable {
  private static final transient Logger logger = Logger.getLogger(ClusteredSingletonTimer.class.getName());

@Resource
transient TimerService ts;
@Resource
private transient SessionContext context;
@Lock
@Timeout
public void executor() {
    StatementPaymentScanner scanner = new StatementPaymentScanner();
    scanner.writeEvent();
    logger.info("EJB businessMethod()");
}
@PostConstruct
void init() {
    /*TimerConfig sTimerConfig = new TimerConfig();
    sTimerConfig.setInfo("DailyTimer_Clustered_singleton");
    ScheduleExpression schedule = new ScheduleExpression();
    schedule.hour("22").minute("47").second("13");*/
    /*context.getTimerService().createCalendarTimer(schedule, sTimerConfig);*/
    Timer timer = ts.createTimer(18000, 600000, "Statement-roller-Timer");/*15 minutes rolling*/
    logger.info("Cluster-wide Construction");
}
@PreDestroy
void destroy() {
    logger.info("Instance-only destruction");
}
}

Java class StatementPaymentScanner will create verify information and write to event log. The log information will include time out fired: server instance IP, Server name and Hazelcast Instance name to help us to find out where server instance is hosting that EJB timer instance.

Deploy it to MigAppServer

Restart domain and all server instances;

When time out, we will get result from event logger.

There are the result as:

From above screen shots you can see, there are two EJB timer instances existed in the cluster by different hosts and Hazelcast instances (one named: loving_pike and other one name: affectionate_yonath) and the two EJB timer fired at about the same time as well. (one: 2024-09-22 10:36:13 other one: 2024-09-22 10:36:13)

This will cause serious issue when the processes trying to access Database and Hazelcast DataGrid, Concurrency and data corruption will cause business application mess.

This issue is very easy to reproduce, please report it as high priority bug to be fixed or advise urgent work around.

Operating System

Rocky Linux 9

JDK Version

JDK 17.0.3

Payara Distribution

Payara Server Full Profile

[lprimak]

You have callPostConstructOnAttach = true which means @PostConstruct gets called on every Payara instance.
Then, you schedule the timer on every instance. This forces your timeouts to get called on every instance.

Also, you have transient on SessionContext and TimerService which isn't going to work
I would suggest you use @Schedule annotation instead,
and verify that your timers work properly without @Clustered singleton first, since they may require some configuration.
@Clustered doesn't affect how timer work at all, they are separate systems.

[cidylong]

@lprimak , Thank you for your advice. I will disable callPostConstructOnAttach and callPreDestroyOnDetach, try it again.

[fturizo]

@cidylong, based on your report, you intend to use a clustered singleton to configure an EJB timer to run only ONCE per cluster instead of on each node.

As per the official documentation of the clustered singleton feature on timer usage you don't need to use a clustered singleton and switch to stateless bean and configure your timer to be persistent using the @Schedule annotation as @lprimak suggested.

Since there are no bugs detected on this issue, we'll proceed to close it.

[cidylong]

@lprimak, @fturizo, Thank you for your advice. I tested @lprimak's advice, it is works as expected. Thank you again.

You are welcome to close this report.