Overview

The Intermediary while starting may sometimes fail with the following error, after which it gets stuck in a re-start loop:

Using SSLEngineImpl.2020-12-30 01:00:21.996:WARN:oejs.AbstractConnector:qtp292938459-4244-acceptor-2@2424b76e-ServerConnector@5e5792a0{SSL,[ssl, http/1.1]}{0.0.0.0:13613}: 
java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:397)
    at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:748)

APPL0014: IO Exception: /cxmonitor/CXIntermediary/working/logs/AureaMonitorIntermediary_20201230_005942_831.ndx/_0.frq (Too many open files)
java.io.FileNotFoundException: /cxmonitor/CXIntermediary/working/logs/AureaMonitorIntermediary_20201230_005942_831.ndx/_0.frq (Too many open files)
    at com.actional.eventlog.EventLogFileResources.closeIndexWriter(EventLogFileResources.java:337)
    at com.actional.eventlog.EventLogFile.closeIndexWriter(EventLogFile.java:320)
    at com.actional.eventlog.FileIndexingThread.doRun(FileIndexingThread.java:98)
    at com.actional.eventlog.FileIndexingThread.run(FileIndexingThread.java:63)

Solution

This happens when Intermediary fails to initialize the listeners and the process gets stuck in a restart loop. Take the following steps to fix the issue:

1. Disable the SSL debug(remove -Djavax.net.debug from the profile). Restart the AI after disabling.
2. Monitor the file descriptor and resource usage. Refer to this article that explains what to gather when it happens.
3. You may consider tuning the open file limit(ulimit -n) and max user processes (ulimit -u). 

Testing

Once the applicable limits are increased, the Intermediary should start in one run.