• Sign In
  1. Messages

Error - A failure occurred during LabKey Server startup.

Installation Forum (Inactive)

View Message

view list print
Error - A failure occurred during LabKey Server startup. tvaisar  2017-06-14 11:21
Status: Closed
 
I have been compiling TPP 5.0 on our Linux cluster (running also Labkey Server 16.3) having lot of problem. But an unexpected problem occurred when I found out that the Labkey Server stopped working -
Below is the error I receive in web browser.
Permissions on logs folder are for tomcat:tomcat

drwxr-xr-x. 2 tomcat tomcat 36864 Jun 14 06:12 logs

Thanks for any advice.

Tomas

500: Unexpected server error
A failure occurred during LabKey Server startup.

java.lang.IllegalStateException: Error rolling labkey-errors.log file, likely a file permissions problem in CATALINA_HOME/logs
       at org.labkey.api.module.ModuleLoader.rollErrorLogFile(ModuleLoader.java:604)
       at org.labkey.api.module.ModuleLoader.doInit(ModuleLoader.java:262)
       at org.labkey.api.module.ModuleLoader.doInit(ModuleLoader.java:318)
       at org.labkey.api.module.ModuleLoader.init(ModuleLoader.java:240)
       at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
       at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
       at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
       at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4855)
       at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5549)
       at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
       at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
       at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
       at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
       at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:677)
       at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1912)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
 
 
Jon (LabKey DevOps) responded:  2017-06-14 11:29
Hello,

What about the actual files in the logs directory?

Can you do a ls -l and confirm they are also showing the Tomcat user as the owner and group?

Regards,

Jon
 
tvaisar responded:  2017-06-14 13:04
Jon,

it is a mixed bag. here is ls -l for labkey*
> ll labkey*
-rw-r--r--. 1 root root 68454 Dec 4 2016 labkey-action-stats.tsv
-rw-r--r--. 1 tomcat tomcat 66185 May 19 2016 labkey-action-stats.tsv.1
-rw-r--r--. 1 tomcat tomcat 66751 May 19 2016 labkey-action-stats.tsv.2
-rw-r--r--. 1 tomcat tomcat 66142 Feb 19 2016 labkey-action-stats.tsv.3
-rw-r--r--. 1 tomcat tomcat 4729287 Apr 19 07:21 labkey-errors.log
-rw-r--r--. 1 root root 1249552 Dec 2 2016 labkey-errors.log.1
-rw-r--r--. 1 root root 0 May 19 2016 labkey-errors.log.2
-rw-r--r--. 1 tomcat tomcat 524606 May 19 2016 labkey-errors.log.3
-rw-r--r--. 1 tomcat tomcat 11874 Jul 22 2015 labkey-file-copy.log
-rw-r--r--. 1 tomcat tomcat 2681970 Jun 8 02:00 labkey.log
-rw-r--r--. 1 tomcat tomcat 10485800 Feb 4 08:20 labkey.log.1
-rw-r--r--. 1 tomcat tomcat 0 Jul 15 2015 labkeyMemory.log
-rw-r--r--. 1 root root 2150238 Dec 4 2016 labkey-query-stats.tsv
-rw-r--r--. 1 root root 1869 May 19 2016 labkey-query-stats.tsv.1
-rw-r--r--. 1 tomcat tomcat 147816 May 19 2016 labkey-query-stats.tsv.2
-rw-r--r--. 1 tomcat tomcat 863582 May 19 2016 labkey-query-stats.tsv.3
 
Jon (LabKey DevOps) responded:  2017-06-14 13:05
Thanks for the confirmation.

Can you change the permissions on all those files to tomcat and try again? None of them should be set to root.

Regards,

Jon
 
tvaisar responded:  2017-06-14 13:09
Done but still get the same error from web access of labkey.

Should the file have wider write privileges?

Tomas

> ll labkey*
-rw-r--r--. 1 tomcat tomcat 68454 Dec 4 2016 labkey-action-stats.tsv
-rw-r--r--. 1 tomcat tomcat 66185 May 19 2016 labkey-action-stats.tsv.1
-rw-r--r--. 1 tomcat tomcat 66751 May 19 2016 labkey-action-stats.tsv.2
-rw-r--r--. 1 tomcat tomcat 66142 Feb 19 2016 labkey-action-stats.tsv.3
-rw-r--r--. 1 tomcat tomcat 4729287 Apr 19 07:21 labkey-errors.log
-rw-r--r--. 1 tomcat tomcat 1249552 Dec 2 2016 labkey-errors.log.1
-rw-r--r--. 1 tomcat tomcat 0 May 19 2016 labkey-errors.log.2
-rw-r--r--. 1 tomcat tomcat 524606 May 19 2016 labkey-errors.log.3
-rw-r--r--. 1 tomcat tomcat 11874 Jul 22 2015 labkey-file-copy.log
-rw-r--r--. 1 tomcat tomcat 2681970 Jun 8 02:00 labkey.log
-rw-r--r--. 1 tomcat tomcat 10485800 Feb 4 08:20 labkey.log.1
-rw-r--r--. 1 tomcat tomcat 0 Jul 15 2015 labkeyMemory.log
-rw-r--r--. 1 tomcat tomcat 2150238 Dec 4 2016 labkey-query-stats.tsv
-rw-r--r--. 1 tomcat tomcat 1869 May 19 2016 labkey-query-stats.tsv.1
-rw-r--r--. 1 tomcat tomcat 147816 May 19 2016 labkey-query-stats.tsv.2
-rw-r--r--. 1 tomcat tomcat 863582 May 19 2016 labkey-query-stats.tsv.3
 
Jon (LabKey DevOps) responded:  2017-06-14 13:14
Hello,

No, the permissions are fine. Did you also make sure to shut down Tomcat and then start it back up again as the tomcat user?

Regards,

Jon
 
tvaisar responded:  2017-06-14 13:34
Not specifically. I rebooted whole server.

Tomas
 
tvaisar responded:  2017-06-14 13:40
OK - I specifically shut down and restarted the tomcat service and now Labkey server works.
Thanks a lot. Thought that just rebooting computer would do it. Apparently not.

Appreciate your help Jon.

Tomas
 
Jon (LabKey DevOps) responded:  2017-06-14 14:00
Glad to hear things are working now Tomas!

Let us know if you have any additional questions.

Regards,

Jon