We had a problem with the Labkey system last Friday. The system halted in the morning around 8:27 AM. The log file reported the following error message:

Nov 9, 2012 8:26:37 AM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (150) are currently busy, waiting. Increase maxThreads (150) or check the servlet status

I touch the threadDumpRequest file and also did a pgstat.sh and than restarted Tomcat. Now the Full-text Search is giving the following error:

java.io.FileNotFoundException: /var/tmp/labkey_full_text_index_2/_bbk.prx (No such file or directory)
       at java.io.RandomAccessFile.open(Native Method)
       at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
       at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:70)
       at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97)

We think this was cause by the search engine not able to find several file in the system. This is similar to the problem we were having a month ago with the folders under the oconnor directory. I need some guidance on how to review the threatdump and fix this problem so it does not happen again.

Thank you

Daniel