Hi Martin,

I'm the author of DISCVR-Seq. Sorry for the trouble - installing/configuring the remote LabKey pipeline probably could use some extended/updated docs and DISCVR-Seq definitely would be helped by this. Two comments:

1) I can see how DISCR-Seq's admin page is misleading about the use of "-DsequencePipelineEnabled=true". I added this flag to support automated testing. The admin page where you uncovered this gives a way to run a series of integration tests that exercise the various tools and pipeline configurations on some small dummy files. I use this as a quick way to see whether a give server/pipeline is working right. It would be useful for you, but not strictly required for regular use.

2) I believe the crux of your issue is that we need to tell the server to run these tasks remotely. By default, all of the DISCVR-Seq's tasks will run locally (on the location 'webserver'). This is so the module works out of the box on a given server. Changing this is done using a file comparable to the ms2Config.xml described in the configureRemoteServer link above. You will need to create the file 'sequenceanalysisConfig.xml' in the /configs folder where your LK server is installed (i.e. /usr/local/labkey/configs/). Below is a link to example config files (username/password: cpas/cpas):

https://hedgehog.fhcrc.org/tor/stedi/trunk/externalModules/labModules/SequenceAnalysis/tools/pipeline_config/

I just added some comments to pipelineConfig_remote.xml and sequenceanalysisConfig.xml. Note: the location name is currently 'sequenceanalysis', which is what we call our remote servers. This can be anything, but it needs to be consistent in both the remote pipelineConfig.xml and sequenceanalysisConfig.xml files.

Also, I believe you will need to deploy sequenceanalysisConfig.xml on your webserver as well.

Let me know if you run into any issues.

-Ben

Thanks a lot Ben.
I've put the files at the right places and reconfigure them. It seems that the webserver is contacting the the ActiveMQ on the remote pipeline server. Now, I'm getting this error message when I start a job in the Sequence Analysis module:
'org.mule.umo.endpoint.MalformedEndpointException: The endpoint "StatusQueue" is malformed and cannot be parsed'

Any idea where I should look?

Hi Martin,

Happy to help. If I had to guess, I would think that's telling you something in your remoteMuleConfig.xml file is malformed. Did you end up adding/editing this file?

Another shot in the dark: the pipelineConfig.xml file I pointed you towards has this line:

<property name="muleConfig" value="/usr/local/labkey/configs/remoteMuleConfig.xml"/>

Which we use on our remote servers because we override some of the default configuration. I think you can comment out or remove this line in your pipelineConfig.xml file, in which case you do not need to provide a remoteMuleConfig.xml file at all. That might be best.

Thanks again Ben,
I had to comment out both this property to fix the "org.mule.umo.endpoint.MalformedEndpointException"

<property name="statusWriter">
<bean class="org.labkey.pipeline.mule.JMSStatusWriter"/>
</property>

This one was giving me some error too:
<property name="jobStore">
<bean class="org.labkey.pipeline.api.PipelineJobMarshaller"/>
</property>

Martin

hi martin,

interesting. these are general LabKey pipeline settings and really specific to DISCVR-Seq. would you mind posting your pipelineConfig.xml (or i could give you a separate place to upload)? There's no reason these should give issues, and they will be useful in order for the log to get updated correctly.

also, what LabKey version are you using?

-ben

Hi Ben,
I'm using LabKey 15.2.
I've attached the webserver pipelineConfig.xml file.

Hi Martin,

That all looks fine to me. I'm a little surprised you see errors w/ the statusWriter and jobStore blocks - this might be telling us something else is wrong.

Just to clarify - did commenting out these pieces allow your server to run? If not, do you have anything from the error log from the remote server? This is usually in /usr/local/labkey/log/ (or similar, depending on what you installed the remote server).

Would anyone from LK be able to comment on that?

Hi Ben,
When the jobstore block is uncommented, I get when submitting a job from sequence analysis module:
"500: Unexpected server error
Method supported only on web server"
and no jobs are submitted to ActiveMQ.

Uncommenting the statusWriter and commenting out the jobStore block gives me the MalformedEndpointException error.

Commenting out both allows jobs to be submitted to the ActiveMQ job.queue, but they are stuck there for now. I don't know if the problem is on the remote pipeline server or elsewhere. I've checked the logs on both servers and I can't find any error message.

Martin

Got it. Again hopefully someone from LK would be willing to jump in on the general pipeline debugging; however, one idea I have on why your messages are stuck in the queue is if the hostName of the remote server doesnt match the location that has been set of the sequence pipeline tasks. You should have added a sequenceanalysisConfig.xml file, which is here you tell the server to run specific tasks on the remote server. My example set the location to 'sequenceanalysis', i think. This file needs to be added to both the web server and remote server. The default location is the '/configs' subfolder where you LK server is installed. You can check whether this file is being read by looking at labkey.log after startup. If you search for 'sequenceanalysisConfig.xml', you should see a line saying that it was read. In theory, not having this set properly could result in jobs being sent to the queue, but no server being setup to pull them.

I think there may be a mixup of where the pipelineConfig.xml file is deployed. The one that Martin attached looks like it's intended for the remote server, not the web server.

On a remote server, there should be no issues with setting the statusWriter and jobStore in the XML file, but that should also be unnecessary - those are the default values on remote servers.

Thanks,
Josh

@jeckels
That pipelineConfig.xml is the one I'm using on the labkey webserver, not the remote. Please, let me know if you see anything that should be corrected. That server runs Win2008. You are right about statusWriter and jobStore blocks not being an issue on the remote server.

@ben
when looking if the sequenceanalysisConfig.xml file was loading on the remote server, I noticed I didn't have the sequenceAnalysis module there! After fixing that, it seems the server is taking jobs now, but I now need to fix a file transfer issue. The remote server tries to access the file I'm sending using the webserver path (c:/Program Files (x86)/labkey server/files/Test Sequence Analysis/5/@files/sequenceImport/SequenceImport20150922/CC136_S61_L001_R1_001.fastq.gz). The remote server is running linux (Centos).

Hi Martin,

2 things:

1) As josh says, you should not use the pipelineconfig.xml file on the webserver. Sorry if i wasnt clear. You should use this on the remote server though.

2) glad the sequenceanalysis module issue was an easy fix. i'm happy to provide links to a custom distribution that contains all these modules if it helps in the future, or for future upgrades.

3) if you look in the pipelineconfig.xml file, there is a commented out section in the bottom for file path mapping. this is how you would tell the remote server to translate a path. I think this might be case sensitive (not 100% certain), so be aware of that.

Note: you will need to put that pipelineConfig.xml file on your remote server in order to configure path mapping.

Thanks Ben and Josh,
I had pipelineConfig.xml on both servers as documentations on how to configure that file was on both the LabKey webserver set up and the remote pipeline server set up.

I still can't get the remote server to translate the path to the files sent by the LabKey webserver.
Here is the error I get:

********************************************************************************
Exception stack is:
1. /C:/Program Files (x86)/LabKey Server/files/Test sequences analysis/5/@files/sequenceImport/SequenceImport_20150922/CCC136_S61_L001_R1_001.fastq.gz (No such file or directory) (java.io.FileNotFoundException)
  java.io.FileInputStream:-2 (null)
2. java.io.FileNotFoundException: /C:/Program Files (x86)/LabKey Server/files/Test sequences analysis/5/@files/sequenceImport/SequenceImport_20150922/CCC136_S61_L001_R1_001.fastq.gz (No such file or directory) (java.lang.RuntimeException)
  org.labkey.api.util.Compress:123 (null)
3. Failed to invoke org.labkey.pipeline.mule.PipelineTaskRunner. Component that caused exception is: PipelineTaskRunnerUMO. Message payload is of type: ActiveMQTextMessage (org.mule.umo.ComponentException)
  org.mule.impl.DefaultLifecycleAdapter:199 (http://mule.mulesource.org/docs/apidocs/org/mule/umo/ComponentException.html)
********************************************************************************

I've mounted the LabKey webserver files directory (C:/Program Files (x86)/LabKey Server/files/) on linux (/usr/local/labkey/files/) using fstab.

I've configured the pathMap (in pipelineConfig.xml on the remote pipeline server):

        <property name="pathMapper">
            <bean class="org.labkey.api.pipeline.file.PathMapperImpl">
                <property name="localIgnoreCase" value="false"/>
                <property name="remoteIgnoreCase" value="true"/>
                <!--This is used to map paths between the webserver and remote server. The remote server requires access to the webserver file root.-->
                <!--If this is mounted in an alternate location, update this-->
                <property name="pathMap">
                    <map>
                        <entry key="/C:/Program Files (x86)/LabKey Server/files/" value="file:/usr/local/labkey/files/"/>
                    </map>
                </property>
            </bean>
        </property>


Thanks again.

Hi Martin,

Both the Windows and Linux paths should start with "file:/". For example:

<entry key="file:/C:/labkey" value="file:/media/sf_labkey"/>

Can you make the edit, restart the web server, and try again?

Thanks,
Josh

I tried and I get the same error message.

Do you know if the spaces in the path should be escaped or if they can cause problems?

Martin

I was able to fix that problem by replacing spaces by %20 (I noticed that in the labkey-error.log on the server).
Here is the pathMap I'm using:

<entry key="/C:/Program%20Files%20(x86)/LabKey%20Server" value="file:/usr/local/labkey"/>

It works now!

Thanks Josh and Ben

Martin