• Sign In

Pipeline Setup

Study Forum (Inactive)

View Message

view list print
Pipeline Setup rick.watts  2010-12-01 13:15
Status: Closed
 
There are many pages of documentation relating to pipelines but I still don't seem to be able to work it out...

I have a study that contains a dataset that I imported from a TSV file. What I'd like to do is set up a pipeline so that this file could be imported every time it changes.

I don't see in my study where I can setup and configure the pipeline. I can include the file in 'manage files' but I don't seem to be able to do anything else. In the demo study you can add a pipeline web part. But my study doesn't have a pipeline option in the list of parts I can add.

Can anyone give me a list of the steps I need to go through?

Thanks.

Rick.
 
 
eknelson responded:  2010-12-01 15:59
Hi Rick,

Sorry, the docs have gotten out of sync with the product here.

To access the pipeline, you can go to:
Admin (top right corner of every page) -> Go to Module -> More Modules -> Pipeline.

For quick access, you can embed a link to this location in a wiki.

In the future, the pipeline UI will be accessible more easily (through a web part). See:
https://www.labkey.org/issues/home/Developer/issues/details.view?issueId=10491

Hope this helps.

-- Elizabeth
 
adam responded:  2010-12-01 18:31
Once you have the pipeline root configured you have a couple options for regularly importing your dataset:
  1. Create a .dataset file with a file pattern that matches your dataset TSV file and place it in the same directory (one way to get a sample .dataset file is to export your study to a study archive). Use "manage files" to navigate and click on the .dataset file. Click "import data" to load the dataset. This is a fine approach if you want to import the file manually.
  2. Create a study archive (exporting an existing study as individual files with only datasets selected is a good way to create a template study archive). Remove datasets you don't want to re-import (remove the TSV files and their entries in the manifest and metadata files). You can manually re-import this study archive by navigating to the study.xml file and importing it, which is nearly identical to the above. However, with this approach you can also use the "manage reloading" page to configure LabKey to automatically reload your data when it changes. For more information about study export/import/reload, see this page: https://www.labkey.org/wiki/home/Documentation/page.view?name=importExportStudy
Adam