128
128
as a starting point. Here is a basic outline for a Work Queue master:
131
q = work_queue_create(port,time);
131
q = work_queue_create(port);
134
134
t = work_queue_task_create(command);
147
147
First create a queue that is listening on a particular TCP port:
149
q = work_queue_create( port , time(0)+60 );
149
q = work_queue_create(port);
152
152
The master then creates tasks to submit to the queue.
154
154
what data is needed, and what data will be produced by the command.
155
155
Input data can be provided in the form of a file or a local memory buffer.
156
156
Output data can be provided in the form of a file or the standard output of the program.
157
In the example, we specify a command that takes a single input file
158
and produces a single output file:
160
t = work_queue_task_create("/usr/bin/gzip -9 < infile > outfile.gz");
161
work_queue_task_specify_input_file(t,"infile","infile");
162
work_queue_task_specify_output_file(t,"outfile.gz","outfile.gz");
157
It is also required to specify whether the data, input or output, need to be cached at
158
the worker site for later use. In the example, we specify a command that takes a
159
single input file, produces a single output file, and requires both files to be
162
t = work_queue_task_create("/usr/bin/gzip < infile > outfile.gz");
163
work_queue_task_specify_file(t,"infile","infile",WORK_QUEUE_INPUT,WORK_QUEUE_CACHE);
164
work_queue_task_specify_file(t,"outfile.gz","outfile.gz",WORK_QUEUE_OUTPUT,WORK_QUEUE_CACHE);
167
If a file does not need to be cached at the execution site to avoid wasteful
168
strorage, it can be specified so:
170
work_queue_task_specify_file(t,"outfile.gz","outfile.gz",WORK_QUEUE_OUTPUT,WORK_QUEUE_NOCACHE);
165
173
You can also run a program that is not necessarily installed at the
168
176
and the plain remote path. For example:
170
178
t = work_queue_task_create("./my_compress_program < infile > outfile.gz");
171
work_queue_task_specify_input_file(t,"/usr/local/bin/my_compress_program","my_compress_program");
172
work_queue_task_specify_input_file(t,"infile","infile");
173
work_queue_task_specify_output_file(t,"outfile.gz","outfile.gz");
179
work_queue_task_specify_file(t,"/usr/local/bin/my_compress_program","my_compress_program",WORK_QUEUE_INPUT,WORK_QUEUE_CACHE);
180
work_queue_task_specify_file(t,"infile","infile",WORK_QUEUE_INPUT,WORK_QUEUE_CACHE);
181
work_queue_task_specify_file(t,"outfile.gz","outfile.gz",WORK_QUEUE_OUTPUT,WORK_QUEUE_CACHE);
176
184
Once a task has been fully specified, it can be submitted to the queue:
228
236
to create local input files for each one. If the master already has the necessary input
229
237
data in memory, it can pass the data directly to the remote task with <tt>work_queue_task_specify_input_buf</tt>.
239
<li><b>Environment Variables.</b> If you have workers distributed across
240
multiple operating systems (such as Linux, Cygwin, Solaris) and/or architectures (such
241
as i686, x86_64) and have files specific to each of these systems, this feature
242
will help. The variables $OS and $ARCH are available for use in the specification of input
243
file names. Work Queue will automatically resolve these variables to the operating system
244
and architecture of each connected worker and transfer the input file corresponding
245
to the resolved file name. For example:
247
work_queue_task_specify_file(t,"infile.$OS.$ARCH","infile",WORK_QUEUE_INPUT,WORK_QUEUE_CACHE)
249
This will transfer infile.Linux.x86_64 to workers running on a Linux system with an
250
x86_64 architecture and infile.Cygwin.i686 to workers on Cygwin with an i686
231
253
<li><b>Statistics.</b> The queue tracks a fair number of statistics that count the number
232
254
of tasks, number of workers, number of failures, and so forth. Obtain this data with <tt>work_queue_get_stats</tt>
233
255
in order to make a progress bar or other user-visible information.
added
removed
doc/workqueue.html
