1
# Distributed under the OSI-approved BSD 3-Clause License. See accompanying
2
# file Copyright.txt or https://cmake.org/licensing for details.
4
#[=======================================================================[.rst:
12
Manage data files stored outside source tree
17
Use this module to unambiguously reference data files stored outside
18
the source tree and fetch them at build time from arbitrary local and
19
remote content-addressed locations. Functions provided by this module
20
recognize arguments with the syntax ``DATA{<name>}`` as references to
21
external data, replace them with full paths to local copies of those
22
data, and create build rules to fetch and update the local copies.
29
set(ExternalData_URL_TEMPLATES "file:///local/%(algo)/%(hash)"
30
"file:////host/share/%(algo)/%(hash)"
31
"http://data.org/%(algo)/%(hash)")
32
ExternalData_Add_Test(MyData
34
COMMAND MyExe DATA{MyInput.png}
36
ExternalData_Add_Target(MyData)
38
When test ``MyTest`` runs the ``DATA{MyInput.png}`` argument will be
39
replaced by the full path to a real instance of the data file
40
``MyInput.png`` on disk. If the source tree contains a content link
41
such as ``MyInput.png.md5`` then the ``MyData`` target creates a real
42
``MyInput.png`` in the build tree.
47
.. command:: ExternalData_Expand_Arguments
49
The ``ExternalData_Expand_Arguments`` function evaluates ``DATA{}``
50
references in its arguments and constructs a new list of arguments::
52
ExternalData_Expand_Arguments(
53
<target> # Name of data management target
54
<outVar> # Output variable
55
[args...] # Input arguments, DATA{} allowed
58
It replaces each ``DATA{}`` reference in an argument with the full path of
59
a real data file on disk that will exist after the ``<target>`` builds.
61
.. command:: ExternalData_Add_Test
63
The ``ExternalData_Add_Test`` function wraps around the CMake
64
:command:`add_test` command but supports ``DATA{}`` references in
67
ExternalData_Add_Test(
68
<target> # Name of data management target
69
... # Arguments of add_test(), DATA{} allowed
72
It passes its arguments through ``ExternalData_Expand_Arguments`` and then
73
invokes the :command:`add_test` command using the results.
75
.. command:: ExternalData_Add_Target
77
The ``ExternalData_Add_Target`` function creates a custom target to
78
manage local instances of data files stored externally::
80
ExternalData_Add_Target(
81
<target> # Name of data management target
84
It creates custom commands in the target as necessary to make data
85
files available for each ``DATA{}`` reference previously evaluated by
86
other functions provided by this module.
87
Data files may be fetched from one of the URL templates specified in
88
the ``ExternalData_URL_TEMPLATES`` variable, or may be found locally
89
in one of the paths specified in the ``ExternalData_OBJECT_STORES``
92
Typically only one target is needed to manage all external data within
93
a project. Call this function once at the end of configuration after
94
all data references have been processed.
99
The following variables configure behavior. They should be set before
100
calling any of the functions provided by this module.
102
.. variable:: ExternalData_BINARY_ROOT
104
The ``ExternalData_BINARY_ROOT`` variable may be set to the directory to
105
hold the real data files named by expanded ``DATA{}`` references. The
106
default is ``CMAKE_BINARY_DIR``. The directory layout will mirror that of
107
content links under ``ExternalData_SOURCE_ROOT``.
109
.. variable:: ExternalData_CUSTOM_SCRIPT_<key>
111
Specify a full path to a ``.cmake`` custom fetch script identified by
112
``<key>`` in entries of the ``ExternalData_URL_TEMPLATES`` list.
113
See `Custom Fetch Scripts`_.
115
.. variable:: ExternalData_LINK_CONTENT
117
The ``ExternalData_LINK_CONTENT`` variable may be set to the name of a
118
supported hash algorithm to enable automatic conversion of real data
119
files referenced by the ``DATA{}`` syntax into content links. For each
120
such ``<file>`` a content link named ``<file><ext>`` is created. The
121
original file is renamed to the form ``.ExternalData_<algo>_<hash>`` to
122
stage it for future transmission to one of the locations in the list
123
of URL templates (by means outside the scope of this module). The
124
data fetch rule created for the content link will use the staged
125
object if it cannot be found using any URL template.
127
.. variable:: ExternalData_NO_SYMLINKS
129
The real data files named by expanded ``DATA{}`` references may be made
130
available under ``ExternalData_BINARY_ROOT`` using symbolic links on
131
some platforms. The ``ExternalData_NO_SYMLINKS`` variable may be set
132
to disable use of symbolic links and enable use of copies instead.
134
.. variable:: ExternalData_OBJECT_STORES
136
The ``ExternalData_OBJECT_STORES`` variable may be set to a list of local
137
directories that store objects using the layout ``<dir>/%(algo)/%(hash)``.
138
These directories will be searched first for a needed object. If the
139
object is not available in any store then it will be fetched remotely
140
using the URL templates and added to the first local store listed. If
141
no stores are specified the default is a location inside the build
144
.. variable:: ExternalData_SERIES_PARSE
145
ExternalData_SERIES_PARSE_PREFIX
146
ExternalData_SERIES_PARSE_NUMBER
147
ExternalData_SERIES_PARSE_SUFFIX
148
ExternalData_SERIES_MATCH
150
See `Referencing File Series`_.
152
.. variable:: ExternalData_SOURCE_ROOT
154
The ``ExternalData_SOURCE_ROOT`` variable may be set to the highest source
155
directory containing any path named by a ``DATA{}`` reference. The
156
default is ``CMAKE_SOURCE_DIR``. ``ExternalData_SOURCE_ROOT`` and
157
``CMAKE_SOURCE_DIR`` must refer to directories within a single source
158
distribution (e.g. they come together in one tarball).
160
.. variable:: ExternalData_TIMEOUT_ABSOLUTE
162
The ``ExternalData_TIMEOUT_ABSOLUTE`` variable sets the download
163
absolute timeout, in seconds, with a default of ``300`` seconds.
164
Set to ``0`` to disable enforcement.
166
.. variable:: ExternalData_TIMEOUT_INACTIVITY
168
The ``ExternalData_TIMEOUT_INACTIVITY`` variable sets the download
169
inactivity timeout, in seconds, with a default of ``60`` seconds.
170
Set to ``0`` to disable enforcement.
172
.. variable:: ExternalData_URL_ALGO_<algo>_<key>
174
Specify a custom URL component to be substituted for URL template
175
placeholders of the form ``%(algo:<key>)``, where ``<key>`` is a
176
valid C identifier, when fetching an object referenced via hash
177
algorithm ``<algo>``. If not defined, the default URL component
178
is just ``<algo>`` for any ``<key>``.
180
.. variable:: ExternalData_URL_TEMPLATES
182
The ``ExternalData_URL_TEMPLATES`` may be set to provide a list of
183
of URL templates using the placeholders ``%(algo)`` and ``%(hash)``
184
in each template. Data fetch rules try each URL template in order
185
by substituting the hash algorithm name for ``%(algo)`` and the hash
186
value for ``%(hash)``. Alternatively one may use ``%(algo:<key>)``
187
with ``ExternalData_URL_ALGO_<algo>_<key>`` variables to gain more
188
flexibility in remote URLs.
193
Referencing Single Files
194
""""""""""""""""""""""""
196
The ``DATA{}`` syntax is literal and the ``<name>`` is a full or relative path
197
within the source tree. The source tree must contain either a real
198
data file at ``<name>`` or a "content link" at ``<name><ext>`` containing a
199
hash of the real file using a hash algorithm corresponding to ``<ext>``.
200
For example, the argument ``DATA{img.png}`` may be satisfied by either a
201
real ``img.png`` file in the current source directory or a ``img.png.md5``
202
file containing its MD5 sum.
204
Multiple content links of the same name with different hash algorithms
205
are supported (e.g. ``img.png.sha256`` and ``img.png.sha1``) so long as
206
they all correspond to the same real file. This allows objects to be
207
fetched from sources indexed by different hash algorithms.
209
Referencing File Series
210
"""""""""""""""""""""""
212
The ``DATA{}`` syntax can be told to fetch a file series using the form
213
``DATA{<name>,:}``, where the ``:`` is literal. If the source tree
214
contains a group of files or content links named like a series then a
215
reference to one member adds rules to fetch all of them. Although all
216
members of a series are fetched, only the file originally named by the
217
``DATA{}`` argument is substituted for it. The default configuration
218
recognizes file series names ending with ``#.ext``, ``_#.ext``, ``.#.ext``,
219
or ``-#.ext`` where ``#`` is a sequence of decimal digits and ``.ext`` is
220
any single extension. Configure it with a regex that parses ``<number>``
221
and ``<suffix>`` parts from the end of ``<name>``::
223
ExternalData_SERIES_PARSE = regex of the form (<number>)(<suffix>)$
225
For more complicated cases set::
227
ExternalData_SERIES_PARSE = regex with at least two () groups
228
ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any
229
ExternalData_SERIES_PARSE_NUMBER = <number> regex group number
230
ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number
232
Configure series number matching with a regex that matches the
233
``<number>`` part of series members named ``<prefix><number><suffix>``::
235
ExternalData_SERIES_MATCH = regex matching <number> in all series members
237
Note that the ``<suffix>`` of a series does not include a hash-algorithm
240
Referencing Associated Files
241
""""""""""""""""""""""""""""
243
The ``DATA{}`` syntax can alternatively match files associated with the
244
named file and contained in the same directory. Associated files may
245
be specified by options using the syntax
246
``DATA{<name>,<opt1>,<opt2>,...}``. Each option may specify one file by
247
name or specify a regular expression to match file names using the
248
syntax ``REGEX:<regex>``. For example, the arguments::
250
DATA{MyData/MyInput.mhd,MyInput.img} # File pair
251
DATA{MyData/MyFrames00.png,REGEX:MyFrames[0-9]+\\.png} # Series
253
will pass ``MyInput.mha`` and ``MyFrames00.png`` on the command line but
254
ensure that the associated files are present next to them.
256
Referencing Directories
257
"""""""""""""""""""""""
259
The ``DATA{}`` syntax may reference a directory using a trailing slash and
260
a list of associated files. The form ``DATA{<name>/,<opt1>,<opt2>,...}``
261
adds rules to fetch any files in the directory that match one of the
262
associated file options. For example, the argument
263
``DATA{MyDataDir/,REGEX:.*}`` will pass the full path to a ``MyDataDir``
264
directory on the command line and ensure that the directory contains
265
files corresponding to every file or content link in the ``MyDataDir``
266
source directory. In order to match associated files in subdirectories,
267
specify a ``RECURSE:`` option, e.g. ``DATA{MyDataDir/,RECURSE:,REGEX:.*}``.
272
The following hash algorithms are supported::
274
%(algo) <ext> Description
275
------- ----- -----------
276
MD5 .md5 Message-Digest Algorithm 5, RFC 1321
277
SHA1 .sha1 US Secure Hash Algorithm 1, RFC 3174
278
SHA224 .sha224 US Secure Hash Algorithms, RFC 4634
279
SHA256 .sha256 US Secure Hash Algorithms, RFC 4634
280
SHA384 .sha384 US Secure Hash Algorithms, RFC 4634
281
SHA512 .sha512 US Secure Hash Algorithms, RFC 4634
282
SHA3_224 .sha3-224 Keccak SHA-3
283
SHA3_256 .sha3-256 Keccak SHA-3
284
SHA3_384 .sha3-384 Keccak SHA-3
285
SHA3_512 .sha3-512 Keccak SHA-3
287
Note that the hashes are used only for unique data identification and
288
download verification.
290
.. _`ExternalData Custom Fetch Scripts`:
295
When a data file must be fetched from one of the URL templates
296
specified in the ``ExternalData_URL_TEMPLATES`` variable, it is
297
normally downloaded using the :command:`file(DOWNLOAD)` command.
298
One may specify usage of a custom fetch script by using a URL
299
template of the form ``ExternalDataCustomScript://<key>/<loc>``.
300
The ``<key>`` must be a C identifier, and the ``<loc>`` must
301
contain the ``%(algo)`` and ``%(hash)`` placeholders.
302
A variable corresponding to the key, ``ExternalData_CUSTOM_SCRIPT_<key>``,
303
must be set to the full path to a ``.cmake`` script file. The script
304
will be included to perform the actual fetch, and provided with
305
the following variables:
307
.. variable:: ExternalData_CUSTOM_LOCATION
309
When a custom fetch script is loaded, this variable is set to the
310
location part of the URL, which will contain the substituted hash
311
algorithm name and content hash value.
313
.. variable:: ExternalData_CUSTOM_FILE
315
When a custom fetch script is loaded, this variable is set to the
316
full path to a file in which the script must store the fetched
317
content. The name of the file is unspecified and should not be
318
interpreted in any way.
320
The custom fetch script is expected to store fetched content in the
321
file or set a variable:
323
.. variable:: ExternalData_CUSTOM_ERROR
325
When a custom fetch script fails to fetch the requested content,
326
it must set this variable to a short one-line message describing
327
the reason for failure.
329
#]=======================================================================]
331
function(ExternalData_add_test target)
332
# Expand all arguments as a single string to preserve escaped semicolons.
333
ExternalData_expand_arguments("${target}" testArgs "${ARGN}")
334
add_test(${testArgs})
337
function(ExternalData_add_target target)
338
if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES)
340
"Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!")
342
if(NOT ExternalData_OBJECT_STORES)
343
set(ExternalData_OBJECT_STORES ${CMAKE_BINARY_DIR}/ExternalData/Objects)
345
set(_ExternalData_CONFIG_CODE "")
347
# Store custom script configuration.
348
foreach(url_template IN LISTS ExternalData_URL_TEMPLATES)
349
if("${url_template}" MATCHES "^ExternalDataCustomScript://([^/]*)/(.*)$")
350
set(key "${CMAKE_MATCH_1}")
351
if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$")
352
if(ExternalData_CUSTOM_SCRIPT_${key})
353
if(IS_ABSOLUTE "${ExternalData_CUSTOM_SCRIPT_${key}}")
354
string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n"
355
"set(ExternalData_CUSTOM_SCRIPT_${key} \"${ExternalData_CUSTOM_SCRIPT_${key}}\")")
358
"No ExternalData_CUSTOM_SCRIPT_${key} is not set to a full path:\n"
359
" ${ExternalData_CUSTOM_SCRIPT_${key}}")
363
"No ExternalData_CUSTOM_SCRIPT_${key} is set for URL template:\n"
368
"Bad ExternalDataCustomScript key '${key}' in URL template:\n"
370
"The key must be a valid C identifier.")
374
# Store custom algorithm name to URL component maps.
375
if("${url_template}" MATCHES "%\\(algo:([^)]*)\\)")
376
set(key "${CMAKE_MATCH_1}")
377
if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$")
378
string(REPLACE "|" ";" _algos "${_ExternalData_REGEX_ALGO}")
379
foreach(algo ${_algos})
380
if(DEFINED ExternalData_URL_ALGO_${algo}_${key})
381
string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n"
382
"set(ExternalData_URL_ALGO_${algo}_${key} \"${ExternalData_URL_ALGO_${algo}_${key}}\")")
387
"Bad %(algo:${key}) in URL template:\n"
389
"The transform name must be a valid C identifier.")
394
# Store configuration for use by build-time script.
395
set(config ${CMAKE_CURRENT_BINARY_DIR}/${target}_config.cmake)
396
configure_file(${_ExternalData_SELF_DIR}/ExternalData_config.cmake.in ${config} @ONLY)
400
# Set a "_ExternalData_FILE_${file}" variable for each output file to avoid
401
# duplicate entries within this target. Set a directory property of the same
402
# name to avoid repeating custom commands with the same output in this directory.
403
# Repeating custom commands with the same output across directories or across
404
# targets in the same directory may be a race, but this is likely okay because
405
# we use atomic replacement of output files.
407
# Use local data first to prefer real files over content links.
409
# Custom commands to copy or link local data.
410
get_property(data_local GLOBAL PROPERTY _ExternalData_${target}_LOCAL)
411
foreach(entry IN LISTS data_local)
412
string(REPLACE "|" ";" tuple "${entry}")
413
list(GET tuple 0 file)
414
list(GET tuple 1 name)
415
if(NOT DEFINED "_ExternalData_FILE_${file}")
416
set("_ExternalData_FILE_${file}" 1)
417
get_property(added DIRECTORY PROPERTY "_ExternalData_FILE_${file}")
419
set_property(DIRECTORY PROPERTY "_ExternalData_FILE_${file}" 1)
421
COMMENT "Generating ${file}"
423
COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR}
424
-Dfile=${file} -Dname=${name}
425
-DExternalData_ACTION=local
426
-DExternalData_CONFIG=${config}
427
-P ${_ExternalData_SELF}
428
MAIN_DEPENDENCY "${name}"
431
list(APPEND files "${file}")
435
# Custom commands to fetch remote data.
436
get_property(data_fetch GLOBAL PROPERTY _ExternalData_${target}_FETCH)
437
foreach(entry IN LISTS data_fetch)
438
string(REPLACE "|" ";" tuple "${entry}")
439
list(GET tuple 0 file)
440
list(GET tuple 1 name)
441
list(GET tuple 2 exts)
442
string(REPLACE "+" ";" exts_list "${exts}")
443
list(GET exts_list 0 first_ext)
444
set(stamp "-hash-stamp")
445
if(NOT DEFINED "_ExternalData_FILE_${file}")
446
set("_ExternalData_FILE_${file}" 1)
447
get_property(added DIRECTORY PROPERTY "_ExternalData_FILE_${file}")
449
set_property(DIRECTORY PROPERTY "_ExternalData_FILE_${file}" 1)
451
# Users care about the data file, so hide the hash/timestamp file.
452
COMMENT "Generating ${file}"
453
# The hash/timestamp file is the output from the build perspective.
454
# List the real file as a second output in case it is a broken link.
455
# The files must be listed in this order so CMake can hide from the
456
# make tool that a symlink target may not be newer than the input.
457
OUTPUT "${file}${stamp}" "${file}"
458
# Run the data fetch/update script.
459
COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR}
460
-Dfile=${file} -Dname=${name} -Dexts=${exts}
461
-DExternalData_ACTION=fetch
462
-DExternalData_CONFIG=${config}
463
-P ${_ExternalData_SELF}
464
# Update whenever the object hash changes.
465
MAIN_DEPENDENCY "${name}${first_ext}"
468
list(APPEND files "${file}${stamp}")
472
# Custom target to drive all update commands.
473
add_custom_target(${target} ALL DEPENDS ${files})
476
function(ExternalData_expand_arguments target outArgsVar)
477
# Replace DATA{} references with real arguments.
478
set(data_regex "DATA{([^;{}\r\n]*)}")
479
set(other_regex "([^D]|D[^A]|DA[^T]|DAT[^A]|DATA[^{])+|.")
481
# This list expansion un-escapes semicolons in list element values so we
482
# must re-escape them below anywhere a new list expansion will occur.
483
foreach(arg IN LISTS ARGN)
484
if("x${arg}" MATCHES "${data_regex}")
485
# Re-escape in-value semicolons before expansion in foreach below.
486
string(REPLACE ";" "\\;" tmp "${arg}")
487
# Split argument into DATA{}-pieces and other pieces.
488
string(REGEX MATCHALL "${data_regex}|${other_regex}" pieces "${tmp}")
489
# Compose output argument with DATA{}-pieces replaced.
491
foreach(piece IN LISTS pieces)
492
if("x${piece}" MATCHES "^x${data_regex}$")
493
# Replace this DATA{}-piece with a file path.
494
_ExternalData_arg("${target}" "${piece}" "${CMAKE_MATCH_1}" file)
495
set(outArg "${outArg}${file}")
497
# No replacement needed for this piece.
498
set(outArg "${outArg}${piece}")
502
# No replacements needed in this argument.
505
# Re-escape in-value semicolons in resulting list.
506
string(REPLACE ";" "\\;" outArg "${outArg}")
507
list(APPEND outArgs "${outArg}")
509
set("${outArgsVar}" "${outArgs}" PARENT_SCOPE)
512
#-----------------------------------------------------------------------------
513
# Private helper interface
515
set(_ExternalData_REGEX_ALGO "MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512")
516
set(_ExternalData_REGEX_EXT "md5|sha1|sha224|sha256|sha384|sha512|sha3-224|sha3-256|sha3-384|sha3-512")
517
set(_ExternalData_SELF "${CMAKE_CURRENT_LIST_FILE}")
518
get_filename_component(_ExternalData_SELF_DIR "${_ExternalData_SELF}" PATH)
520
function(_ExternalData_compute_hash var_hash algo file)
521
if("${algo}" MATCHES "^${_ExternalData_REGEX_ALGO}$")
522
file("${algo}" "${file}" hash)
523
set("${var_hash}" "${hash}" PARENT_SCOPE)
525
message(FATAL_ERROR "Hash algorithm ${algo} unimplemented.")
529
function(_ExternalData_random var)
530
string(RANDOM LENGTH 6 random)
531
set("${var}" "${random}" PARENT_SCOPE)
534
function(_ExternalData_exact_regex regex_var string)
535
string(REGEX REPLACE "([][+.*()^])" "\\\\\\1" regex "${string}")
536
set("${regex_var}" "${regex}" PARENT_SCOPE)
539
function(_ExternalData_atomic_write file content)
540
_ExternalData_random(random)
541
set(tmp "${file}.tmp${random}")
542
file(WRITE "${tmp}" "${content}")
543
file(RENAME "${tmp}" "${file}")
546
function(_ExternalData_link_content name var_ext)
547
if("${ExternalData_LINK_CONTENT}" MATCHES "^(${_ExternalData_REGEX_ALGO})$")
548
set(algo "${ExternalData_LINK_CONTENT}")
551
"Unknown hash algorithm specified by ExternalData_LINK_CONTENT:\n"
552
" ${ExternalData_LINK_CONTENT}")
554
_ExternalData_compute_hash(hash "${algo}" "${name}")
555
get_filename_component(dir "${name}" PATH)
556
set(staged "${dir}/.ExternalData_${algo}_${hash}")
557
string(TOLOWER ".${algo}" ext)
558
_ExternalData_atomic_write("${name}${ext}" "${hash}\n")
559
file(RENAME "${name}" "${staged}")
560
set("${var_ext}" "${ext}" PARENT_SCOPE)
562
file(RELATIVE_PATH relname "${ExternalData_SOURCE_ROOT}" "${name}${ext}")
563
message(STATUS "Linked ${relname} to ExternalData ${algo}/${hash}")
566
function(_ExternalData_arg target arg options var_file)
567
# Separate data path from the options.
568
string(REPLACE "," ";" options "${options}")
569
list(GET options 0 data)
570
list(REMOVE_AT options 0)
572
# Interpret trailing slashes as directories.
573
set(data_is_directory 0)
574
if("x${data}" MATCHES "^x(.*)([/\\])$")
575
set(data_is_directory 1)
576
set(data "${CMAKE_MATCH_1}")
579
# Convert to full path.
580
if(IS_ABSOLUTE "${data}")
581
set(absdata "${data}")
583
set(absdata "${CMAKE_CURRENT_SOURCE_DIR}/${data}")
585
get_filename_component(absdata "${absdata}" ABSOLUTE)
587
# Convert to relative path under the source tree.
588
if(NOT ExternalData_SOURCE_ROOT)
589
set(ExternalData_SOURCE_ROOT "${CMAKE_SOURCE_DIR}")
591
set(top_src "${ExternalData_SOURCE_ROOT}")
592
file(RELATIVE_PATH reldata "${top_src}" "${absdata}")
593
if(IS_ABSOLUTE "${reldata}" OR "${reldata}" MATCHES "^\\.\\./")
594
message(FATAL_ERROR "Data file referenced by argument\n"
596
"does not lie under the top-level source directory\n"
599
if(data_is_directory AND NOT IS_DIRECTORY "${top_src}/${reldata}")
600
message(FATAL_ERROR "Data directory referenced by argument\n"
602
"corresponds to source tree path\n"
604
"that does not exist as a directory!")
606
if(NOT ExternalData_BINARY_ROOT)
607
set(ExternalData_BINARY_ROOT "${CMAKE_BINARY_DIR}")
609
set(top_bin "${ExternalData_BINARY_ROOT}")
611
# Handle in-source builds gracefully.
612
if("${top_src}" STREQUAL "${top_bin}")
613
if(ExternalData_LINK_CONTENT)
614
message(WARNING "ExternalData_LINK_CONTENT cannot be used in-source")
615
set(ExternalData_LINK_CONTENT 0)
620
set(external "") # Entries external to the source tree.
621
set(internal "") # Entries internal to the source tree.
622
set(have_original ${data_is_directory})
623
set(have_original_as_dir 0)
626
set(series_option "")
627
set(recurse_option "")
628
set(associated_files "")
629
set(associated_regex "")
630
foreach(opt ${options})
631
# Regular expression to match associated files.
632
if("x${opt}" MATCHES "^xREGEX:([^:/]+)$")
633
list(APPEND associated_regex "${CMAKE_MATCH_1}")
634
elseif(opt STREQUAL ":")
635
# Activate series matching.
636
set(series_option "${opt}")
637
elseif(opt STREQUAL "RECURSE:")
638
# Activate recursive matching in directories.
639
set(recurse_option "${opt}")
640
elseif("x${opt}" MATCHES "^[^][:/*?]+$")
641
# Specific associated file.
642
list(APPEND associated_files "${opt}")
644
message(FATAL_ERROR "Unknown option \"${opt}\" in argument\n"
650
if(data_is_directory)
651
message(FATAL_ERROR "Series option \"${series_option}\" not allowed with directories.")
653
if(associated_files OR associated_regex)
654
message(FATAL_ERROR "Series option \"${series_option}\" not allowed with associated files.")
657
message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.")
659
# Load a whole file series.
660
_ExternalData_arg_series()
661
elseif(data_is_directory)
662
if(associated_files OR associated_regex)
663
# Load listed/matching associated files in the directory.
664
_ExternalData_arg_associated()
666
message(FATAL_ERROR "Data directory referenced by argument\n"
668
"must list associated files.")
672
message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.")
674
# Load the named data file.
675
_ExternalData_arg_single()
676
if(associated_files OR associated_regex)
677
# Load listed/matching associated files.
678
_ExternalData_arg_associated()
682
if(NOT have_original)
683
if(have_original_as_dir)
684
set(msg_kind FATAL_ERROR)
685
set(msg "that is directory instead of a file!")
687
set(msg_kind AUTHOR_WARNING)
688
set(msg "that does not exist as a file (with or without an extension)!")
690
message(${msg_kind} "Data file referenced by argument\n"
692
"corresponds to source tree path\n"
698
# Make the series available in the build tree.
699
set_property(GLOBAL APPEND PROPERTY
700
_ExternalData_${target}_FETCH "${external}")
701
set_property(GLOBAL APPEND PROPERTY
702
_ExternalData_${target}_LOCAL "${internal}")
703
set("${var_file}" "${top_bin}/${reldata}" PARENT_SCOPE)
705
# The whole series is in the source tree.
706
set("${var_file}" "${top_src}/${reldata}" PARENT_SCOPE)
710
macro(_ExternalData_arg_associated)
711
# Associated files lie in the same directory.
712
if(data_is_directory)
713
set(reldir "${reldata}")
715
get_filename_component(reldir "${reldata}" PATH)
718
set(reldir "${reldir}/")
720
_ExternalData_exact_regex(reldir_regex "${reldir}")
722
set(glob GLOB_RECURSE)
723
set(reldir_regex "${reldir_regex}(.+/)?")
728
# Find files named explicitly.
729
foreach(file ${associated_files})
730
_ExternalData_exact_regex(file_regex "${file}")
731
_ExternalData_arg_find_files(${glob} "${reldir}${file}"
732
"${reldir_regex}${file_regex}")
735
# Find files matching the given regular expressions.
738
foreach(regex ${associated_regex})
739
set(all "${all}${sep}${reldir_regex}${regex}")
742
_ExternalData_arg_find_files(${glob} "${reldir}" "${all}")
745
macro(_ExternalData_arg_single)
746
# Match only the named data by itself.
747
_ExternalData_exact_regex(data_regex "${reldata}")
748
_ExternalData_arg_find_files(GLOB "${reldata}" "${data_regex}")
751
macro(_ExternalData_arg_series)
752
# Configure series parsing and matching.
753
set(series_parse_prefix "")
754
set(series_parse_number "\\1")
755
set(series_parse_suffix "\\2")
756
if(ExternalData_SERIES_PARSE)
757
if(ExternalData_SERIES_PARSE_NUMBER AND ExternalData_SERIES_PARSE_SUFFIX)
758
if(ExternalData_SERIES_PARSE_PREFIX)
759
set(series_parse_prefix "\\${ExternalData_SERIES_PARSE_PREFIX}")
761
set(series_parse_number "\\${ExternalData_SERIES_PARSE_NUMBER}")
762
set(series_parse_suffix "\\${ExternalData_SERIES_PARSE_SUFFIX}")
763
elseif(NOT "x${ExternalData_SERIES_PARSE}" MATCHES "^x\\([^()]*\\)\\([^()]*\\)\\$$")
765
"ExternalData_SERIES_PARSE is set to\n"
766
" ${ExternalData_SERIES_PARSE}\n"
767
"which is not of the form\n"
768
" (<number>)(<suffix>)$\n"
769
"Fix the regular expression or set variables\n"
770
" ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any\n"
771
" ExternalData_SERIES_PARSE_NUMBER = <number> regex group number\n"
772
" ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number\n"
775
set(series_parse "${ExternalData_SERIES_PARSE}")
777
set(series_parse "([0-9]*)(\\.[^./]*)$")
779
if(ExternalData_SERIES_MATCH)
780
set(series_match "${ExternalData_SERIES_MATCH}")
782
set(series_match "[_.-]?[0-9]*")
785
# Parse the base, number, and extension components of the series.
786
string(REGEX REPLACE "${series_parse}" "${series_parse_prefix};${series_parse_number};${series_parse_suffix}" tuple "${reldata}")
787
list(LENGTH tuple len)
788
if(NOT "${len}" EQUAL 3)
789
message(FATAL_ERROR "Data file referenced by argument\n"
791
"corresponds to path\n"
793
"that does not match regular expression\n"
796
list(GET tuple 0 relbase)
797
list(GET tuple 2 ext)
799
# Glob files that might match the series.
800
# Then match base, number, and extension.
801
_ExternalData_exact_regex(series_base "${relbase}")
802
_ExternalData_exact_regex(series_ext "${ext}")
803
_ExternalData_arg_find_files(GLOB "${relbase}*${ext}"
804
"${series_base}${series_match}${series_ext}")
807
function(_ExternalData_arg_find_files glob pattern regex)
809
cmake_policy(SET CMP0009 NEW)
810
file(${glob} globbed RELATIVE "${top_src}" "${top_src}/${pattern}*")
812
set(externals_count -1)
813
foreach(entry IN LISTS globbed)
814
if("x${entry}" MATCHES "^x(.*)(\\.(${_ExternalData_REGEX_EXT}))$")
815
set(relname "${CMAKE_MATCH_1}")
816
set(alg "${CMAKE_MATCH_2}")
818
set(relname "${entry}")
821
if("x${relname}" MATCHES "^x${regex}$" # matches
822
AND NOT "x${relname}" MATCHES "(^x|/)\\.ExternalData_" # not staged obj
824
if(IS_DIRECTORY "${top_src}/${entry}")
825
if("${relname}" STREQUAL "${reldata}")
826
set(have_original_as_dir 1)
829
set(name "${top_src}/${relname}")
830
set(file "${top_bin}/${relname}")
832
if(NOT "${external_${externals_count}_file_name}" STREQUAL "${file}|${name}")
833
# 0 works around negative number bug in older CMake
834
math(EXPR externals_count "0${externals_count} + 1")
835
set(external_${externals_count}_file_name "${file}|${name}")
837
list(APPEND external_${externals_count}_algs "${alg}")
838
elseif(ExternalData_LINK_CONTENT)
839
_ExternalData_link_content("${name}" alg)
840
list(APPEND external "${file}|${name}|${alg}")
842
list(APPEND internal "${file}|${name}")
844
if("${relname}" STREQUAL "${reldata}")
850
if(${externals_count} GREATER -1)
851
foreach(ii RANGE ${externals_count})
852
string(REPLACE ";" "+" algs_delim "${external_${ii}_algs}")
853
list(APPEND external "${external_${ii}_file_name}|${algs_delim}")
854
unset(external_${ii}_algs)
855
unset(external_${ii}_file_name)
858
set(external "${external}" PARENT_SCOPE)
859
set(internal "${internal}" PARENT_SCOPE)
860
set(have_original "${have_original}" PARENT_SCOPE)
861
set(have_original_as_dir "${have_original_as_dir}" PARENT_SCOPE)
864
#-----------------------------------------------------------------------------
865
# Private script mode interface
867
if(CMAKE_GENERATOR OR NOT ExternalData_ACTION)
871
if(ExternalData_CONFIG)
872
include(${ExternalData_CONFIG})
874
if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES)
876
"Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!")
879
function(_ExternalData_link_or_copy src dst)
880
# Create a temporary file first.
881
get_filename_component(dst_dir "${dst}" PATH)
882
file(MAKE_DIRECTORY "${dst_dir}")
883
_ExternalData_random(random)
884
set(tmp "${dst}.tmp${random}")
885
if(UNIX AND NOT ExternalData_NO_SYMLINKS)
886
# Create a symbolic link.
889
# Use relative path if files are close enough.
890
file(RELATIVE_PATH relsrc "${relative_top}" "${src}")
891
file(RELATIVE_PATH relfile "${relative_top}" "${dst}")
892
if(NOT IS_ABSOLUTE "${relsrc}" AND NOT "${relsrc}" MATCHES "^\\.\\./" AND
893
NOT IS_ABSOLUTE "${reldst}" AND NOT "${reldst}" MATCHES "^\\.\\./")
894
file(RELATIVE_PATH tgt "${dst_dir}" "${src}")
897
execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink "${tgt}" "${tmp}" RESULT_VARIABLE result)
900
execute_process(COMMAND "${CMAKE_COMMAND}" -E copy "${src}" "${tmp}" RESULT_VARIABLE result)
903
file(REMOVE "${tmp}")
904
message(FATAL_ERROR "Failed to create\n ${tmp}\nfrom\n ${obj}")
907
# Atomically create/replace the real destination.
908
file(RENAME "${tmp}" "${dst}")
911
function(_ExternalData_download_file url file err_var msg_var)
914
math(EXPR retry "${retry} - 1")
915
if(ExternalData_TIMEOUT_INACTIVITY)
916
set(inactivity_timeout INACTIVITY_TIMEOUT ${ExternalData_TIMEOUT_INACTIVITY})
917
elseif(NOT "${ExternalData_TIMEOUT_INACTIVITY}" EQUAL 0)
918
set(inactivity_timeout INACTIVITY_TIMEOUT 60)
920
set(inactivity_timeout "")
922
if(ExternalData_TIMEOUT_ABSOLUTE)
923
set(absolute_timeout TIMEOUT ${ExternalData_TIMEOUT_ABSOLUTE})
924
elseif(NOT "${ExternalData_TIMEOUT_ABSOLUTE}" EQUAL 0)
925
set(absolute_timeout TIMEOUT 300)
927
set(absolute_timeout "")
929
file(DOWNLOAD "${url}" "${file}" STATUS status LOG log ${inactivity_timeout} ${absolute_timeout} SHOW_PROGRESS)
930
list(GET status 0 err)
931
list(GET status 1 msg)
933
if("${msg}" MATCHES "HTTP response code said error" AND
934
"${log}" MATCHES "error: 503")
935
set(msg "temporarily unavailable")
937
elseif("${log}" MATCHES "\nHTTP[^\n]* 503")
939
set(msg "temporarily unavailable")
941
if(NOT err OR NOT "${msg}" MATCHES "partial|timeout|temporarily")
944
message(STATUS "[download terminated: ${msg}, retries left: ${retry}]")
947
set("${err_var}" "${err}" PARENT_SCOPE)
948
set("${msg_var}" "${msg}" PARENT_SCOPE)
951
function(_ExternalData_custom_fetch key loc file err_var msg_var)
952
if(NOT ExternalData_CUSTOM_SCRIPT_${key})
954
set(msg "No ExternalData_CUSTOM_SCRIPT_${key} set!")
955
elseif(NOT EXISTS "${ExternalData_CUSTOM_SCRIPT_${key}}")
957
set(msg "No '${ExternalData_CUSTOM_SCRIPT_${key}}' exists!")
959
set(ExternalData_CUSTOM_LOCATION "${loc}")
960
set(ExternalData_CUSTOM_FILE "${file}")
961
unset(ExternalData_CUSTOM_ERROR)
962
include("${ExternalData_CUSTOM_SCRIPT_${key}}")
963
if(DEFINED ExternalData_CUSTOM_ERROR)
965
set(msg "${ExternalData_CUSTOM_ERROR}")
971
set("${err_var}" "${err}" PARENT_SCOPE)
972
set("${msg_var}" "${msg}" PARENT_SCOPE)
975
function(_ExternalData_get_from_object_store hash algo var_obj var_success)
976
# Search all object stores for an existing object.
977
foreach(dir ${ExternalData_OBJECT_STORES})
978
set(obj "${dir}/${algo}/${hash}")
980
message(STATUS "Found object: \"${obj}\"")
981
set("${var_obj}" "${obj}" PARENT_SCOPE)
982
set("${var_success}" 1 PARENT_SCOPE)
988
function(_ExternalData_download_object name hash algo var_obj var_success var_errorMsg)
989
# Search all object stores for an existing object.
991
foreach(dir ${ExternalData_OBJECT_STORES})
992
set(obj "${dir}/${algo}/${hash}")
994
message(STATUS "Found object: \"${obj}\"")
995
set("${var_obj}" "${obj}" PARENT_SCOPE)
996
set("${var_success}" "${success}" PARENT_SCOPE)
1001
# Download object to the first store.
1002
list(GET ExternalData_OBJECT_STORES 0 store)
1003
set(obj "${store}/${algo}/${hash}")
1005
_ExternalData_random(random)
1006
set(tmp "${obj}.tmp${random}")
1009
foreach(url_template IN LISTS ExternalData_URL_TEMPLATES)
1010
string(REPLACE "%(hash)" "${hash}" url_tmp "${url_template}")
1011
string(REPLACE "%(algo)" "${algo}" url "${url_tmp}")
1012
if(url MATCHES "^(.*)%\\(algo:([A-Za-z_][A-Za-z0-9_]*)\\)(.*)$")
1013
set(lhs "${CMAKE_MATCH_1}")
1014
set(key "${CMAKE_MATCH_2}")
1015
set(rhs "${CMAKE_MATCH_3}")
1016
if(DEFINED ExternalData_URL_ALGO_${algo}_${key})
1017
set(url "${lhs}${ExternalData_URL_ALGO_${algo}_${key}}${rhs}")
1019
set(url "${lhs}${algo}${rhs}")
1022
string(REGEX REPLACE "((https?|ftp)://)([^@]+@)?(.*)" "\\1\\4" secured_url "${url}")
1023
message(STATUS "Fetching \"${secured_url}\"")
1024
if(url MATCHES "^ExternalDataCustomScript://([A-Za-z_][A-Za-z0-9_]*)/(.*)$")
1025
_ExternalData_custom_fetch("${CMAKE_MATCH_1}" "${CMAKE_MATCH_2}" "${tmp}" err errMsg)
1027
_ExternalData_download_file("${url}" "${tmp}" err errMsg)
1029
set(tried "${tried}\n ${url}")
1031
set(tried "${tried} (${errMsg})")
1033
# Verify downloaded object.
1034
_ExternalData_compute_hash(dl_hash "${algo}" "${tmp}")
1035
if("${dl_hash}" STREQUAL "${hash}")
1039
set(tried "${tried} (wrong hash ${algo}=${dl_hash})")
1040
if("$ENV{ExternalData_DEBUG_DOWNLOAD}" MATCHES ".")
1041
file(RENAME "${tmp}" "${store}/${algo}/${dl_hash}")
1045
file(REMOVE "${tmp}")
1048
get_filename_component(dir "${name}" PATH)
1049
set(staged "${dir}/.ExternalData_${algo}_${hash}")
1053
file(RENAME "${tmp}" "${obj}")
1054
message(STATUS "Downloaded object: \"${obj}\"")
1055
elseif(EXISTS "${staged}")
1056
set(obj "${staged}")
1057
message(STATUS "Staged object: \"${obj}\"")
1060
set(tried "\n (No ExternalData_URL_TEMPLATES given)")
1063
set("${var_errorMsg}" "Object ${algo}=${hash} not found at:${tried}" PARENT_SCOPE)
1066
set("${var_obj}" "${obj}" PARENT_SCOPE)
1067
set("${var_success}" "${success}" PARENT_SCOPE)
1070
if("${ExternalData_ACTION}" STREQUAL "fetch")
1071
foreach(v ExternalData_OBJECT_STORES file name exts)
1072
if(NOT DEFINED "${v}")
1073
message(FATAL_ERROR "No \"-D${v}=\" value provided!")
1077
string(REPLACE "+" ";" exts_list "${exts}")
1084
foreach(ext ${exts_list})
1085
file(READ "${name}${ext}" hash)
1086
string(STRIP "${hash}" hash)
1088
if("${ext}" MATCHES "^\\.(${_ExternalData_REGEX_EXT})$")
1089
string(TOUPPER "${CMAKE_MATCH_1}" algo)
1090
string(REPLACE "-" "_" algo "${algo}")
1092
message(FATAL_ERROR "Unknown hash algorithm extension \"${ext}\"")
1095
list(APPEND hash_list ${hash})
1096
list(APPEND algo_list ${algo})
1099
list(LENGTH exts_list num_extensions)
1100
math(EXPR exts_range "${num_extensions} - 1")
1101
foreach(ii RANGE 0 ${exts_range})
1102
list(GET hash_list ${ii} hash)
1103
list(GET algo_list ${ii} algo)
1104
_ExternalData_get_from_object_store("${hash}" "${algo}" obj succeeded)
1110
foreach(ii RANGE 0 ${exts_range})
1111
list(GET hash_list ${ii} hash)
1112
list(GET algo_list ${ii} algo)
1113
_ExternalData_download_object("${name}" "${hash}" "${algo}"
1114
obj succeeded algoErrorMsg)
1115
set(errorMsg "${errorMsg}\n${algoErrorMsg}")
1122
message(FATAL_ERROR "${errorMsg}")
1124
# Check if file already corresponds to the object.
1125
set(stamp "-hash-stamp")
1126
set(file_up_to_date 0)
1127
if(EXISTS "${file}" AND EXISTS "${file}${stamp}")
1128
file(READ "${file}${stamp}" f_hash)
1129
string(STRIP "${f_hash}" f_hash)
1130
if("${f_hash}" STREQUAL "${hash}")
1131
set(file_up_to_date 1)
1136
# Touch the file to convince the build system it is up to date.
1137
execute_process(COMMAND "${CMAKE_COMMAND}" -E touch "${file}")
1139
_ExternalData_link_or_copy("${obj}" "${file}")
1142
# Atomically update the hash/timestamp file to record the object referenced.
1143
_ExternalData_atomic_write("${file}${stamp}" "${hash}\n")
1144
elseif("${ExternalData_ACTION}" STREQUAL "local")
1145
foreach(v file name)
1146
if(NOT DEFINED "${v}")
1147
message(FATAL_ERROR "No \"-D${v}=\" value provided!")
1150
_ExternalData_link_or_copy("${name}" "${file}")
1152
message(FATAL_ERROR "Unknown ExternalData_ACTION=[${ExternalData_ACTION}]")
added
removed
CMake/ExternalData.cmake
