3 [![License](https://img.shields.io/github/license/voxpupuli/puppet-archive.svg)](https://github.com/voxpupuli/puppet-archive/blob/master/LICENSE)
4 [![Build Status](https://travis-ci.org/voxpupuli/puppet-archive.png?branch=master)](https://travis-ci.org/voxpupuli/puppet-archive)
5 [![Code Coverage](https://coveralls.io/repos/github/voxpupuli/puppet-archive/badge.svg?branch=master)](https://coveralls.io/github/voxpupuli/puppet-archive)
6 [![Puppet Forge](https://img.shields.io/puppetforge/v/puppet/archive.svg)](https://forge.puppetlabs.com/puppet/archive)
7 [![Puppet Forge - downloads](https://img.shields.io/puppetforge/dt/puppet/archive.svg)](https://forge.puppetlabs.com/puppet/archive)
8 [![Puppet Forge - endorsement](https://img.shields.io/puppetforge/e/puppet/archive.svg)](https://forge.puppetlabs.com/puppet/archive)
9 [![Puppet Forge - scores](https://img.shields.io/puppetforge/f/puppet/archive.svg)](https://forge.puppetlabs.com/puppet/archive)
10 [![Camptocamp compatible](https://img.shields.io/badge/camptocamp-compatible-orange.svg)](https://forge.puppet.com/camptocamp/archive)
12 #### Table of Contents
14 1. [Overview](#overview)
15 2. [Module Description](#module-description)
18 * [Example](#usage-example)
19 * [Puppet URL](#puppet-url)
20 * [File permission](#file-permission)
21 * [Network files](#network-files)
22 * [Extract customization](#extract-customization)
23 * [S3 Bucket](#s3-bucket)
24 * [Migrating from puppet-staging](#migrating-from-puppet-staging)
25 5. [Reference](#reference)
26 6. [Development](#development)
30 This module manages download, deployment, and cleanup of archive files.
34 This module uses types and providers to download and manage compress files,
35 with optional lifecycle functionality such as checksum, extraction, and
36 cleanup. The benefits over existing modules such as
37 [puppet-staging](https://github.com/voxpupuli/puppet-staging):
39 * Implemented via types and provider instead of exec resource.
40 * Follows 302 redirect and propagate download failure.
41 * Optional checksum verification of archive files.
42 * Automatic dependency to parent directory.
43 * Support Windows file extraction via 7zip.
44 * Able to cleanup archive files after extraction.
46 This module is compatible with [camptocamp/archive](https://forge.puppet.com/camptocamp/archive).
47 For this it provides compatibility shims.
51 The module requires 7zip for windows clients which is installed via `include
52 '::archive'`. On posix systems, curl is the default provider. The default
53 provider can be overwritten by configuring resource defaults in site.pp:
61 Users of the module is responsbile for archive package dependencies for
62 alternative providers and all extraction utilities such as tar, gunzip, bunzip:
65 if $::facts['osfamily'] != 'windows' {
76 require => Package['wget', 'bunzip'],
83 Archive module dependency is managed by the archive class. This is only
84 required for windows platform. By default 7zip is installed via chocolatey, but
85 can be adjusted to use the msi package instead:
89 seven_zip_name => '7-Zip 9.20 (x64 edition)',
90 seven_zip_source => 'C:/Windows/Temp/7z920-x64.msi',
91 seven_zip_provider => 'windows',
98 include '::archive' # NOTE: optional for posix platforms
100 archive { '/tmp/jta-1.1.jar':
103 extract_path => '/tmp',
104 source => 'http://central.maven.org/maven2/javax/transaction/jta/1.1/jta-1.1.jar',
105 checksum => '2ca09f0b36ca7d71b762e14ea2ff09d5eac57558',
106 checksum_type => 'sha1',
107 creates => '/tmp/javax',
111 archive { '/tmp/test100k.db':
112 source => 'ftp://ftp.otenet.gr/test100k.db',
113 username => 'speedtest',
114 password => 'speedtest',
120 Since march 2017, the Archive type also supports puppet URLs. Here is an example
125 archive { '/home/myuser/help':
126 source => 'puppet:///modules/profile/help.tar.gz',
128 extract_path => $homedir,
129 creates => "${homedir}/help" #directory inside tgz
135 When extracting files as non-root user, either ensure the target directory
136 exists with the appropriate permission (see [tomcat.pp](tests/tomcat.pp) for
137 full working example):
140 $dirname = 'apache-tomcat-9.0.0.M3'
141 $filename = "${dirname}.zip"
142 $install_path = "/opt/${dirname}"
144 file { $install_path:
152 path => "/tmp/${filename}",
153 source => 'http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.0.M3/bin/apache-tomcat-9.0.0.M3.zip',
154 checksum => 'f2aaf16f5e421b97513c502c03c117fab6569076',
155 checksum_type => 'sha1',
157 extract_path => '/opt',
158 creates => "${install_path}/bin",
162 require => File[$install_path],
166 or use an subscribing exec to chmod the directory afterwards:
169 $dirname = 'apache-tomcat-9.0.0.M3'
170 $filename = "${dirname}.zip"
171 $install_path = "/opt/${dirname}"
173 file { '/opt/tomcat':
175 target => $install_path
179 path => "/tmp/${filename}",
180 source => "http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.0.M3/bin/apache-tomcat-9.0.0.M3.zip",
181 checksum => 'f2aaf16f5e421b97513c502c03c117fab6569076',
182 checksum_type => 'sha1',
184 extract_path => '/opt',
185 creates => $install_path,
187 require => File[$install_path],
190 exec { 'tomcat permission':
191 command => "chown tomcat:tomcat $install_path",
193 subscribe => Archive[$filename],
199 For large binary files that needs to be extracted locally, instead of copying
200 the file from the network fileshare, simply set the file path to be the same as
201 the source and archive will use the network file location:
204 archive { '/nfs/repo/software.zip':
205 source => '/nfs/repo/software.zip'
207 extract_path => '/opt',
208 checksum_type => 'none', # typically unecessary
209 cleanup => false, # keep the file on the server
213 ### Extract Customization
215 The `extract_flags` or `extract_command` parameters can be used to override the
216 default extraction command/flag (defaults are specified in
217 [achive.rb](lib/puppet_x/bodeco/archive.rb)).
220 # tar striping directories:
221 archive { '/var/lib/kafka/kafka_2.10-0.8.2.1.tgz':
224 extract_command => 'tar xfz %s --strip-components=1',
225 extract_path => '/opt/kafka_2.10-0.8.2.1',
227 creates => '/opt/kafka_2.10-0.8.2.1/config',
230 # zip freshen existing files (zip -of %s instead of zip -o %s):
231 archive { '/var/lib/example.zip':
233 extract_path => '/opt',
234 extract_flags => '-of',
242 S3 support is implemented via the [AWS
243 CLI](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html).
244 By default this dependency is only installed for Linux VMs running on AWS, or
245 enabled via `aws_cli_install` option:
249 aws_cli_install => true,
252 # See AWS cli guide for credential and configuration settings:
253 # http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
254 file { '/root/.aws/credentials':
258 file { '/root/.aws/config':
263 archive { '/tmp/gravatar.png':
265 source => 's3://bodecoio/gravatar.png',
269 NOTE: Alternative s3 provider support can be implemented by overriding the
270 [s3_download method](lib/puppet/provider/archive/ruby.rb):
272 ### Download customizations
274 In some cases you may need custom flags for curl/wget/s3 which can be
275 supplied via `download_options`. Since this parameter is provider specific,
276 beware of the order of defaults:
278 * s3:// files accepts aws cli options
280 archive { '/tmp/gravatar.png':
282 source => 's3://bodecoio/gravatar.png',
283 download_options => ['--region', 'eu-central-1'],
286 * puppet `provider` override:
288 archive { '/tmp/jta-1.1.jar':
290 source => 'http://central.maven.org/maven2/javax/transaction/jta/1.1/jta-1.1.jar',
292 download_options => '--continue',
295 * Linux default provider is `curl`, and Windows default is `ruby` (no effect).
297 This option can also be applied globally to address issues for specific OS:
299 if $::facts['osfamily'] != 'RedHat' {
301 download_options => '--tlsv1',
306 ### Migrating from puppet-staging
308 It is recommended to use puppet-archive instead of puppet-staging.
309 Users wishing to migrate may find the following examples useful.
311 #### Simple example without extraction
317 path => '/tmp/staging',
320 staging::file { 'master.zip':
321 source => 'https://github.com/voxpupuli/puppet-archive/archive/master.zip',
328 archive { '/tmp/staging/master.zip':
329 source => 'https://github.com/voxpupuli/puppet-archive/archive/master.zip',
333 #### Example with zip file extraction
339 path => '/tmp/staging',
342 staging::file { 'master.zip':
343 source => 'https://github.com/voxpupuli/puppet-archive/archive/master.zip',
345 staging::extract { 'master.zip':
346 target => '/tmp/staging/master.zip',
347 creates => '/tmp/staging/puppet-archive-master',
354 archive { '/tmp/staging/master.zip':
355 source => 'https://github.com/voxpupuli/puppet-archive/archive/master.zip',
357 extract_path => '/tmp/staging',
358 creates => '/tmp/staging/puppet-archive-master',
367 * `archive`: install 7zip package (Windows only) and aws cli for s3 support.
368 * `archive::staging`: install package dependencies and creates staging directory
369 for backwards compatibility. Use the archive class instead if you do not need
370 the staging directory.
374 * `archive::artifactory`: archive wrapper for [JFrog Artifactory](http://www.jfrog.com/open-source/#os-arti)
376 * `archive::go`: archive wrapper for [GO Continuous Delivery](http://www.go.cd/)
378 * `archive::nexus`: archive wrapper for [Sonatype Nexus](http://www.sonatype.org/nexus/)
380 * `archive::download`: archive wrapper and compatibility shim for [camptocamp/archive](https://forge.puppet.com/camptocamp/archive).
381 This is considered private API, as it has to change with camptocamp/archive.
382 For this reason it will remain undocumented, and removed when no longer needed
383 . We suggest not using it directly. Instead please consider migrating to
384 archive itself where possible.
390 * `ensure`: whether archive file should be present/absent (default: present)
391 * `path`: namevar, archive file fully qualified file path.
392 * `filename`: archive file name (derived from path).
393 * `source`: archive file source, supports http|https|ftp|file|s3 uri.
394 * `username`: username to download source file.
395 * `password`: password to download source file.
396 * `allow_insecure`: Ignore HTTPS certificate errors (true|false). (default: false)
397 * `cookie`: archive file download cookie.
398 * `checksum_type`: archive file checksum type (none|md5|sha1|sha2|sha256|sha384|
399 sha512). (default: none)
400 * `checksum`: archive file checksum (match checksum_type)
401 * `checksum_url`: archive file checksum source (instead of specify checksum)
402 * `checksum_verify`: whether checksum will be verified (true|false). (default: true)
403 * `extract`: whether archive will be extracted after download (true|false).
405 * `extract_path`: target folder path to extract archive.
406 * `extract_command`: custom extraction command ('tar xvf example.tar.gz'), also
407 * `temp_dir`: specify an alternative temporary directory to use for file downloads, if unset the OS default is used
408 support sprintf format ('tar xvf %s') which will be processed with the
409 filename: sprintf('tar xvf %s', filename)
410 * `extract_flags`: custom extraction options, this replaces the default flags.
411 A string such as 'xvf' for a tar file would replace the default xf flag. A
412 hash is useful when custom flags are needed for different platforms. {'tar'
413 => 'xzf', '7z' => 'x -aot'}.
414 * `user`: extract command user (using this option will configure the archive
415 file permission to 0644 so the user can read the file).
416 * `group`: extract command group (using this option will configure the archive
417 file permission to 0644 so the user can read the file).
418 * `cleanup`: whether archive file will be removed after extraction (true|false).
420 * `creates`: if file/directory exists, will not download/extract archive.
421 * `proxy_server`: specify a proxy server, with port number if needed. ie:
422 `https://example.com:8080`.
423 * `proxy_type`: proxy server type (none|http|https|ftp)
425 #### Archive::Artifactory
427 * `path`: fully qualified filepath for the download the file or use
428 archive_path and only supply filename. (namevar).
429 * `ensure`: ensure the file is present/absent.
430 * `url`: artifactory download url filepath. NOTE: replaces server, port,
432 * `server`: artifactory server name (deprecated).
433 * `port`: artifactory server port (deprecated).
434 * `url_path`: artifactory file path
435 `http:://{server}:{port}/artifactory/{url_path}` (deprecated).
436 * `owner`: file owner (see archive params for defaults).
437 * `group`: file group (see archive params for defaults).
438 * `mode`: file mode (see archive params for defaults).
439 * `archive_path`: the parent directory of local filepath.
440 * `extract`: whether to extract the files (true/false).
441 * `creates`: the file created when the archive is extracted (true/false).
442 * `cleanup`: remove archive file after file extraction (true/false).
444 #### Archive::Artifactory Example
447 $dirname = 'gradle-1.0-milestone-4-20110723151213+0300'
448 $filename = "${dirname}-bin.zip"
450 archive::artifactory { $filename:
451 archive_path => '/tmp',
452 url => "http://repo.jfrog.org/artifactory/distributions/org/gradle/${filename}",
454 extract_path => '/opt',
455 creates => "/opt/${dirname}",
459 file { '/opt/gradle':
461 target => "/opt/${dirname}",
467 #### Archive::Nexus Example
470 archive::nexus { '/tmp/jtstand-ui-0.98.jar':
471 url => 'https://oss.sonatype.org',
472 gav => 'org.codehaus.jtstand:jtstand-ui:0.98',
473 repository => 'codehaus-releases',
481 We highly welcome new contributions to this module, especially those that
482 include documentation, and rspec tests ;) but will happily guide you through
483 the process, so, yes, please submit that pull request!
485 Note: If you are writing a dependent module that include specs in it, you will
486 need to set the puppetversion fact in your puppet-rspec tests. You can do that
487 by adding it to the default facts of your spec/spec_helper.rb:
490 RSpec.configure do |c|
491 c.default_facts = { :puppetversion => Puppet.version }