Release Notes

v0.90.4

Bug Fixes

• Update the create script to include V3 starter flavors on the cloud provider. Newer V3 starter flavors run on newer hardware and could potentially reduce costs by half and ideal for running jobs or tests as compared to standard flavors.

v0.90.3

Bug Fixes

• Fix JaCoCo exclude patterns format. JJB 6.x does not process the job config when double quotes are used and returns the following error.

  Error: requests.exceptions.HTTPError: 500 Server Error: Server Error for url: https://jenkins.opendaylight.org/releng/job/distribution-merge-managed-argon/config.xml

  Using YAML multiline introduces spaces between lines that causes JJB updates to fail when the job exists on Jenkins. Therefore, as workaround use double-quoted string with newline escape.

Other Notes

• Bump GitHub actions/setup-python from v4 to v5

v0.90.2

Bug Fixes

• Address several linting errors in scripts, mostly unquoted variables

• Remove Openstack SDK version pinning from shell scripts. This will now be handled by the other modules as a declared dependency. The previous version pin was extremely old.

v0.90.1

Bug Fixes

• Fix JaCoCo exclude patterns format. JJB 6.x does not process the job config when double quotes are used and returns the following error.

  Error: requests.exceptions.HTTPError: 500 Server Error: Server Error for url: https://jenkins.opendaylight.org/releng/job/distribution-merge-managed-argon/config.xml

• Update tox.ini to workaround an undeclared setuptools dependency.

• Rename view name ‘All’. JJB 6x has introduced a regression with view name that causing view updates to fail on sandbox cleanup.

v0.90.0

Prelude

Jenkins Job Builder 6.0.0 is released and breaks backward compatibility with 5.1.0. The breaking changes with previous 5.1.0 release:

• Jobs are now expanded the same way as job templates.

• Macros without parameters are now expanded the same way as macros with parameters.

• Tags !include-raw: and !include-raw-escape: should now be used the same way in jobs and macros without parameters as they are used in job templates and macros with parameters.

Known Issues

• This breaks backward compatibility with older version of JJB therefore care must be taken during upgrade on the ci-man repositories.

Upgrade Notes

• Upgrade Jenkins-job-builder to 6.0.0 as the default version.

  Ref:

  https://storyboard.openstack.org/#!/story/2010963 https://storyboard.openstack.org/#!/story/2010588

v0.89.9

Prelude

A new version of JJB 6.x is released which breaks backword compatibility.

Bug Fixes

• Pin global-jjb to 5.1.0 until the codebase is moved to JJB 6x.

  https://groups.google.com/g/jenkins-job-builder/c/miYGnWgYZYM

• Add Java update alternatives step to allow users to use a different Java version for their Sonarqube builds.

v0.89.8

Bug Fixes

• Update lf-infra-sonar macro to allow the sonar builder to take a specific JDK version.

v0.89.7

New Features

• Allow SBOM reports to be generated for docker-maven stage jobs.

Bug Fixes

• Add regex with HCL file extensions to trigger merge jobs when HCL file format is found.

v0.89.6

New Features

• Use Sigstore Cosign to sign docker images and push signature. In order to enable, the project needs to create their keypair and credentials in Jenkins for cosign-password (keypair password) and cosign-private-key.

v0.89.5

Bug Fixes

• Verify and merge release jobs need to be trigger for only the branch in question. Currently, all active branches verify jobs are triggering causing verify conflicts when projects are releasing more than one branch simultaniously.

v0.89.4

Bug Fixes

• A network error while getting the instance type in job-cost.sh will no longer mark a build as failed/unstable.

v0.89.3

Deprecation Notes

• Sonarcloud upload will stop working with java11 on November 15th 2023 Update OpenJDK to release 17 for sonar jobs Also revert from JDK13 -> JDK11 in a couple of places where the previous release/update had unintended scope

v0.89.2

Deprecation Notes

• Sonarcloud upload will stop working with java11 on November 15th 2023 Update OpenJDK to release 13 for sonar jobs

v0.89.0

Known Issues

• Supports urllib3 to newer DEFAULT_TIMEOUT, therefore unpin urllib3<2.0.0 Ref: https://review.opendev.org/c/jjb/python-jenkins/+/882757

  Fixes Jenkins version 2.387.1 or earlier can return ‘all’ as view name when requested is ‘All’.

Upgrade Notes

• lf-jacoco-report publisher now uses the exclude-pattern variable to set the exclusion pattern. Any job implementing this publisher needs to define this variable.

  Additionally, all lf-maven job templates that utilize the lf-jacoco-report publisher define the exclusion pattern with the variable jacoco-exclude-pattern. This can be overwritten by the calling project definition/template in order to set a custom exclusion pattern.

• Upgrade Jenkins-job-builder to 5.0.4

v0.88.6

Bug Fixes

• Fix Gradle job template where verify is not working/triggering correctly

v0.88.5

Bug Fixes

• Rename view type to lower case.

  Jenkins version 2.387.1 or earlier can return ‘all’ as view name when requested is ‘All’. Add workaround for that.

v0.88.3

Deprecation Notes

• Remove WhiteSource job templates. WhiteSource tool for code scanning is no longer in use for LF projects.

v0.88.2

Bug Fixes

• Gradle build job needs to run docker login step to allow docker operations and access to Nexus3.

• Gerrit release verify and merge jobs need to have a {stream} variable in their name to allow the creation of the same job under different branches without conflicting.

v0.88.1

Bug Fixes

• Pass target file where the config file should be created. New version of packer accepts only .json or .hcl extension filenames.

v0.88.0

Prelude

As of packer version 1.7.0 HCL2 is the preferred way to write Packer templates. HCL2 preserves existing workflows while leveraging HCL2’s advanced features like variable interpolation and configuration composability.

Upgrade Notes

• Upgrade packer version to v1.9.1. JSON format templates are deprecated and no longer work with packer version > 1.9.x. Project specific templates require to be upgraded to HCL2 format.

Deprecation Notes

• Support for ‘.json’ templates will be removed from common-packer in subsequent releases. Therefore, the jobs are expected to work with older templates.

Bug Fixes

• Rewrite packer jobs to work with HCL2 format.

v0.87.1

Bug Fixes

• Add a JDK configuration step in the Gradle based jobs.

v0.87.0

New Features

• Add initial lf-gradle file for gradle based builds.

v0.86.9

Bug Fixes

• Pin urllib3 to <2.0.0 for the JJB cleanup

  The latest version of module breaks compatibility with python-jenkins.

  Error:

  ValueError: Timeout value connect was <object object at
  0x7fe57a4948a0>, but it must be an int, float or None.
  

  Reference:

  Launchpad#2018567 <https://bugs.launchpad.net/python-jenkins/+bug/2018567>

v0.86.7

Bug Fixes

• Pin urllib3~=1.26.15 in pypi distribution jobs

v0.86.6

Bug Fixes

• Pin urllib3~=1.26.15 in jjb-deploy job

v0.86.4

Bug Fixes

• Pin urllib3 to <2.0.0 for the RTD jobs

  The latest version of module breaks compatibility with RTDv3 jobs during tox install and run.

  Error:

  ValueError: Timeout value connect was <object object at
  0x7fe57a4948a0>, but it must be an int, float or None.
  

  Reference:

  Launchpad#2018567 <https://bugs.launchpad.net/python-jenkins/+bug/2018567>

v0.86.3

Bug Fixes

• Pin urllib3 to <2.0.0 for the verify jobs

  The latest version of module breaks compatibility with python-jenkins.

  Error:

  ValueError: Timeout value connect was <object object at
  0x7fe57a4948a0>, but it must be an int, float or None.
  

  Reference:

  Launchpad#2018567 <https://bugs.launchpad.net/python-jenkins/+bug/2018567>

v0.86.2

Bug Fixes

• Pin urllib3 to <2.0.0

  The latest version of module breaks compatibility with python-jenkins.

  Error:

  ValueError: Timeout value connect was <object object at
  0x7fe57a4948a0>, but it must be an int, float or None.
  

  Reference:

  Launchpad#2018567 <https://bugs.launchpad.net/python-jenkins/+bug/2018567>

v0.86.1

Bug Fixes

• Use the GERRIT_REFSPEC while pushing code and tags separately.

v0.86.0

Prelude

JJB 5x treats recursive parameters as an error.

JJB’s 5.0.1 has a bug that return an error. TypeError: ‘NoneType’ object is not a mapping

JJB’s YAML parser is re-written in JJB 5.x release for a fine-tuned control over YAML parsing, YAML objects and parameter expansion logic.

Known Issues

• This breaks the JJB test on the existing ci-man repositores when the macro is null value.

• This breaks backward compatibility with older version of JJB therefore care must be taken during upgrade.

Upgrade Notes

• Upgrade Jenkins-job-builder to 5.0.2 which has the fix for the issue.

  Ref: https://review.opendev.org/c/jjb/jenkins-job-builder/+/880589

• Upgrade Jenkins-job-builder to 5.0.1 as the default version.

  Ref: https://review.opendev.org/c/jjb/jenkins-job-builder/+/871965

Bug Fixes

• Remove recursive parameters set as defaults for packagecloud jobs

• Use the python3 module option instead of calling pip directly. This fixes the issue when pip is not available the PATH.

v0.85.0

Deprecation Notes

• Remove daily cron on maven-stage and maven-docker-stage jobs. Cron triggers stay configurable.

Bug Fixes

• Fix condition before pushing the object.

• Address the problem where the tag is not pushed to the mainline branch therefore causing the tag missing in the git history.

  To fix this check commit count between the HEAD and origin/${GERRIT_BRANCH} before the fetch and merge operation. This is done to ensure that the tag lands on the target branch. If the branch has already moved forward from the tagging point, then a spur commit is created for the tag.

• Un-pin tox version from 3.27.1 and remove tox-pyenv. Testing has demonstrated that tox-pyenv is no longer required to obtain correct Python runtime versions when running tests. Also, removed Python 3.8 from the VENV setup where it was being specifically requested.

  Due to unpinning of the tox version, tox.ini configuration files may need modifying to reflect a change in configuration syntax; where whitelist_externals needs to be replaced with allowlist_externals.

v0.84.0

New Features

• Introduce Docker Snyk CLI scanner jobs. These jobs can be triggered to download the latest version of Snyk’s CLI scanner and trigger a scan for Docker based repos. These jobs produce a report which is published into Snyk’s dashboard. These reports are fetched and reflected back into the LFX Security tool.

• Introduce Go Snyk CLI scanner jobs. These jobs can be triggered to download the latest version of Snyk’s CLI scanner and trigger a scan for Go based repos. These jobs produce a report which is published into Snyk’s dashboard. These reports are fetched and reflected back into the LFX Security tool.

• Introduce Maven Snyk CLI scanner jobs. These jobs can be triggered to download the latest version of Snyk’s CLI scanner and trigger a scan for Maven based repos. These jobs produce a report which is published into Snyk’s dashboard. These reports are fetched and reflected back into the LFX Security tool.

• Introduce Python Snyk CLI scanner jobs. These jobs can be triggered to download the latest version of Snyk’s CLI scanner and trigger a scan for Python based repos. These jobs produce a report which is published into Snyk’s dashboard. These reports are fetched and reflected back into the LFX Security tool.

Bug Fixes

• Add SNYK_CLI_OPTIONS parameter which can be used to pass additional Snyk CLI options as per https://docs.snyk.io/snyk-cli/cli-reference.

• The path and command for update-alternatives/alternatives was not being set correctly between CentOS7/8 and was incorrect under all tested ubuntu versions. It did not seem to cause jobs to break, so was perhaps not being detected in all cases.

• The latest (2.42.0.01) clm-maven-plugin introduced an error in our environment.

  Failed to execute goal com.sonatype.clm:clm-maven-plugin:2.42.0-01:index (default-cli) on project babel: Failed to invoke Maven build. Maven execution failed, exit code: 1 -> [Help 1]

  This fix will pin the clm-maven-plugin to the previous version (2.41.0-02)

v0.83.5

Bug Fixes

• The Nexus IQ script was outputting the wrong variable during execution, which could be misleading in the job console logs. Also added a warning message if the NEXUS_TARGET_BUILD variable has not been set/populated. Reports were not receiving module dependencies so the script has been amended to download them into the target directory, as per the Nexus IQ documentation.

v0.83.4

Bug Fixes

• Pin setuptools to the latest version before v66.0.0 to avoid PEP-440 non conforming versions errors while packages we use fix their version string formats to be compatible. To maintain congruency we are pinning setuptools to 65.7.0 in all places under global-jjb.

v0.83.3

Bug Fixes

• setuptools 66.0.0 enforces specific version number check. A lot of plugins will fails on this, and generates the following error pkg_resources.extern.packaging.version.InvalidVersion

  To temporarily fix this, we pin setuptools to <66.0.0

  Further information https://github.com/pypa/setuptools/issues/3772#issuecomment-1384342813 https://setuptools.pypa.io/en/latest/history.html#v66-0-0

v0.83.2

Bug Fixes

• Pin tox version on rtd-verify.sh script until tox>=4.0.2 and tox-pyenv>=1.1.0 compatibility issues and bugs are resolved.

v0.83.1

Bug Fixes

• Pin tox version until tox>=4.0.2 and tox-pyenv>=1.1.0 compatibility issues and bugs are resolved.

v0.83.0

New Features

• Replace the usage of plaintext sonarcloud api token with a Jenkins credential. The default value for the credential ID is ‘sonarcloud-api-token’ and we are standarizing it for all projects so this parameter does not require an override.

Upgrade Notes

• Parameter sonarcloud-api-token is NOT required anymore after upgrading to this GlobalJJB version, please remove it from all JJB projects.

v0.82.4

Bug Fixes

• Remove unnecessary quotes around the variable that processes glob patterns.

v0.82.3

Bug Fixes

• Replace Nexus IQ build Target from “${REQUIREMENTS_FILE}” to “${NEXUS_TARGET_BUILD}”. The scanner is only including the requirements.txt file in its scan which should not contain other information than python package requirements. Instead, use a “${NEXUS_TARGET_BUILD}” parameter which the user can optionally provide to the scanner to indicate a file or directory to include in the scan. By default, this variable is configured to scan all files in the repo.

v0.82.2

Prelude

The SBOM generator script creates an spdx file in the root level. When the artifacts are staged the file gets overwritten.

Bug Fixes

• Create the spdx file as ${PROJECT}-sbom-${release_version}.spdx and then copy the spdx file under the namespace ${group_id_path} dir.

• Minor changes to improve openstack cleanup scripts.

v0.82.1

Known Issues

• line 48: tox: command not found

Bug Fixes

• Addresses a bug whereby the openstack orphaned objects/ports scripts exit early with an error when grep/awk do not match any orphaned objects. The fix allows jobs using the scripts to continue when no cleanups operations are required.

• The venv created for tox is unavailable when the semantics of the script are split across files, therefore ensure venv is created with –venv-file option and set.

v0.82.0

New Features

• Added a script to cleanup generic openstack objects.

Bug Fixes

• Addresses failures when cleaning up orphaned openstack ports. The main “openstack <object> list” command no longer accepts the “-c created_at” option, which has been moved to a property of the object and must now be queried with “openstack show object UUID”. Also, the created_at parameter sometimes returns “None” instead of a timestamp, and the existing version of the script does not catch this condition.

v0.81.6

Bug Fixes

• Remove line break in the docker_push_command variable causing the varible to not be set properly

v0.81.5

Prelude

PyPI verify jobs requires Python 3.x. The tox run picks up default version of python instead of the version made available through pyenv.

Known Issues

• Re-factor lf-activate-venv() to skip a return, while the venv is re-used, so that the PATH can be set.

Bug Fixes

• Fix the docker-push script which is broken due to a synxtax error causing the docker_push_command variable to not be set.

• Update the tox install and run script to Call lf-avtivate-venv().

v0.81.4

Bug Fixes

• Sonar CLI job needs to use the credential that matches the name of the project. That is, “sonar-token-{project-name}”.

v0.81.3

New Features

• Add gerrit-cli-sonar and github-cli-sonar scanner job for non maven based repos. This job downloads a specific Sonar CLI version and runs sonnar-scanner on the code to produce a report which is pushed in SonarCloud.

v0.81.2

Prelude

OpenDaylight jenkins maven jobs with jdk17 and CentOS7 currently fails with a confusing message stating that the JAVA_HOME variable is not correctly set. This can happen in various cases, usually when there is a mismatch between the jdk used by maven and the folder pointed by JAVA_HOME. It appears that openjdk17 is not available with CentOS7 and that the folder indeed does not exist

Known Issues

• Current message (JAVA_HOME variable is not set) is confusing and can lead to erroneous interpretations.

Bug Fixes

• Add a folder existence check in related script before propagating JAVA_HOME variable to other scripts. If no folder was found, try to find an approaching solution and exit in case of failure with a more relevant error message.

• Install yq in the venv that is called by the builder scripts of RTDv3 and docker jobs.

Other Notes

• Adapt and refactor code consequently to be more agnostic to distribution and jdk installation specificities

v0.81.1

Known Issues

• git-review tries to copy commit-msg hook to submodules with incorrect source file path (.git/hooks/commit-msg) and fails - the path should be ../.git/hooks/commit-msg if a relative path is used since the copy command is run in the submodule directory

• lf-activate-venv creates a virtual environment in the current working directory where lf-activate-venv is run. This clutters the repository and all the files for the virtual environment are added for update.

Bug Fixes

• Set ‘core.hooksPath’ with the absolute path of the top-level hooks directory so that the correct source path can be used regardless of the working directory.

• Use the correct command depending on the $install_args value to avoid creating an additional virtual environment in the current working directory.

v0.81.0

Prelude

Update openstack images with the auto update image requires more recent version of git-review > 2.2.

New Features

• Add support for a new option to set venv file.

  lf-activate-venv –venv-file /tmp/.robot_venv robotframework

  Modify lf-activate-venv() to allow creation of a venv file and re-use the venv to improve job performance. When a dependency is already installed, pip skips the package therefore reduces the time it takes to create venv in every script.

  Precedence for venv file.

  1. Re-use an existing venv file if one exists.

     1. Use venv file path from –venv-file

     2. Use default venv file path “/tmp/.os_lf_venv”

  2. Create new venv when 1. and 2. is absent

  Note: The default file “/tmp/.os_lf_venv” is created by a pre-build script (../shell/python-tools-install.sh).

  In the situation where a fresh venv is required remove “/tmp/.os_lf_venv” before calling lf-activate-venv().

  Update all the required scripts that call lf-activate-venv().

Known Issues

• Error:

  Errors running git rebase -p -i remotes/gerrit/master fatal: –preserve-merges was replaced by –rebase-merges

  Ref: https://review.opendev.org/c/opendev/git-review/+/818219

Upgrade Notes

• The previous version of git-review is incompatible with the latest version of git due to renaming flags. This is fixed in git-review 2.2.0.

Bug Fixes

• Clean up conditions introduce in the shell scripts, while these checks are performed within lf-activate-venv().

v0.80.2

Known Issues

• Error: openstack: command not found

Bug Fixes

• lf-pyver() fails to include the currently selected version in the output of ‘pyenv versions’, which makes the version change every time the local version is set by pyenv with the version from lf-pyver().

  Fix the command to extract the list of Python versions to include all the numeric versions in the list.

• Use lf-activate-venv to install openstack deps

  Using python-tools-install.sh for the pre/post build is not recommended approach for installing python dependencies since this installs the dependencies with –user option (removed in I821a86ac3b54f284e8).

  Instead use lf-activate-venv to setup an venv and pull in the required dependencies and save the path of the virtualenv in a temp file that can be checked before attempting to create a venv.

v0.80.1

Known Issues

• yq: command not found

Bug Fixes

• Install yq through lf-activate-venv rather than using the python tools install script.

v0.80.0

Known Issues

• ERROR: Not installed on host: python3.8.13 ERROR: Can not perform a ‘–user’ install. User site-packages are not visible in this virtualenv.

Bug Fixes

• Set the default version to ‘python3’ instead of ‘3.8.x’ since some of the older images may not have the specifc version installed. The default version is only used when lf-env.sh is not available.

  CR I821a86ac3b54f2 sets and uses python 3.x version made available by pyenv therefore remove the –user option which is no longer required.

v0.79.4

Known Issues

• Addresses problems found while troubleshooting IT-24352

Upgrade Notes

• Use pyenv whih is the standard way to manage, set and use a python3 installation on the system.

  The required version of python3 for all jobs should be > 3.8.x, to avoid PyPI dependencies conflicts with outdated versions. However, the lf-activate-venv() uses the system default version python installed through packages. This can cause warning and build failures that source lf-env.sh.

  Update lf-activate-venv to use pyenv versions of python3 installed through the lfit.python-install galaxy ansible role.

• Upgrade Packer version to v1.8.2. The version is more recent v1.8.2 and has security updates.

  Ref: https://github.com/hashicorp/packer/releases/tag/v1.8.2

Bug Fixes

• Fix lf-activate-env code comment. The comment suggests using just the version number --python <x.y>, however as per the code the correct format as per the code is --python python<x.y>

• Added support for debian in update-java-alternatives.sh (addresses potential Ubuntu detection bug) Safer handling of unset/null SONARCLOUD_JAVA_VERSION variable preventing java runtime issues

v0.79.3

Bug Fixes

• Copy SBOM report to the project’s m2repo so that is signed by SIGUL and pushed in the same staging package as the maven artifacts.

v0.79.2

Bug Fixes

• Set lf-activate-env to use Python 3.8 while running lftools deploy logs. This fixes the below warnings which when jobs try to use default version of python 3.6 which is EOL.

  CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.

  PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later.

v0.79.0

Bug Fixes

• Update to the latest version of SBOM (v0.0.15) that allows the usage of a custom maven settings file to resolve transitive dependencies. Update thebom-generator script to pass the project’s global settings file and update the sbom file name so is better identifiable.

v0.78.0

Upgrade Notes

• Upgrade NexusIQ Client to more recent version 1.140.0-01.

v0.77.4

Deprecation Notes

• Our Sigul bridges now have publicly accessible DNS names, so it is no longer necessary to create an entry in the hosts file. Since this process relies on up-to-date IP addresses being configured in each project’s global env vars, it can cause avoidable errors. It is therefore being removed.

v0.77.3

Bug Fixes

• Fix URL path indent, add a default ARG to the Dockerfile to remove WARNING. Set the .asc files permissions to jenkins after the sigul has signed the files.

v0.77.2

Bug Fixes

• Update the sigul-sign-dir.sh to sign artifacts using docker. The docker image is built on CentOS Streams 8/9. The newer version of sigul 1.1.1 available for CentOS 8 is not backwords compatible with the version of sigul on CentOS 7.

  As a temporary workaround build a CentOS7 docker image with sigul installed and use it for signing artificats on platforms where sigul is not readly available.

  Note: the executor node needs to have docker installed, so it can’t be a “vanilla” build node but must be a docker node.

v0.77.0

Upgrade Notes

• Jenkins Job Builder 4.1.0 is now the default version. global-jjb has been pegged to version 2.8.0 since v0.55.3 released on 2020-07-21. Since this release JJB has dropped support for Python 2.7 and version 4.1.0 of JJB has required fixes needed for dealing with versions of plugins that are now shipping for Jenkins that cause issues.

  Projects that set override their JJB version should either remove the pin and take what global-jjb defaults to, or reset their pin to 4.1.0

v0.76.4

Bug Fixes

• SBOM’s path flag does not work as expected. We need to introduce a new flag called SBOM_PATH to isolate the path where SBOM is going to be extracted to and executed from. By default this is set to $WORKSPACE but some projects need to execute the sbom from a different location in their code. See https://github.com/opensbom-generator/spdx-sbom-generator/issues/227

• Optionally run a script before and/or after maven goals. This will help add dependencies and post process builds with more flexibility to the project’s needs.

v0.76.3

New Features

• Post-build script capture-instance-metadata.sh will attempt to choose the best possible version of Python to use. If the $PYTHON variable is set, this will be used. If not, we check to see if python3 is available, as this should point to the latest version. If this is also not available, we run with the basic python command.

v0.76.0

New Features

• Append build result to cost.csv file. This would be useful to capture stats for unattended jobs and potentially send out reports to PTL’s on resource usage stats.

v0.75.1

Bug Fixes

• Activate the virtual environment. Python may not be available by default on all versions.

v0.75.0

New Features

• Add new conditional builder step which calls a specific version of SPDX SBOM generator which runs a scan to generate a software bill of materials report in a specific repo.

v0.74.0

New Features

• Process orphaned coe clusters for K8S jobs

  K8s (COE cluster) jobs by default creates stacks names that does not match JOB_NAME, therefore ignore them while processing orphaned stacks and handle them separatly when cleaning up the orphaned clusters.

  The stack naming scheme is limited to take first 20 chars from the JOB_NAME while the rest is randomly generated for uniqueness: https://github.com/openstack/magnum/blob/master/magnum/drivers/heat/driver.py#L202-L212 This breaks the openstack cron jobs.

v0.73.0

Prelude

Enable support for OpenJDK17.

New Features

• Add support for the latest version of JDK17 to be used with lf-update-java-alternatives. This allows job to switch between the required version of JDK.

v0.72.0

New Features

• Add gerrit-go-verify job that will support running unit test for Go projects. It will run the command ‘go test ./…’ inside the indicated GO_ROOT path.

v0.71.0

New Features

• Add support for wait flag on SonarCloud quality gates, this way jobs won’t finish until the quality gate reports back the result during the analysis step, which will fail anytime the quality gate fails.

v0.70.0

Bug Fixes

• gerrit-maven-sonar-verify was using the “default” branch choosing strategy, which pulls master, rather than the “gerrit” strategy for pulling feature branches.

v0.69.1

Prelude

Improve global-jjb code and documentation to minimize non-inclusivity. https://developers.google.com/style/inclusive-documentation

New Features

• Add packer builder macros to create a ‘ansible.cfg’ file. This is required by packer build jobs to set ansible host configuration. The job or image specific configuration can be created using JCasC custom files in the ci-man repository.

  Example configuration: https://github.com/ansible/ansible/blob/stable-2.11/examples/ansible.cfg

Upgrade Notes

• This change requires creation of a custom managed file (ansible.cfg) on the Jenkins environment with the default line “remote_src = ~/.ansible/tmp”.

Bug Fixes

• Fix shell script maven-sonar.sh to handle parameter $SCAN_DEV_BRANCH as string and not as boolean, so we can achieve the desired IF logic

Other Notes

• Rename ‘whitelist’ to ‘allowlist’

v0.69.0

New Features

• Add new Maven SonarCloud verify job that will execute SonarCloud scans before a change gets merged.

v0.68.1

Bug Fixes

• The OpenStack JCasC yaml converter has learned how to properly differentiate between volumeFromImage and image boot sources.

• Pin pyparsing<3.0.0 which is required by httplib2 0.20.1. A new version of pip 21.3.1 is out that has removed the dependency pyparsing<3,>=2.4.2 as required by httplib2.

v0.68.0

Prelude

JDK8 is soon going to be EOL by Mar 31, 2022. with most of the LF projects already using JDK11 (LTS), upgrade default version to JDK11 (LTS).

Upgrade Notes

• Upgrade default version to JDK11 (LTS).

  https://www.oracle.com/java/technologies/java-se-support-roadmap.html

Bug Fixes

• The JCasC convert for OpenStack was improperly converting executor definitions. The script has learned the proper syntax.

v0.67.1

New Features

• Add sonarcloud-java-version parameter to LF Sonar builders, which allows setting the JDK version to use with the Sonar scanner (default: openjdk11).

v0.67.0

New Features

• Add “Unmaintained” as a valid lifecycle_state to be used in INFO.yaml files.

v0.66.1

Bug Fixes

• Add missing v3-standard Vexxhost flavors to create_jenkins_clouds_openstack_yaml.

v0.66.0

New Features

• Add v3-starter flavors to cloud lookup in create_jenkins_clouds_openstack_yaml script.

• Add support in lf-maven-sonar scan jobs to process short lived dev branches.

v0.65.3

Bug Fixes

• Openstack labels need to include the config name, in addition to any labels explicitly defined. This also changes the builder name to match the config name, rather than using the labels (which can be only one label, but is technically a space-separated list).

• If no volume_size is defined, the default behavior was to set one to 10GB. However, the proper way to handle this is to use an “Image” boot source rather than “Volume From Image”.

v0.65.2

Bug Fixes

• When running create_jenkins_clouds_openstack_yaml.py, rather than quitting if there are no labels defined, we can instead use the agent name (e.g. “centos7-2c-1g”) as the default label. This recreates the functionality of the groovy scripts previously used.

v0.65.0

New Features

• Parallel jobs are now natively supported by tox since version 3.7.0 thanks to the option “-p” / “–parallel”. This new option offers more possibilities than detox and other options have also been introduced to tune tox behavior in parallel mode. This evolution allows more tox parallel mode configurations in yaml templates.

v0.64.0

New Features

• Create a bashate tox profile. Bashate is a shell linter inspired from PEP8. It is now enforced by CI to improve bash code style.

• Update to the sysstat script to add support for operating system Ubuntu 20.04 and Docker systems

Bug Fixes

• Fix the issues detected by bashate (E043, E010, E011, E002, E020, E006, E003).

v0.63.1

Bug Fixes

• Set S3 URL in the framename of the target attribute

  The description-setter plugin does not read the URL when the framename is unset.

  Issue: LF JIRA RELENG-3269

Other Notes

• Conventional Commit message subject lines are now enforced. This affects CI. Additionally, if developers want to protect themselves from CI failing on this please make sure of the following

  • you have pre-commit installed

  • that you have run pre-commit install –hook-type commit-msg

• yamllint is now enforced on commits via pre-commit. This affects both CI as well as developers that have pre-commit properly configured in their environment.

v0.63.0

Bug Fixes

• Provision global-settings to replace the default used by the Unified Agent.

• The choosing strategy for the docker-merge-{stream} jobs for gerrit was improperly configured to use. This is now corrected.

v0.62.0

New Features

• Add sonar-prescan-script jobs for maven, allowing maven sonar jobs to execute a shell script prior to the scan.

• Add artifact distribution type for self releases. This support allows customers to specify a particular name and path of an artifact in Nexus which will be downloaded locally, re-tagged and posted into the releases repository of the matching repo.

Bug Fixes

• Pin git review to 1.78

  The latest version of module trys to look for git hook recursively within the submodules.

  Error:

  Running: git submodule foreach cp -p .git/hooks/commit-msg "$(git rev-parse --git-dir)/hooks/"
  Problems encountered installing commit-msg hook
  The following command failed with exit code 128
       "git submodule foreach cp -p .git/hooks/commit-msg "$(git rev-parse --git-dir)/hooks/""
  -----------------------
  Entering 'global-jjb'
  cannot stat '.git/hooks/commit-msg': Not a directory
  fatal: run_command returned non-zero status for global-jjb
  

  Remove workaround that has been resolved in v1.28 and use lf-activate-venv to install git-review

• Reorder functions and add function labels to make release-job.sh easier to read.

• Update the create script to include V3 flavors. Newer V3 flavors are faster and garuntees the to run on new hardware.

v0.61.1

Bug Fixes

• Pin cryptography to 3.3.2

  The latest version of module breaks compatibility with the latest version of pip.

  Error:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-vqk6fya9/cryptography/setup.py", line 14,
    in <module> from setuptools_rust import RustExtension
  ModuleNotFoundError: No module named 'setuptools_rust'
  

  Reference:

  PYCA#5753 <https://github.com/pyca/cryptography/issues/5753> PYCA#5771 <https://github.com/pyca/cryptography/issues/5771>

v0.61.0

New Features

• Enable support for artifact type releases. Add initial schema for artifact release verification.

Upgrade Notes

• Upgrade Packer version to v1.6.6. v1.6.6 gives more debug messages which is not seen with 1.4.2.

Bug Fixes

• Add self release verify and merge jobs for GitHub based projects.

• Fixes an bug with ‘job-cost.sh’ that would cause builds to be marked as unstable if not run on AWS or OpenStack.

• The old version of rtd-verify is missing the “–init” flag, which causes it to not add new submodules. Additionally, the “–recursive” flag has been included to ensure proper loading of recursive submodules.

• Updates the ‘capture-instance-metadata.sh’ script to skip attempting to capture instance metadata needed for job-cost.sh if the build is being run on an unsupported cloud or platform.

• Updates the ‘sudo-logs.sh’ script to set ownerhsip to current build user and user’s login group, instead of the explicit ‘jenkins:jenkins’. This will allow sudoer log ownership to work on builders not using ‘jenkins’ as their build username.

v0.60.5

Bug Fixes

• Add regex to trigger packer jobs when common-packer templates are updated.

• Fix the release job script to handle LOG_DIR unbound variable and condition to check if the LOGS_SERVER or CDN_URL is being used.

v0.60.4

Bug Fixes

• Remove python 2.7 support

  As per the deprecation notice python 2.7 is not long supported. This causing job failures since the dependencies install are not maintained.

  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.

v0.60.3

Bug Fixes

• Rename “tag-gerrit-repo” function in self releases to “tag-git-repo”. This change is in preparation for self releases support for GitHub based projects. Use the name “git-repo” to fit both Gerrit and Github projects.

v0.60.0

New Features

• Add pipeline-verify jobs. This adds a simple pipeline verification job that will lint any pipelines found, and check to make sure that they are not set up to run on master.

Bug Fixes

• Fix release merge jobs to work with AWS S3_BUCKET when CDN_URL is set.

  Log shipping is being migrated from Nexus2 Log to AWS S3 Buckets. This requires the release job scripts to handle CDN_URL and set the correct value for LOG_SERVER.

v0.59.0

Bug Fixes

• Activate lftools env before processing the logs-deploy script. This will allow us to fetch the latest version of pyOpenSSL and cryptography to resolve the following issue https://github.com/pyca/pyopenssl/issues/973

v0.58.1

Bug Fixes

• Pin idna to 2.9 to resolve package incompatibility with lftools 0.35.1.

v0.58.0

New Features

• Adds a create_jenkins_managed_files_yaml.py administrative script for generating JCasC yaml files from a directory for control of the Managed Configuration Files plugin

Bug Fixes

• Allow projects to override Semantic Version (SemVer).

  It’s recommended to use Semantic Versions (SemVer) for releases. Refer to https://semver.org for more details on SemVer. For projects that do not follow SemVer can use a job build parameter (OVERRIDE_SEMVER_REGEX) with the release job. This build param overrides the default SemVer regex.

  The default SemVer regex is taken from https://github.com/fsaintjacques/semver-tool.

v0.57.4

Deprecation Notes

• Cmake builds upload to SonarCloud. Fix build env to use openjdk11 sonarcloud upload will stop working with java8 on October 1st 2020

Bug Fixes

• Fix the capture-instance-metadata script to cleanly exit on AWS nodes.

• When evaluating jenkins-config management changes, if a system does not have an openstack cloud defined, we should not fail the job. Instead we now just skip that configuration and verification

• openstack-protect-in-use-images.sh Projects that do not have images in jjb were failing this step of the build. code now runs with set -eu -o pipefail for the duration of the script. shellcheck disable comments no longer needed and removed. Code now only merges arrays if non-empty. Simplify dedupe array code.

• Fix rtd-verify.sh to better validate submodules. rtd-verify.sh was using “git submodule” commands to validate submodules, but Jenkins reads the .gitmodules file and executes a “git config” command for each submodule. Because of this, if a bad submodule was added to .gitmodules, it would pass verify but cause failures on subsequent builds after it was merged.

  This change closes that loophole by reading the .gitmodules file, and then running through the same “git config” command for each submodule that Jenkins runs when pulling in the main branch.

v0.57.3

Prelude

Provides method to notify administrators if important jobs are failing.

New Features

• Allows customization of email address and email prefix. failure-notification-prefix failure-notification

Upgrade Notes

• Updating to this version of JJB will _require_ updating jjb/defaults.yaml in the same patchset. The following will need to be defined: failure-notification: “foo@bar.org” failure-notification-prefix: “[Some Prefix]”

Bug Fixes

• Improve alpine compatibility by changing the mktemp call in python-tools-install.sh to be compatible with both GNU mktemp as well as BusyBox mktemp (which is used in alpine).

v0.57.1

Bug Fixes

• Check condition when ${NOMAD_DC} is undefined or unset

  Check if the environment variable $NOMAD_DC is not defined or unset, this avoids the script from exiting without capturing instance metadata.

v0.57.0

New Features

• Packer CI jobs now have the ability to specify which “builder” to use in the job. Default configuration is the for the “openstack” builder.

Upgrade Notes

• Requires common-packer v0.7.0 if using packer CI jobs.

• If your project is in AWS and uses the Packer CI jobs to build AMIs you must set packer-builder: aws when upgrading to this version of global-jjb or your jobs will default to openstack and fail.

Bug Fixes

• Optionally get docker container tag to be set to the value of stream.

• Fix email notifications for lfdocs-conf releases. The name of the repo does not match the configuration in the code. Add a new PROJECT_SLUG variable to use in case the name of the repo is different from its configuration.

• Parallel mode for tox environments is broken if the user passes a JJB bool value to the job-template. We now lowercase the PARALLEL variable when comparing in the bash script to ensure the user provided value is compared against the lowercase value.

• Run Sonar scans using JDK11. java-version will only set java for the maven build part of the job, the sonar scan will use sonarcloud-java-version which is set to openjdk11. Projects not compatible with JDK11 will be able to run their build with java-version set to their JDK preference. https://sonarcloud.io/documentation/user-guide/move-analysis-java-11/

v0.56.0

New Features

• Capture instance metadata for the VM’s. This data is useful while working with the cloud provider and debuging VM issues on the infrastructure.

Bug Fixes

• Fix image update script to handle $image_type correctly.

  This fixes the issue when a specific image type overwrites other image types incorrectly in the .cfg and some images names getting excluded.

• Minor fixes to correct shellcheck warnings/errors.

• Retains the expected behaviour of Gerrit Trigger job configuration for the comment-event-parameter-mode when a project upgrades their JJB to 3.5.0 or newer.

  In JJB 3.5.0 support was added to configure the comment-event-parameter-mode however, while it’s default mode in JJB matches the documented help text for the configuration in Jenkins the Gerrit Trigger plugin itself appears to default to PLAIN mode when the parameter is excluded. This patch retains what we expect to be the default behaviour.

• Update sigul-install.sh to check for sigul. If sigul is already installed, we can skip the installation. This is important due to kojipkgs being unreliable. We now have the sigul binary baked into the base image creation, so newer images should already have sigul on board. If they do and we still try to do this manual installation, we could still run into connection issues with kojipkgs.

v0.55.3

Bug Fixes

• Fix the race condition by checking the created_at timestamp and clean up ports that were created at least 30 minutes before.

  There a race condition in the openstack-cron job that causes the script to delete ports in DOWN state and are still in use by the VM, causing the ODL CSIT jobs to fail.

  JSD ISSUE: RELENG-3062

v0.55.2

New Features

• Introduce lf-pipelines-verify job to test the LF’s global pipeline library.

Deprecation Notes

• lftools_activate should no longer be used and will be removed in a future release.

Bug Fixes

• Fix the jjb-deploy job to accept a JJB_VERSION parameter in the parameters field so that projects can select which version of JJB they want for the job.

  The change that ended up breaking jjb-deploy was caused by this Gerrit patch here:

  https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/62788

  The bug was introduced in the lf-env.sh when the lf-pip-install macro was removed. Prior to the removal the jjb-deploy job was installing the latest and greatest JJB version. However now jjb-deploy is now installing the fallback version of JJB_VERSION from the lf-env.sh script which is 2.8.0.

  Reference: RELENG-3073

• Fix quoting bug in rtdv3.sh script that caused failures to exit improperly.

• Resolves the lftools_activate failure below.

  ---> maven-deploy.sh
  ImportError: cannot import name 'enquote_executable'
  

  Fix is to replace lftools_activate with lf-activate-venv by sourcing lf-env.sh lf-activate-venv is a more error resistant way to make sure a pip package is installed

• Release verify will not allow valid semver versions like “1.3.2-1” which should be accepted. Update semver regex to conform to https://semver.org/ This regex was taken from https://github.com/fsaintjacques/semver-tool which follows the semver guidances and works in bash.

v0.55.1

Bug Fixes

• Pin importlib-resources to 2.0.0 to resolve package incompatibility between it and the currnet virtualenv version (20.0.21). Until a compatibile virtualenv is released this dependency needs to be pinned.

• Fixed logic for discovering new branches When a new branch exists on gerrit, but has never been seen by RTD we need to trigger a job so that the branch is discovered we can then mark it active in the following step. API changed under out feet, where a 404 was returned before we now get a 200 with the string null. Change code to explicitly match the returned string of “null”

v0.55.0

New Features

• verify-upstream-global-jjb jobs have been improved in two ways: the upstream Gerrit name has been changed to lf-releng to match the precedent set by OpenDaylight, and we have added a Github version of the job.

Bug Fixes

• Fix Job key in commit message body rather than trailers section (AKA footer). The commit message produced by gerrit-push-patch which currently creates a commit message where the Job key appears in the commit message body rather than the trailer.

  For example:

  An example commit message
  
  Job: builder-job/123
  
  Change-Id: 1234567
  Signed-off-by: Jenkins <jenkins@example.org>
  

  This fixes it to:

  An example commit with proper trailer
  
  Job: builder-job/123
  Change-Id: 123457
  Signed-off-by: Jenkins <jenkins@example.org>
  

• The packer merge job has a boolean option that sets an UPDATE_CLOUD_IMAGE variable. This variable was always evaluating to true which caused issues with builds always executing the section of the build. This causes problems for builds that do not produce information that is expected by the section of code. In specific AWS / EC2 builds fail as the build engine outputs different name data than expected. The variable is now properly evaluated.

v0.54.0

New Features

• Add new templates to build CXX projects with GNU autotools. These support the “configure && make && make install” pattern used by many open-source projects. Supports generation of the configure script for projects that do not store generated files in version control. Includes gerrit and github versions of autotools-packagecloud-stage, autotools-sonarqube and autotools-verify.

Bug Fixes

• Echo error response from RTD without formatting. The script rtd-trigger-build.sh starts a build at ReadTheDocs and parses the response, a small JSON like this:

  {“build_triggered”:false,”project”:”unicorn”,”versions”:[]}

  This change drops the invocation of jq that attempts to pretty-print the JSON to the log when the build is not triggered, because that call was failing for inexplicable reasons.

• Extend scripts that invoke the docker CLI to report the version by invoking docker –version. Also add echo command to report end of script where it was missing. Includes docker-build.sh, docker-login.sh, docker-push.sh and release-job.sh. No functional change to any script behavior, just a line of extra output.

v0.53.2

Known Issues

• Update regexp in lf-infra-ship-logs macro. This fixes a regression from update to deploy-logs.sh script in v0.53.0 which causes build logs links to disappear on the Jenkins jobs description setter.

  Tests: https://jenkins.opendaylight.org/releng/view/builder/job/builder-packer-verify/274

v0.53.1

Upgrade Notes

• Before upgrading to this version of global-jjb, must remove any uses of the job templates named above. These jobs did not yield any results, so it’s unlikely they were being used, and dropping them entirely should not cause any loss of information.

Bug Fixes

• Fix update cloud image list job to handle newlines correctly and update an existing gerrit change request.

  A gerrit change submitted through git-review checks for an existing change-id on Gerrit and either updates the patchset or creates a new one. For this to work correctly the change-id should go into the commit footer and not in the GERRIT_COMMIT_MESSAGE. The embedded newlines are not processed correctly in the script and fails to separate the footer and the GERRIT_COMMIT_MESSAGE, instead it creates a new patches everytime rather than updating an existing one.

  The change is tested now existing CR’s are updated rather than pushing a new CR everytime.

  https://git.opendaylight.org/gerrit/c/releng/builder/+/89136

• Drop jobs gerrit-python-xc-clm and github-python-xc-clm from the two python job groups. Those templates were replaced by gerrit-tox-nexus-iq-clm and github-tox-nexus-iq-clm. Not adding those templates to the job groups because successful use requires additional project configuration to report the package requirements.

v0.53.0

New Features

• Add “logs” prefix to s3_path(“logs/$SILO/$JENKINS_HOSTNAME/$JOB_NAME/ $BUILD_NUMBER/”). We are not allowed to have an index.html file at the root level of the S3 bucket. Adding the additional prefix allows us to mirror the directory structure of Nexus where you are able to see both production and sandbox logs when browsing to https://logs.<project>.org/.

• Add templates gerrit-pypi-stage/github-pypi-stage to allow projects flexibility to control when Jenkins publishes a redistributable Python package, either on merge (with the merge template) or on command (with the stage template).

• New templates gerrit-cmake-packagecloud-stage and github-cmake-packagecloud-stage for building DEB/RPM package files and publishing them to a PackageCloud.io repository on posted comment. Adds builder macros lf-packagecloud-file-provider and lf-packagecloud-push. Adds script packagecloud-push.sh to call Ruby gem package_cloud. The new templates are lifted from the ORAN project.

• Verify build nodes named in YAML files against config files. Extend lf-jjb-verify anchor with boolean configuration variable check-build-node-labels that guards a conditional build step. If true, run script jjb-verify-build-nodes.sh to check build-node labels in YAML files within the jjb subdirectory against labels defined by config files in the jenkins cloud configuration directory. Disabled by default. Projects should enable and configure the job; e.g., for external build node labels.

• Add a packer-verify-build job. This job is made to be manually triggered in order to build the packer image, so that the full build process can be validated before merging. If this is done on the final patch that is merged, the merge job will not run another build (thereby avoiding building two identical images).

• Optionally tag repo during release process. Set to true by default. Allow projects to skip repo tag in cases where multiple release types happen whithin the same repo.

Bug Fixes

• Standardize yaml parsing tools used across global-jjb. Switch from using niet package to yq for parsing yaml files in release-job.sh.

• Fix issue where job fails if the global variable S3_BUCKET is not set. Set conditional parameter “on-evaluation-failure” to “dont-run”.

• Schema type “bool” is not valid. Fix to “boolean”

• Fix script to remove outputting $ARCHIVE_ARTIFACTS. Outputting required that $ARCHIVE_ARTIFACTS be set for every job. Instead it will output ${pattern_opts:-}, which if $ARCHIVE_ARTIFACTS is not set, will be blank.

• Fix the rtd verify script failure. The script attempts to install lftools dev with –user and fails on the error.

  Error:

  [Errno 13] Permission denied: ‘/usr/local/lib/python3.6’

  This code is actually required when a new commands are added to lftools, the docs verify job needs to run the new command and install dev version lftools version.

• Document the jjb-verify feature that checks build-node labels named in JJB YAML files against nodes in cloud config files. Change the feature’s JJB configuration variables to share a prefix. Improve the shell script to be robust to errors such as a suffix of “.cfg” or an external label of “” (just two double quotes).

• Add @weekly cron default value to docker_merge_common anchor, no longer empty. The merge job for docker images is like the stage job for java artifacts, every docker image is a release candidate. Run the merge job regularly to check dependencies like base images and to push updated images to the Nexus3 staging registry.

• Allow rtdv3 jobs to run in parallel.

• Revise tox-run.sh to guard against glob matching no tox log files. In that case the pattern is passed to the cp command, which fails. Detect the cp failure (‘cp: cannot stat ..’) and exit the loop. This new and undesired behavior was introduced by adding -e in change 76a0761, so the script stops when the cp command fails.

v0.52.0

New Features

• New templates gerrit-cmake-sonarqube and github-cmake-sonarqube use the SonarQube Jenkins plug-in to analyze CXX code and publish the results. Modeled after the generic gerrit/github-sonar templates. The new templates lift the limitations of the existing templates gerrit/github-cmake-sonar which download and install a Sonar scanner, and cannot report unit-test code coverage statistics.

• New templates gerrit-tox-sonarqube and github-tox-sonarqube use the SonarQube Jenkins plug-in to analyze Python code and publish the results. Modeled after the generic gerrit/github-sonar templates. The new templates eliminate the need for mvn-settings in the job configuration and the need for a pom.xml file in the repo.

Bug Fixes

• Add missing config property “stream: master” to lf_cmake_sonar macro in lf-c-cpp-jobs.yaml to match existing “branch: master” property. This makes the template consistent with other Sonar templates.

  Remove double-quote chars that were added recently around $make_opts from the invocation of the build wrapper in cmake-sonar.sh. This makes the invocation consistent with the invocation of cmake in the same file.

  Update documentation to reflect correct property names and defaults.

• Workarounds for aging python 3.5 on Ubuntu 16.04 builders Ubuntu 16.04 calls a python2 version of tox we must upgrade it even if we are testing with python3 ubuntu 16.04 runs python 3.5 we must pin the zipp package more pins will be needed as packages drop support for python 3.5 ideally projects will move away from this distro.

• Update jq validation of returned json blocks to work properly with jq v1.5.

• Switch info-file-validate.sh to use lf-activate-venv moves job from python2 to python3

• Revise script nexus-iq-cli.sh to stop on error or unbound variable. This should fail the build if the scanner returns a non-zero code, for example if credentials are missing or wrong.

• Refactor templates in lf-c-cpp-jobs.yaml with common anchor/alias lf_cmake_common to reduce redundant configuration. This includes gerrit-cmake-sonar, github-cmake-sonar, gerrit-cmake-sonarqube, github-cmake-sonarqube, gerrit-cmake-stage, github-cmake-stage, gerrit-cmake-verify and github-cmake-verify. No functional change.

  Document maven settings parameters in g*t-cmake-stage templates.

• Revise templates gerrit-cmake-sonarqube and gerrit-tox-sonarqube to move the triggering comment string into a parameter that can be overridden in a job definition. Github comment remains hardcoded. The default is still “run-sonar”.

  Rename gerrit-cmake-sonarqube tempplate configuration parameter from sonar-prescan-script to pre-build to be consistent with existing CMake stage and verify templates. This is a breaking change for any early adopters of this new template.

• Reduce volume of output from wget and unzip by adding the -q flag to the invocations in cmake-sonar.sh There’s little need to see wget progress messages or archive contents in the build log.

• Revise tox-run.sh to guard against unbound variable TOX_ENVS; stop on error or unbound variable (options -eu); and to print commands before executing (option -x). Add echo command at end.

v0.51.0

New Features

• New script has been added: scrape-job-cost. This script will be executed by cron as nexus on the Nexus Server for each project. It will extract cost data from the nexus directory for each Jenkins Builder (production & sandbox). The cost data for each builder will be appended to separate cost files located in ~nexus/cost on the nexus server. The files will be named sandbox-YYYY.csv and production-YYYY.csv.

• New templates gerrit-sonar-prescan-script and github-sonar-prescan-script accept an arbitrary shell-script body that can do work like install prerequisites, build and test to generate a code-coverage report for the Sonar Scanner to find and upload. This adds flexibility that the existing gerrit-sonar-prescan and github-sonar-prescan templates lack.

Known Issues

• Removes ref-update from rtd-merge jobs which is triggering unnecessary jobs to be queued in Jenkins. This ref-update was originally added to enable Jenkins to trigger builds when a release tag is pushed to update the docs however it’s now triggering many unnecessary jobs wasting project CI resources.

Bug Fixes

• Removed broken code that removed leading/trailing white-space from variables. Use lf-activate-venv() to install openstack. Enabled ‘set -euf pipefail’ and updated code to handle errors. Updated out of date comments. Some minor cleanup of code for clarity.

• Call “lftools jenkins” after credentials are set to fix failures due to the call being made without credentials being set first. The previous method did not require credentials, so the failure was introduced when we switched to using lftools. The os_plugin_version variable is not needed before the JENKINS_USER and JENKINS_PASSWORD are set, so no other changes are necessary.

• A recent change has made the “null” string a bad value for FLOATING_IP_POOL. By making it an empty string, we recreate the old functionality of having the default floating IP pool set to “No value”.

• Fix release file detection on commit with multiple parents

• Branch discovery and build polling implemented. If a Branch has not been seen by rtd we trigger a build with rtd and poll till that build is complete. we can then enable the branch and trigger a build against it, again polling all builds untill they are complete.

• Fix to disable cloning submodules for rtdv3 verify job

• Add configurable doc-dir defaults to “docs/_build/html” needed for relative path modifications if you change the tox-dir Modified tox-run.sh with “ARCHIVE_DOC_DIR” variable so that relative paths can be handed when uploading generated docs to the log server

• Use git choosing strategy default in tox and pypi merge jobs for gerrit. This makes those jobs consistent with maven and other merge jobs for gerrit that always build from tip of the target branch to create artifacts from the latest & greatest code. Building from tip (not from Gerrit commit/merge point) avoids confusion about content when changes are merged out of order. For example, a fix is submitted and merged, but the merge job fails. In the mean time, a different change that happened earlier in commit history gets merged (or the merge job is retriggered), causing a new artifact to be pushed. But that artifact does not have the expected fix.

  Add comments to release merge jobs why their choosing strategy is not default.

  Document the git commit choosing strategy for the release merge jobs.

• Extend release-job.sh to detect if distribution_type is missing from the release yaml file and show a meaningful error. The shell option pipefile causes the script to halt silently if niet fails to find that key, which utterly baffles users.

• Use choosing strategy Gerrit Trigger in container/jar and package cloud release merge jobs. This retains the current behavior in the simple merge case, and ensures that a job triggered by a “remerge” comment uses the release file at that commit. The previous choosing strategy, default, uses the tip of the target branch. That does not allow recovery from merge job failure if the target branch has advanced past the commit with the release file.

v0.50.0

New Features

• Generate javadoc from a project in a subdirectory, which lifts the assumption that all files are in the git repository root. Extend maven-javadoc publish and javadoc-verify templates with mvn-dir configuration parameter (like tox-dir), defaults to ‘.’ to keep existing behavior. Extend maven-javadoc-generate.sh script to invoke mvn with the -f option and use the directory name when archiving the generated HTML. Document the new configuration parameter.

Upgrade Notes

• Projects using macros lf-maven-javadoc-publish or lf-maven-javadoc-verify (i.e., using job templates gerrit-maven-javadoc-publish, github-maven-javadoc-publish, gerrit-maven-javadoc-verify or github-maven-javadoc-verify) must ensure the maven -f option is not used in config parameter mvn-params.

• The lf-pip-install builder macro has been deleted. At some point we will also be deleting shell/pip-install.sh.

Bug Fixes

• Fix zgrep pattern for verify_packagecloud_match_release function. Function was looking for “Successfully Uploaded” pattern, which is not the correct response when using packagecloud gem to push packages to packagecloud.

  Fix lf-release docs example to reflect changes to package_name. Package_name should not include the version in the name.

• Disable the pip install warning that are not useful in the console logs.

  00:25:10 WARNING: The scripts easy_install and easy_install-3.6 are

  installed in ‘/home/jenkins/.local/bin’ which is not on PATH.

  00:25:10 Consider adding this directory to PATH or, if you prefer to

  suppress this warning, use –no-warn-script-location.

• Revise shell script maven-javadoc-generate.sh to supply -f argument to maven with argument translated by call to readlink. This works around the javadoc:aggregate behavior of silently doing nothing if invoked “-f .”

• Extend shell scripts that invoke pip freeze to show python and pip versions also

• Revise python-tools-install.sh to drop creation of virtualenv in ~/.local. That is NOT a Python virtualenv and should not be created as such. Using –user installs python modules into ~/.local/lib/PYTHON_VERSION/site-packages. Making ~/.local a virtualenv messes with the paths in site-packages and causes runtime errors like this:

  ERROR: Can not perform a ‘–user’ install. User site-packages are not visible in this virtualenv.

  Reverts part of change I4b2d778f3fd81565c5dd009d50c969696faba0d2

• Extend script python-tools-install.sh to invoke pip freeze. This shows installed package versions in the log and allows detecting changes like the Jan 2020 release of pip v20 that broke various assumptions and behaviors.

• Extend macro lf_tox_sonar with parameters macro lf-infra-maven-parameters so job definitions can configure mvn-opts, mvn-params and mvn-version values. Default values are defined so existing jobs are not affected. This change makes the python-tox sonar template consistent with the maven sonar template.

  Extend the PyPI package structure recommendation with a way to share the docs/ folder.

• The update-cloud-images.sh script was using virtualenv to install openstack which fails because the latest version of openstack requires python3. It fails quietly because that is no error handling in the script (all errors are ignored). The script now uses lf-activate-venv() and it runs with ‘set -euo pipefail’.

• The jjb-deploy-job.sh was ‘activating’ a venv that was created by pip-install.sh in /tmp/v. This venv was based on python2. Now jjb-deploy-job.sh calls lf-activate-venv() to install jenkins-jobs in a venv based on python3.

v0.49.0

Prelude

This release supports breaking changes in the upstream OpenStack Jenkins plugin. Version 2.47+ of the OpenStack Cloud plugin for Jenkins adds two new SlaveOptions params, node_properties and config_drive. Failure to send these params results in a failed request.

A project may need more than one type of release job this patch changes the trigger to match on release file name for pypi and packagecloud release jobs.

New Features

• Extend tag feature of release jobs: Extend release-yaml schema files to allow git_tag entry. Extend lf-build-with-parameters-maven-release macro with GIT_TAG. Extend release-job.sh: detect and use optional git_tag string. detect and reject an existing lightweight tag that blocks push of a gpg-signed tag. change GERRIT_HOST to GERRIT_URL in method for obtaining LF umbrella project to allow testing in the sandbox. rename function from tag to tag-gerrit-repo. simplify tests for presence of Jenkins parameter values show more details about steps as INFO statements Include schema file contents into documentation; remove the copies.

• Add support for automation of promoting packagecloud packages.

• Add gerrit-branch-lock to gerrit ci-jobs, and make it so that it can be triggered from any change, for any branch.

• Generate Job cost information Each job it will archive a CSV file (cost.csv). It will contain a single CSV record containing the following fields: JobName , BuildNumber , Date , InstanceType , Uptime , Cost1 , Cost2 The Date field can be sorted as a string and is readable by your favorite spreadsheet. The Date/Time is GMT. The Uptime is uptime of the build agent (secs). The Cost1 field is the cost($$) of the build node & Cost2 is cost associated with the stack. If the job is not a Openstack job, then Cost2 is ‘0’. The project cost file will based on the year (cost-2019.csv).

• Add support for these new parameters, node_properties and config_drive. Add a new function test_version, which will do a numerical (float) comparison between installed plugin versions and user-supplied test version numbers.

• Add the lf-kubernetes-create and lf-kubernetes delete macros which allow ad-hoc kubernetes cluster creation with a selectable disk size for the docker volume.

• Make tox-dir configurable default is “.” can now be set to “docs/”

• Project can now set their default docs landing page used for self serve release of docs. By default landing page is /latest/ on release landing page should generally be set to /stable/

• Release jobs now support non-ff merges.

• Extend packagecloud release:

  Release additionally requires log directory, ref and version. Add set_variables_packagecloud & verify_packagecloud_match_release to allow tagging the git repo. Update packagecloud jobs to include sigul signing.

Known Issues

• Release jobs still trigger when a release file is deleted.

Upgrade Notes

• The openstack-stack-delete.sh script installs the latest tagged version of lftools and it uses that to get the stack cost. Any version of lftools >= v0.29.0 will contain the required changes to get the stack cost.

• If you are using the INSTANCE_MIN_CAPMAX paramater in your cloud configs, you will need to change it to INSTANCE_MIN when using v2.47+ of the plugin.

• default-version is now an optional parameter ‘latest’ is the default behavior change to ‘stable’ when docs are released

Deprecation Notes

• Deprecated the config option INSTANCE_MIN_CAPMAX, replaced with INSTANCE_MIN when using v2.47+ of the OpenStack plugin.

Bug Fixes

• Currently a Job Config Tarball is added to the archive. The directory layout created by ‘jenkins-jobs test’ has changed and as a result the code that sorted/reformated the directory and generated the tarball was broken.

  The prefix path of the tarball is now ‘job-configs’. The number of log messages has been improved and greatly reduced. The tarball generated contains directory names with spaces and upper-case characters.

• Revise the lf-docker-get-container-tag macro to use include-raw, not include-raw-escape, to silence JJB warning. Improve documentation of lf-docker-get-container-tag macro.

• Removed dependency on ‘user’ venv created by python-tools-install.sh from the sandbox cleanup job. It will call lf-acitivate-venv() instead. Removed references to jjb-install.sh & jjb-cleanup.sh.

• Removed dependency on ‘user’ venv created by python-tools-install.sh from the builder-jjb-verify job. It will call lf-acitivate-venv() instead.

• Make shell script builder step customizable in lf-docker-get-container-tag. This will allow ONAP to call their own script. Set to default to “../shell/docker-get-container-tag.sh” which is what teams were already using.

Other Notes

• Disable generation of pip package lists during builds: packages_end.txt.gz, packages_start.txt.gz & packages_diff.txt.gz. They are no longer valid.

v0.48.1

Bug Fixes

• Change multi-cloud image validation to look for an OS_CLOUD variable in the cloud.cfg instead of overloading CLOUD_CREDENTIAL_ID as that variable does not usually point to an ID that is in the openstack clouds.yaml file.

v0.48.0

New Features

• By default, the lf-venv-create() function only installs/upgrades ‘pip’ and and any packages that need to be tied to a specific version like. Currently the only version specific package is: ‘jenkins-job-builder==2.8.0’. The current version of lf-venv-create() supports being called twice but that functionality has NOT yet been thoroughly tested. RELENG-2508 has been created to validate & optimize that functionality.

• Created new function lf-activate-venv(). This function creates a venv in /tmp and prepends the bin directory to the PATH. The ‘pip install’ command now specifies: ‘–upgrade-strategy eager’. lf-activate-venv() supports an optional –python flag to specify which python to use to create the venv, the default is python3.

  Two new functions: lf-git-validate-jira-urls() and lf-jjb-check-ascii(). They will be used to replace the git-validate-jira-urls.sh & jjb-check-unicode.sh scripts at some point. For now, they are not being used.

Bug Fixes

• Update the gerrit trigger regexes for the docker job templates to use the shorter and more readable versions of the regexes used in other verify and merge job templates. Clarify documentation for comment triggers.

• Multi-cloud image validation was not properly working as we force set the OS_CLOUD environment variable before validation of images. This has been rectified to dynamicaly modify the OS_CLOUD variable based upon the cloud.cfg that will be used to define the cloud in Jenkins.

• We now validate that a cloud.cfg file exists for any defined OpenStack cloud.

• Removes undocumented, and now unneeded, openstack-cloud variable from the jenkins-cfg-verify job definition

• Change compare-type to REG_EXP in macros lf_python_clm_xc and lf_pypi_common for config parameter gerrit_trigger_file_paths with the regular expression .* Previously was ANT, which didn’t match anything with pattern .* so recheck/reverify comment triggers did not work. Clarify documentation of comment triggers for python jobs.

• Add .pypirc config file provider to builders in release job. Extend PyPI documentation with recommended directory layout.

• Change the comment-added-contains-event strings for the release-verify and release-merge templates to the same regexes used in other verify and merge templates, instead of custom versions that didn’t seem to work; verify now uses ‘^Patch Sets+d+:s+(recheck|reverify)s*$’ merge now uses ‘^Patch Sets+d+:s+remerges*$’

• Change pip’s upgrade-strategy to “eager” for python-tools-install.sh. Pip changed its default upgrade-strategy to “only-if-required”, which is not correctly upgrading requests (and potentially other packages in the future) to meet other packages’ requirements. This results in errors in the build log. By using the upgrade-strategy “eager”, pip is able to properly install what is needed.

Other Notes

• The lf-venv-add(), lf-venv-create() & lf-venv-activate functions have been removed. No-one is accessing it yet.

v0.47.0

New Features

• Refactor PyPI release-verify and release-merge templates to download distribution files from a PyPI staging index and upload the files to a PyPI release index. Remove all the builders that were previously used. Call pip upgrade to install latest version of setuptools and twine. Split the PyPI job groups because the verify & merge templates do not accept the same arguments as release-verify & release-merge templates. Remove stream, branch usage; just one PyPI release job per project now. Extend the PyPI release yaml file schema for log_dir and other values. All this makes the PyPI release ver/mrg templates highly similar to the release-job ver/mrg templates. Revise documentation appropriately. Move the PyPI release features to the lf-release-jobs.(rst,yaml) files. Extend the release-job.sh script to have pypi release functions, and drop the pypi-tag-release.sh script.

• Added two new functions: lf-venv-create(), lf-venv-add() Changed name of lf-activate() -> lf-venv-activate() Updated functionality of lf-venv-activate (support for paths & python versions

Bug Fixes

• Added ‘|| return 1’ to appropriate commands. Functions do not support ‘-e’ and a function will normally continue after a command fails.

Other Notes

• Improved Comments

v0.46.0

New Features

• Global job that triggers on any docs changes. Creates docs project if absent Creates subproject association with master doc project Triggers docs build

• Read-the-docs job will run but skip its verification bits unless the repo has .readthedocs.yaml file in the root of its repository. This allows projects to commit changes to their docs/ dir without having to configure the read the docs builds.

Upgrade Notes

• Extend lf_tox_sonar with sonarcloud, sonarcloud-project-key, sonarcloud-project-organization, sonarcloud-api-token, tox-dir and tox-envs properties; also with lf-infra-tox-parameters macro. Use new sonarcloud property as guard for conditional builder step, if true use lf-infra-maven-sonarcloud, else use lf-infra-maven-sonar.

Bug Fixes

• Add missing trigger “remerge” to the PyPI release merge template. Move trigger definitions into PyPI templates, instead of defining four separate trigger definition blocks and using each exactly once. Document the required comment text for the triggers.

• Add macro with lf-infra-wrappers block to set jenkins-ssh-credential parameter to the value in parameter jenkins-ssh-release-credential, which makes the PyPI release merge templates parallel to the release-job merge template. Both need privileges to push a tag on the Jenkins minion. Document the revised configuration parameter. Silence yamllint issues.

• Add missing underscore in two echo commands in release-job.sh. Move comment for shellcheck disable to new line in lf-env.sh. Extend documentation of the container self-release process. Exclude lf-rtdv3.rst from WriteGoodBear coala processing.

• Add a verification step to maven releases to make sure the version being defined in the releases file matches the actual version produced by the maven-stage job that created the release candidate. This is to prevent releases being pushed in Nexus with a version different from what the developer intended in the releases file.

• Add missing {branch} parameter to branch-pattern in gerrit trigger blocks in PyPI release-verify and release-merge templates. Jobs were starting on all defined branches, not limited to target. Change pypi-tag-release script to continue if tag exists, not stop.

• Update to lf-env.sh. Change lf-set-maven-options(): MAVEN_OPTIONS to maven_options to better support shellcheck.

v0.45.0

New Features

• Add lf-env.sh ‘library’ script in ~jenkins. Sourcing this ‘library’ script from a bash script provides access to a number of functions. The script is installed by the ‘init’ script at boot time so that is is accessible from any Jenkins build script. Hopefully over time other functions will be added to this library.

Upgrade Notes

• Add sonar-project-file parameter to ci sonar jobs. By enabling the caller to override the default project file name with an empty string, we enable the ability to provide project settings directly in the sonar-properties field. This removes the requirement for a “sonar-project.properties” file in the repo.

Bug Fixes

• Fix the auto update image script to compare the image type before updating an the image in the source repository. This fixes the bug that updates images although they are the same flavour but a diffirent type.

• Add DRY_RUN parameter to the PyPI verify and merge template common macro because it’s required by the pypi_upload.sh script. Adjust verbosity of shell scripts - quiet down pip, add repository name to echo output.

• Correct doc .pypirc test URL to https://test.pypi.org/legacy/ Change doc .pypirc to use API tokens instead of username/password. Use pypi-test as the default repository name in the PyPI merge template (instead of “staging”) to match established ONAP practice.

v0.44.1

New Features

• Archive ‘sudo’ logs. The log will be located in the ‘sudo’ sub-directory of the archive. The actual name of the log-file depends on the OS of the builder.

Bug Fixes

• Change Packer version back to 1.4.2. Packer 1.4.3 frequently has issues setting metadata after a build completes. Packer 1.4.4 is not expected until October, so this change is needed now to fix the broken builds.

• In the PyPI merge template, add cron parameter to support daily build and push to a staging repo, like the maven merge template. In PyPI release templates, change name of gerrit and github trigger file patterns parameter. This avoids accidental overriding by jobs that limit their actions to subdirectories. The release file patterns are hardcoded in a shell script. Remove params from RST doc. In all PyPI templates, add disabled option and disable-job parameter to be consistent with other python templates.

v0.44.0

New Features

• Add an additonal Sonar job that allows the caller to provide a builder that runs prior to the Sonar scan.

• Add template to update OpenStack cloud images.

• This job finds and updates OpenStack cloud images on the ci-management source repository.

• The job is triggered in two ways:

• 1. When a packer merge job completes, the new image name created is passed down to the job.

  2. Manually trigger the job to update all images.

• When the job is triggered through an upstream packer merge job, this only generates a change request for the new image built.

• When the job is triggered manually, this job finds the latest images on OpenStack cloud and compares them with the images currently used in the source ci-management source repository. If the compared images have newer time stamps are all updated through a change request.

• This job requires a Jenkins configuration merge and verify job setup and working on Jenkins.

• New templates to build and push Python source and binary distributions to a PyPI server. Includes: {project-name}-pypi-verify-{stream}, gerrit-pypi-verify, github-pypi-verify, {project-name}-pypi-merge-{stream}, gerrit-pypi-merge, github-pypi-merge, {project-name}-pypi-release-verify-{stream}, gerrit-pypi-release-verify, github-pypi-release-verify, {project-name}-pypi-release-merge-{stream}, gerrit-pypi-release-merge, github-pypi-release-merge,

Upgrade Notes

• Packer merge jobs have a new build parameter when checked also updates the cloud image.

• lf-infra-packer-build macro now requires 1 new variables to be passed.

• 1. update-cloud-image: Set to true when images need to be updated on Jenkins.

Bug Fixes

• Changed the trigger to run sonar from stage-release to run-sonar. This makes it more concistent with the other parts.

• Builders may have diffrent pyenv versions installed. Programically pick the latest pyenv version. Since we change pyenv version when building images, we do not know which pyenv version are avaliable.

• Run WhiteSource scan jobs weekly on Sunday.

• Pip install pyenv from python2 should force more-itertools to 5.0.0 In a fresh python2.7 venv “pip install pyenv” correctly pulls down more-itertools [required: Any, installed: 5.0.0] If for some reason a higher version is already installed this will downgrade more-itertools to a py2 compatible version

• Allow java-opts to be defined in WhiteSource scans. This avoids java heap failures.

v0.43.1

Upgrade Notes

• CONTAINER_PULL_REGISTRY and CONTAINER_PUSH_REGISTRY need to be defined in Jenkins global environment varaibles.

Bug Fixes

• Pin python-cinderclient to 4.3.0.

  A new version of python-cinderclient 5.0.0 is released which breaks the openstack jobs.

• openstack –os-cloud vex limits show –absolute Error: No module named v1.contrib

• Debug info shows: File “/home/jenkins/.local/lib/python2.7/site-packages/openstackclient/volume/client.py”, line 40, in make_client from cinderclient.v1.contrib import list_extensions ImportError: No module named v1.contrib

• Update echo commands in release-job.sh for easy identification.

• Update release-container-schema to use CONTAINER_PULL_REGISTRY and CONTAINER_PUSH_REGISTRY from Jenkins global variables. These will be used to pull and push operations for self release containers. Can be overwritten in the releases files.

• Fix warnings from tox (shellcheck).

• Remove quotes from optional var WSS_UNIFIED_AGENT_OPTIONS. When empty, the quotes will cause WS Unified Agent failures.

• Upgrade WS Unified Agent CLI to latest version 19.8.1

v0.43.0

New Features

• Add python tox merge job templates for gerrit and github. These templates are triggered by merge events to test the Python code on the branch that received the merge. Renamed tox-verify macro to tox-common.

Bug Fixes

• Import GPG signing key in release jobs before verifying Gerrit tag details.

• INFO validate job checks that repositories matches $PROJECT Catches projects that replace / with - in their INFO file Also ensures that repositories only has one entry. We are not supporting multiple projects with a single INFO.yaml file.

• Remove unused the MAVEN_CENTRAL_URL variable. The self-release job is designed to work with any Nexus repository info published in staging-repo.txt.gz, which makes the MAVEN_CENTRAL_URL redundant, hence remove the unused variable.

• Revise tox-install.sh script to invoke python using environment variable PYTHON instead of hardcoding “python”, and remove the pip –quiet flag. Extend the RTD template to pass python-version, default python2.

• Use existing builder lf-infra-maven sonar, drop incomplete builder lf-tox-maven-sonar, to gain desired behavior of pushing code analysis results to Sonar. Use trivial goal ‘validate’ by default. The script maven-sonar.sh calls maven twice, first with build goals and then with Sonar goals. The incomplete builder did not supply the build goals.

• Release schema verification needs to happen first before we attempt to assign values to the variables. Validate version only after the schema validation has passed and the variables are assigned.

• Organize variable setup into functions. Maven release files expects different variables than container release files.

• Rename “version” variable in container release files to “container_release_tag” which is a better user friendly name given the fact that container versions are rather called tags. Internally, we still process it as “version” to allow reuse of the tag function.

v0.42.1

New Features

• Add DRY_RUN build param to do a test run the job with publishing artifacts.

Upgrade Notes

• Update lftools version to v0.26.2.

Bug Fixes

• Verify both repos before attempting release. We have run into a case where the repo on ODL nexus was good, and the repo on Sonatype nexus was missing. Cover this case by running the verify loop over each repo before attempting release.

v0.42.0

New Features

• Add support for distribution_type “container”

• Add function maven_release_file and container_release_file and the logic to choose the correct one. No functional change to maven_release_file.

• Add docker login step when docker releases are being processed.

• container_release_file downloads log_dir/console.log.gz and parses it to get a list of container name and version. Verifies pulls container and grabs the image_id then performs the merge then tags and pushes the container.

• Add lf-sonar-common job-template to lf-ci-jobs.yaml and add lf-infra-sonar to macros.yaml. The purpose of a new job template is to adopt using jenkins sonar plug-in along with the sonar-project.properties file versus pom.xml. Lastly the new job template will ensure that anything using lf-infra-tox-sonar is unaffected.

• Add support for “Build with Parameters” for projects that do not want to use a release file for maven builds.

Upgrade Notes

• release-verify and merge will need to run on a docker build-node for example centos7-docker-8c-8g Lftools will need to be updated to 0.26.0 so that -v is supported for lftools nexus release

• Update lftools version to v0.26.1.

Bug Fixes

• Fix missing extension in ID for release-schema.yaml.

• Make “distribution_type” mandatory in future release files.

• Rename “RELEASE_FILE” parameter to “USE_RELEASE_FILE” in release-jobs. This will match the actual varaible default value better and will not collide with the local “release_file” in the script.

• Fix “USE_RELEASE_FILE” if statement. We are now using a bool instead of a string. Changing the if statements to evaluate bools.

• Add pre-build-script parameter to python clm, tox and sonar templates. Gives flexibility to install prerequisite libraries, rearrange the source tree, etc.

• Pin more-itertools ~= 5.0.0, since version 6.0.0 requires Python 3.4 https://github.com/erikrose/more-itertools/releases

  Error: more-itertools requires Python ‘>=3.4’ but the running Python is 2.7.5

• Restructure shell/release-job.sh into functions.

v0.41.0

Upgrade Notes

• Update lftools version to v0.26.0.

• Update packer version to 1.4.3. This new packer version fixed an issues in docker image, where its unable to install the packages into docker containers due to checking of wrong container OS. Fix in 1.4.3: builder/docker: Check container os, not host os, when creating container dir default [GH-7939]

Bug Fixes

• Project job sections that define gerrit_trigger_file_paths are overriding ones set in Default parameters of lf_release_common Hard Code gerrit_trigger_file_paths to fix this.

v0.40.4

Upgrade Notes

• Update lftools version to v0.25.4.

Bug Fixes

• Git fetch needs the dashed version git fetch “$PATCH_DIR/${PROJECT////-}.bundle”. Fix if statement.

• Release creds are only required for promoting the repo, which uses diff ACL as compared to normal user. Therefore dont use the release creds for the verify jobs and the scm sections in both the job templates.

• 1. The release merge job is a one way operation. Given this, Release jobs should only exist in the format {project-name}-release-verify and {project-name}-release-merge As these jobs trigger from a change to any branch/** These jobs Must Exit 0 if release job has already tagged a repo. Inevitably a release file will be pulled in from master to branch or a remerge will be requested This will again trigger the release jobs. since in this case the repo is already tagged, the job should not report a failure. This is solved by having the verfiy and merge exit 0 when the repo is already tagged 2. Rather than use project as defined in the release file use ${PROJECT////-} This changes PROJECT=”optf/osdf/foo/bar” to optf-osdf-foo-bar so that we can fetch the log files. by changing /’s in the project names to -‘s

v0.40.3

Known Issues

• Update release job template to tigger on any branch name, and not just ‘master’. ODL projects branches are version ‘4.0.x’ which requires passing the branch name to the template.

Bug Fixes

• Fix the release job script to handle any trailing ‘/’ set on log_dir and also handle unbound variables correctly.

• Update lftools version to v0.25.3.

v0.40.1

Upgrade Notes

• Projects using lf-release-job will need to add the project’s signing public key in their Jenkins Settings Files.

Bug Fixes

• Use {GERRIT_PROJECT} when calling Gerrit in release-merge job.

• Project pattern was incorrectly set to ** must be {project}

• The self-release jobs does not handle multiple repositories listed in staging-repo.txt file. This fixes the issue by deriving the NEXUS_URL and the STAGING_REPO from each entry in the file. This approach also eliminates the need for having multiple release.yaml files for every staging-repo.

• Allow lf_release_verify and lf_release_merge to verify tag signature.

v0.40.0

New Features

• Add packer image $NAME to the description setting so that the image name is displayed above the job logs URL. This saves a some time from looking for the image name in the jobs logs.

• Allows projects to promote their own builds. Requires setup of accounts and permissions in Gerrit, Jenkins and Nexus. Please refer to the lf-release-jobs documentation for details.

• Remove orphaned ports from the openstack cloud environment. These orphaned ports are residue of CSIT jobs that needs to be purged, as a part of the openstack cron job. A large number of stale jobs could cause IP address allocation failures.

• Enable JAVA_HOME to point to openjdk12 install path for CentOS 7.

Upgrade Notes

• Consolidated lf-infra-jjbini macros with JJB 2.0. This requires renaming any Jenkins managed files “jjbini-sandbox” to “jjbini” to switch to the format supported in JJB > 2.0.

• Projects using lf-release-jobs need to make sure they have the global variable NEXUSPROXY added in Jenkins production and Jenkins sandbox servers. The value of this variable should be the URL to the project’s Nexus server. Previous commit 118b7cbf171aca498d1a0a3a485bad990ad2e7b6 missed this variable.

• Projects using lf-release-jobs need to make sure they have the global variable NEXUSPROXY added in Jenkins production and Jenkins sandbox servers. The value of this variable should be the URL to the project’s Nexus server.

• This change will require to update lf-release-job calls. Update from using “{project-name}-releases-merge-{stream}”, “{project-name}-releases-verify-{stream}” to “{project-name}-release-merge-{stream}”, “{project-name}-release-verify-{stream}”. No upgrade need to be done if using “{project-name}-gerrit-release-jobs” group.

Bug Fixes

• There is no way on finding out the $JOB_NAME pushed to sandbox with the jjb-deploy command in the logs. The change outputs the $JOB_NAME to the logs, which is useful for debugging purposes.

• Add release-schema used to validate the releases yaml file as part of lf-release-jobs.

• Tarball the $JAVADOC_DIR as a workaround for javadoc verfiy jobs to avoid uploading a large number of small files. Uploading a large number of small files does not work well with Nexus unpack plugin which fails on 504 gateway timeout.

• Allow maven goals to be configured in sonatype-clm.sh. Set to “clean install” by default.

• Allow maven goals to be configured in maven-sonar.sh. Set to “clean install” by default.

• Add support for JAVA_OPTIONS in sonar job. Some of the maven build options in the sonar job require to set the JAVA_OPTIONS to specific value for the build to pass This option will help to pass the JAVA_OPTIONS from the template.

• Perform “lftools schema verify” command to validate the release files against schema/release-schema.yaml Obtain optional maven central URL inside the loop that scans release files.

• Allow only semantic release versions like “v${SEMVER}” or “${SEMVER}”. Fail the script if the version is not valid. Do not append any additional characters to the release version during tag and push steps.

• Delete stacks with the --force option to ensure that any delete failures does not stop the openstack-cron jobs from continuing.

• Download raw version of release-schema.yaml to compare against release files using lftools.

• Avoid the usage of project specific variables. Do not use ODLNEXUSPROXY var, but instead use a generalized variable.

• Avoid the usage of project specific variables. Do not use ODLNEXUSPROXY var, but instead use a generalized variable.

• Using “releases” and “release” in different places is becoming confusing. Standardize to “release” to match lftools command and the majority of the exisiting wording.

  Use “releases” for the list of tech team releases and trggers since it is intuitive there. For example “releases/1.1.1.yaml”

• Move info-schema to schema/info-schema.yaml to keep schemas consistency.

• Download only needed files for lf-info-yaml-verify rather than cloning the entire repo.

• Allow lf-maven-stage jobs to be triggered using either “stage-release” or “stage-maven-release”.

• Allow lf-maven-docker-stage jobs to be triggered using either “stage-release” or “stage-docker-release”.

• Upgrade to the WhiteSource Unified Agent version 19.7.1.

v0.39.1

Bug Fixes

• Update lftools version to v0.25.2.

v0.39.0

Critical Issues

• lftools v0.24.0 introduced a major issue which caused the Jenkins cloud configuration merge job for OpenStack clouds to fail. This has been corrected in lftools v0.25.1

Bug Fixes

• Add missing git-url variable in {project-name}-releases-merge-{stream} job.

• Add the {stream} name in releases-verify and releases-merge jobs.

• Some ONAP components like DCAEGEN2 do not host a version.properties file in the root of their repos. We need to be able to provide a location and/or different name for the version.properties file for jobs using the lf-maven-versions-plugin builder step.

v0.38.4

New Features

• Group {project-name}-releases-verify and {project-name}-releases-merge into {project-name}-gerrit-release-jobs.

  Add test jobs for lf-release-jobs.

Bug Fixes

• Change parameter names used to specify container tag method, with a new default to use the fixed string ‘latest’ as a tag. Merge two shell scripts into one instead of using Jenkins conditional steps. Extend to accept a custom directory for the container-tag.yaml if the yaml-file method is used to set the docker tag information; this is an optional variable which is set to empty by default, and falls back to DOCKER_ROOT.

• Add missing scm block in gerrit-releases-merge job definition. Add missing submodule-disable variable for jobs using lf-infra-gerrit-scm. Update documentation for gerrit-releases-merge and gerrit-releases-verify to remove submodule options as optional parameters.

• Update lftools version to v0.25.0

• Add missing $ to variable tag_file so the yq query can pull the tag from the container-tag.yaml file.

• Add -l to /bin/bash shebang line at top of docker-get-container-tag.sh to make it a login shell, which automatically includes /home/jenkins/.local/bin on the path, because that is where pip installs the yq command.

v0.38.3

Bug Fixes

• Add trigger on cron to docker merge macro to support regular rebuilds. This makes the merge macro match the behavior of most other jobs.

• Add yamllint verification to INFO.yaml files.

Other Notes

• Update lftools version to v0.24.0.

v0.38.2

Bug Fixes

• Add missing config for triggering on file paths to docker macros and templates, namely gerrit_trigger_file_paths and github_included_regions, to make the verify and merge macros and templates match the behavior of other jobs.

v0.38.1

Bug Fixes

• Install yq to be used to read yaml files. In specific, it will be needed to read container-tag.yaml.

• When calling builder step macros “lf-docker-get-container-tag”, “lf-docker-build” and “lf-docker-push”, make sure the needed variables are also passed explicitly to avoid these variables appear as undefined.

• Allow DOCKER_ARGS to be empty in docker-build.sh. This is not a required parameter, it can be empty.

  Remove container reference in docker-get-git-describe.sh. The CONTAINER_PUSH_REGISTRY already gets added in the docker-push script. No need to add it again.

  Rename image_name to image_build_tag in docker-get-yaml-tag to match docker-get-git-describe. Add missing “DOCKER_NAME” in the DOCKER_IMAGE.

  docker-get-git-describe.sh and docker-get-yaml-tag.sh should only export the tag variable. Let docker-build.sh process DOCKER_NAME

Other Notes

• Add yq install as part of python-tools-install.sh Allow future scripts to use yq package.

v0.38.0

New Features

• gerrit-tox-verify now has a new parameter gerrit-skip-vote (bool) to control whether Jenkins should skip voting depending on the build outcome. It defaults to false since it is the default used by the Jenkins Gerrit Trigger Plugin.

• gerrit-docker-verify runs for new commits and runs a build of the affected Docker images.

• gerrit-docker-merge runs for merged commits, runs a build of the affected Docker images and pushes the images to a specified Docker registry.

• New lf-release-job-merge and lf-release-job-verify templates allow projects to have self-serve releases. Project will create a tagname.yaml file in the releases/ directory of their git repo. example:

  $ cat releases/4.0.0.yaml
  ---
  distribution_type: 'maven'
  version: '4.0.0'
  project: 'odlparent'
  log_dir: 'odlparent-maven-release-master/11/'
  #below is optional
  maven_central_url: 'oss.sonatype.org'
  

• lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-disable parameter (bool) to control whether submodules are ignored or not during git fetch operations.

• All job-templates now provide an optional submodule-disable parameter for git fetch operations, defaulting to false.

Upgrade Notes

• Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-disable value. It is recommended to default this value to false since it is the default used by the Jenkins Git Plugin.

• Update gerrit comment trigger to use a more standard regex and avoid triggering jobs, when these keywords are intended to be used as code review comments between users. Also improve the regexs to make them more succinct and readable.

Bug Fixes

• Add missing config for triggering on file paths to maven stage macros and templates, namely gerrit_trigger_file_paths nad github_included_regions, to make those macros and templates match the behavior of maven verify and maven merge.

• fix multiple jobs created using same job-template update same github check status due to hard coded status-context to Maven Verify. Now appending status-context with maven-version and java-version to make it unique. And create different status checks in the github. fix applied for maven verify and maven docker verify jobs

• Fix log shipping script to not require a LOGS_SERVER. There was a regression that caused the log shipping script to start requiring a LOGS_SERVER which fails in the case of a system that does not have that optional environment variable set.

• Handle multiple search extension or patterns passed by upstream JJB ARCHIVE_ARTIFACTS param as a single string by spliting these values before being passed to lftools deploy archives.

  ARCHIVE_ARTIFACTS="**/*.prop \
                    **/*.log \
                    **/target/surefire-reports/*-output.txt \
                    **/target/failsafe-reports/failsafe-summary.xml \
                    **/hs_err_*.log **/target/feature/feature.xml"
  

  For example, the above env variable passed to the script and to lftools deploy archives as:

  lftools deploy archives -p **/*.prop \
                    **/*.log \
                    **/target/surefire-reports/*-output.txt \
                    **/target/failsafe-reports/failsafe-summary.xml \
                    **/hs_err_*.log **/target/feature/feature.xml \
                    "$NEXUS_URL" \
                    "$NEXUS_PATH" \
                    "$WORKSPACE"
  

  The correct way of passing this as per lftools implmentation is:

  lftools deploy archives -p '**/*.prop' \
                    -p '**/*.log' \
                    -p '**/target/surefire-reports/*-output.txt' \
                    -p '**/target/failsafe-reports/failsafe-summary.xml' \
                    -p '**/hs_err_*.log' \
                    -p '**/target/feature/feature.xml' \
                    "$NEXUS_URL" \
                    "$NEXUS_PATH" \
                    "$WORKSPACE"
  

• Fix error with handling unbound arrays for search extensions, when using set -u. The correct way of using this.

  set -u
  arr=()
  echo "output: '${arr[@]}'"
  bash: arr[@]: unbound variable
  echo "output: '${arr[@]:-}'"
  foo: ''
  

• lf-maven-versions-plugin builder step needs to run before maven-patch-release.sh as this second script contains a condition to confirm if the maven vesions plugin was selected as a way to remove the ‘SNAPSHOT’ pattern from the pom.xml files. lf-maven-docker-stage was based on lf-maven-stage and it seems that these particular builder steps were switched in place accidentally.

• The logs-deploy.sh script now allows ARCHIVE_ARTIFACTS to contain zero or more files.

• request-2.22.0 does not work with python-3.4.9, so pin requests to v2.21.0 to address the tox failures.

v0.37.2

Bug Fixes

• Use maven goal install (not deploy) in the maven + docker verify job. An image cannot be pushed by a verification job, and the deploy target directs the plugin to push.

v0.37.1

Bug Fixes

• Remove maven-versions-plugin-set-version variable in newly added macro. This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version”.

v0.37.0

New Features

• Add verify, merge and stage templates for Java projects that build and wrap a JAR (e.g., a Spring-Boot application) inside a Docker image, and do not need to deploy any JAR libraries or POM files.

Upgrade Notes

• The next release of common-packer will require a minimum version of 1.3.2 for packer. The current release of packer is 1.4.0.

Bug Fixes

• The packer-merge job for Gerrit systems was improperly configured to use the Gerrit Trigger choosing strategy and not default. This caused issues unexpected issues with retriggering merged changes when the expectation was that it would pick up the lastest change as per normal.

• This is a variable that does not need to be defined by the users of the jobs. The version needed in this builder step is inherited from versions.properties as “release_version” and it is fixed as that. This also helps teams not having to define this version in 2 places and just rely on version.properties.

• Projects using maven versions plugin let this plugin take care or updating their versions in the pom.xml. When maven-versions-plugin is set to “true”, skip the stripping of SNAPSHOTS from the pom.xml files. maven-versions-plugin is set to “false” by default.

Other Notes

• Update example Jenkins Init Script in README to redirect all output to a log file.

v0.36.0

Prelude

WhiteSource is a security and license compliance management platform. It is used to perform scans on a great variety of coding and scripting languages.

New Features

• New comment-to-gerrit builder will comment back to gerrit patchset if a file called gerrit_comment.txt is created by the build.

• Allows maven to run a clean install step before the WhiteSource scan runs the Unified Agent to fetch additional dependencies. Set to false by default.

• Job {project-name}-whitesource-scan-{stream} uses the WhiteSource Unified Agent scanner CLI tool to perform the code scan and report the results into the WhiteSource dashboard.

Bug Fixes

• Tag releases will now trigger a docs build to regenerate and update the release note link.

• Update jenkins-cfg-verify job to validate new images names obtained from $GERRIT_REFSPEC instead of the master branch.

• Hardcode project version to the “GERRIT_BRANCH”. Follow previous convention from CLM where reports were versioned after the branch name. Fix minor nits with bash varaibles.

• wss-unified-agent.config file should not be opened for configuration to tech teams. The config file should be part of Jenkins Settings Files and called via Managed Files. wss-unified-agent.config must be created in Jenkins config files based on wss-unified-agent.config.example.

Other Notes

• To run this job, a configuration file is needed (wss-unified-agent.config.example). A new secret text credential will need to be created. (ID=wss-apiKey Secret=WhiteSource organization API key)

• Update lftools version to v0.23.1.

v0.35.0

New Features

• The jjb-merge job now has a new parameter jjb-workers to allow configuration of the number of threads to run update with. Default is 0 which is equivalent to the number of CPU cores available on the system.

• New info-vote-verify macro Will count votes against an INFO.yaml change and sumbit automatically if a majority of committers vote +1 or +2 on the change. Job is triggered by +2 votes or a comment of “vote”

Other Notes

• The Maven Verify job will now call -Dmaven.source.skip to skip source jar generation in the verify job. This saves us some time in the verify build as the source artifacts are not useful in a verify job.

v0.34.0

New Features

• jenkiins-init-scripts The ‘ciman’ repo is not longer required to be located in ‘/opt/ciman’.

• lf-maven-set-version conditional step for lf-maven-stage to allow teams to run Maven versions plugin to update their artifact versions. Step will run if maven-versions-plugin is set to true.

• Support for the Throttle Plugin is added to JJB jobs so static build servers can restrict the number of concurrent JJB jobs ran at the same time.

  This must be explicitly enabled by setting throttle-enabled on the jobs.

Bug Fixes

• Adapt maven path search for files and dirs. The “-f” maven param can specify both a directory, in which case it will look for “pom.xml” in the directory, or a specific file. The original version of this search was only compatible with directories that contain a pom.xml file.

• Update the lf-maven-cental macro documentation and example templates with the missing requireed params.

• Fix JAVA_HOME for openjdk11 on CentOS 7 to use the OpenJDK version installed in /usr/lib/jvm/java-11-openjdk.

• The JJB Deploy Job is configured to trigger only if the Gerrit comment starts with the jjb-deploy keyword.

  Without the regex being optimized the job triggers on any occurance of the jjb-deploy keyword in a Gerrit comment, with is waste infra resources.

  Example of a valid command in Gerrit comment that triggers the job:

  jjb-deploy builder-jjb-*

  Example of a invalid command in Gerrit comment that would _not_ trigger the job:

  Update the job. jjb-deploy builder-jjb-*

v0.33.0

New Features

• jenkins-init-scripts If the environmental variable ‘SWAP_SIZE’ is set when the ‘init.sh’ script is called, then a ‘SWAP_SIZE’ GB swap space will be configured. If ‘SWAP_SIZE’ is ‘0’ or is not a valid integer, then no swap space is configured. If it is unset then 1GB of swap will be configured. Previously the swap size was fixed at 1GB.

• jenkins-init-scripts If the work directory or volume (/w) aleady exists, the ownership will be recursivly set to ‘jenkins:jenkins’. Previously only the top directory /w was owned by ‘jenkins:jenkins’

• lf-sigul-sign-dir macros now supports a sign-mode parameter which allows jobs to choose to sign artifacts using either parallel mode or serial mode (default).

Upgrade Notes

• lf-sigul-sign-dir users need to add a new parameter sign-mode to their job-templates setting either parallel or serial as the value, we recommend setting serial mode for this setting.

  {project-name}-maven-stage-{stream}’s Sigul signer now defaults to serial mode instead of the previous parallel behaviour. To change this back to the previous behaviour pass the “sign-mode” parameter to the job template:

  - project:
      name: parallel-sign
      jobs:
        - gerrit-maven-stage:
            sign-mode: parallel
  

v0.31.0

New Features

• New job-template {project-name}-release-announce for lf-releng projects to automate release announcement emails.

• Add support for pushing Sonar results to SonarCloud. Refer to Maven Sonar docs for details.

Upgrade Notes

• Jobs using the lf-maven-stage macro now need to update to the new usage. Preparation calls to lf-provide-maven-settings, lf-infra-create-netrc, and lf-provide-maven-settings-cleanup are no longer necessary to prepare the lf-maven-stage macro.

  Usage:

  - lf-maven-stage:
     mvn-global-settings: 'global-settings'
     mvn-settings: 'settings'
     mvn-staging-id: 'staging profile id'
  

v0.30.0

New Features

• Packer merge jobs now include the image name in the Jenkins build description.

Bug Fixes

• Extend ${JOB_NAME} to include {java-version} parameter to support jobs to build with multiple versions of openjdk{8,11}.

• Modified lf-maven-jobs.yaml sonar cron entry to ‘{obj:cron}’ to pass value from custom user config file.

v0.29.0

New Features

• Add a puppet-verify job to lf-ci-jobs. This job will perform Puppet linting on the specified repository.

  - project:
      name: lf-infra-puppet-mymodule
      project-name: lf-infra-puppet
      project: puppet/modules/mymodule
      jobs:
        - gerrit-puppet-verify
  

Bug Fixes

• maven-fetch-metadata.sh was not respecting the “-f” (for file path) flag in MAVEN_PARAMS, causing lf-maven-merge jobs that utilize this flag to fail. It will now set a path based on this flag if it is present, or default to the current working directory.

• Check openjdk $VERSION before setting $JAVA_HOME. This enables jobs to pass “openjdk10” or “openjdk11” on CentOS 7 images to use the OpenJDK version installed in /opt.

v0.28.3

Bug Fixes

• Compress and upload all jjb-verify XML files to Nexus, to ease out the IOPs on cron jobs that manage the logs on Nexus and optimize job performace by ~8 mins. This is because the job generates around ~2.3K XML files (small files) which is uploaded to Nexus in every run of jjb-verify. Doing this is faster as compared to the Nexus Unpack plugin in the Nexus end unpacking the zip file we upload takes longer.

v0.28.1

Other Notes

• Update lftools version to v0.19.0.

v0.28.0

New Features

• New lf-stack-create macro allows job-templates to setup a OpenStack Heat stack, useful for spinning up CSIT labs to run integration tests against. Use with the lf-stack-delete macro.

• Concurrency for the gerrit-jjb-verify job can now be configured by setting the ‘build-concurrent’ parameter.

• New macro lf-maven-central is available to deploy artifacts to OSSRH staging for jobs that want to eventually deploy to Maven Central.

  ---
  - job-template:
      name: lf-maven-central-macro-test
  
      #####################
      # Default variables #
      #####################
  
      mvn-central: true
      mvn-global-settings: ""
      mvn-settings: ""
      ossrh-profile-id: ""
  
      #####################
      # Job configuration #
      #####################
  
      builders:
        - lf-maven-central:
            mvn-central: "{mvn-central}"
            mvn-global-settings: "{mvn-global-settings}"
            mvn-settings: "{mvn-settings}"
            ossrh-profile-id: "{ossrh-profile-id}"
  

• The GERRIT_REFSPEC build parameter can now be used to trigger a test build from the Jenkins Sandbox system against a work in progress packer image patch from a GitHub Pull Request.

Upgrade Notes

• lf-stack-delete has been modified to be a companion macro to lf-stack-create in order to cleanup the stack at the end of a job run. It now includes a required parameter openstack-cloud to choose the clouds.yaml cloud configuration for the project. Existing users of this macro will need to update their job templates accordingly.

• Requires JJB 2.8.0 for the jenkins-sandbox-cleanup job to not fail.

  Note

  Despite the failure if JJB 2.8.0 is not available the job will successfully delete all jobs and views, the primary purpose of this job.

Bug Fixes

• RELENG-1450 All view disappears on Jenkins Sandbox after views are deleted. The All view is now recreated after delete-all is run.

v0.27.0

New Features

• Add the ability to configure the location of JJB’s cache directory for CI jobs.

• New view-templates project-view, common-view, and csit-view are available for projects to manage Jenkins views through code.

  To use the project-view template in a project:

  - project:
      name: aaa-view
      views:
        - project-view
  
      project-name: aaa
  

  To use the common-view template in a project:

  - project:
      name: daily-builds
      views:
        - common-view
  
      view-name: Periodic
      view-regex: '.*-periodic-.*'
  

  To use the csit-view template in a project:

  - project:
      name: csit
      views:
        - csit-view
  
      view-name: CSIT
      view-regex: '.*csit.*'
  
  - project:
      name: csit-1node
      views:
        - csit-view
  
      view-name: CSIT-1node
      view-regex: '.*-csit-1node-.*'
  

• Add support to maven-stage jobs to publish to Maven Central via OSSRH.

  This is accomplished by adding these 2 new optional parameters to the job configuration.

  - gerrit-maven-stage:
      mvn-central: true
      ossrh-profile-id: 7edbe315063867
  

• The openstack-cron job now has the ability to remove images older than a specified age (default: 30).

• The openstack-cron job now has the ability to remove orphaned servers.

• The openstack-cron job now has the ability to remove orphaned stacks.

• The openstack-cron job now has the ability to remove orphaned volumes.

• lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-timeout parameter to provide a timeout value (in minutes) for git fetch operations.

• All job-templates now provide an optional submodule-timeout parameter for git fetch operations, defaulting to 10 minutes.

Upgrade Notes

• Some LF projects are already using a common-view template in their local ci-management repo. This common-view is called project-view in global-jjb so rename all instances of common-view to project-view when upgrading and remove the local common-view view-template definition from ci-management.

• The openstack-cron job now requires a new parameter configured jenkins-urls in order to use the job.

• Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-timeout value. It is recommended to default this value to 10 since it is the default used by the Jenkins Git Plugin.

Bug Fixes

• The jenkins-init scripts dir is now updated to reflect changes recommended for the v0.25.0 release. We unfortunately missed this critical piece in the v0.25.0 release.

• Specify refspec to be blank for SCM config on github-maven-merge job. Setting the refspec to +refs/pull/*:refs/remotes/origin/pr/* causes there to be no merge job triggered.

• New detox version requires tox >= 3.5 and < 4. Instead of explicitly pulling tox, we are now implicitly pulling via detox. (https://jira.opendaylight.org/browse/RELENG-136)

Other Notes

• lftools’ openstack module will now be installed as part of pre-build.

• The openstack-cron job now runs every hour instead of daily. This is because stack cleanup should happen more regularly.

v0.26.0

New Features

• Add a new nexus-iq-namespace optional parameter to insert a namespace into Nexus IQ AppID. This is useful for shared Nexus IQ systems where projects might have concern about namespace collision.

  Note

  We recommend when using the namespace to add a trailing - to the value. Eg. ‘odl-’, this is to make the namespace look nice for example “odl-aaa” is the result of namespace odl-, and project name aaa.

• Add lf-infra-publish-windows. A publisher for use at the end of Windows based job-templates.

Bug Fixes

• Fix packer-verify job to correctly work with clouds.yaml config model implemented in global-jjb v0.25.0.

v0.25.1

Bug Fixes

• jjb-cleanup.sh may be merged with other shell scripts that set -u which causes Jenkins to fail when activating virtualenvs

v0.25.0

New Features

• Add support to the packer-build job to use clouds.yaml for openstack builder configuration rather than through the cloud-env file. This allows us to simplify the template configuration for openstack builders moving forward.

• New macro lf-sigul-sign-dir available to sign artifacts in a provided directory using Sigul.

  Usage:

  - lf-sigul-sign-dir:
      sign-dir: '$WORKSPACE/m2repo'
  

  This macro also requires a boolean variable to SIGN_ARTIFACTS to be set to true to activate the macro. We recommend the job-template that uses this macro to define it in the job parameters section.

  Example:

  - bool:
      name: SIGN_ARTIFACTS
      default: '{sign-artifacts}'
      description: Use Sigul to sign artifacts.
  

• Add Sigul signing support to the maven-staging job. To activate Sigul signing make sure to set sign-artifacts: true. Example:

  - project:
      name: abc
      jobs:
        - gerrit-maven-stage
  
      sign-artifacts: true
  

• Add lf-stack-delete macro to delete an openstack heat stack at the end of the job.

  This macro requires a parameter defined in the job named STACK_NAME containing the name of the stack to delete.

• Add lf-infra-wrappers-windows to handle Windows specific wrapper configuration.

• Refactor lf-infra-wrappers to be for Linux systems and split out the non-linux specific components into a new lf-infra-wrappers-common. This change is seamless for current users of lf-infra-wrappers.

Upgrade Notes

• Upgrade to global-jjb v0.24.6 before performing this upgrade. This ensures that jjb-verify job pulls in a regex fix that will allow it to verify the v0.25.0 upgrade.

• Global JJB now has non-JJB YAML configuration and requires action on the ci-management repo when upgrading to this version of Global JJB to prevent JJB from picking up these YAMLs as config. Follow the instructions below BEFORE upgrading globall-jjb:

  cd <git-root>
  git mv jjb/global-jjb global-jjb
  mkdir jjb/global-jjb
  ln -s ../../global-jjb/shell jjb/global-jjb/shell
  ln -s ../../global-jjb/jjb jjb/global-jjb/jjb
  git add jjb/global-jjb
  git commit -sm "Prepare repo for global-jjb v0.25.0"
  

• Minimum packer version 1.2.5 is now required for the packer-build job.

• lf-infra-packer-build macro now requires 2 new variables to be passed.

  1. openstack: Set to true if template is built using the openstack builder

  2. openstack-cloud: The clouds.yaml cloud to use when running packer build

Deprecation Notes

• lftools-install.sh is deprecated and will be removed in a future release. We recommend installing lftools via pip install –user lftools to install instead of using this script.

Bug Fixes

• Fix pip install pip setuptools which seems to fail against the Nexus 3 proxy. Run them as separate calls to make things happier.

• jjb-verify will now test on all changes in the jjb directory. The previous pattern was too specific and sometimes missed verifying patches that should be verified.

• Replace jjb-verify to test on all changes in the shell/* directory.

• Fix the lftools virtualenv workaround we had to put in place in the tox-verify job by using pip install --user for global tool installs.

• Fix jobs failing with UNSTABLE build due to install pip==18.0 missing. This change moves all the jobs to using lf-infra-pre-build to install lftools via –user command.

• Use python -m pip to ensure that we are using the pip version that was installed rather than the OS wrapper version of pip.

• Fix package listing script in post-builder from causing UNSTABLE build due to difference in the two files being compared.

• Fix RTD job failing to find PBR install.

Other Notes

• Update lftools to ~ 0.17.1