GitLab Variables in CI/CD pipeline

GitLab pipeline variables are essential components in CI/CD workflows, serving as dynamic placeholders that store and manage data values within the GitLab CI/CD environment. These variables enable users to customize and parameterize their pipeline configurations, allowing for more flexible and reusable automation scripts.

Types of Variable

There are currently two types of variables in gitlab, Variable and File. The regular variable are used to store simple key-value pairs. File type variables are used to store the contents of files as variable, an example could be maven settings.xml, PEM certificate file etc. The file type variables are mostly created at the project, group or instance level.

test_job:
  stage: test
  variables:
    A_FILE_VAR: $MAVEN_SETTINGS_FILE # (eg: MAVEN_SETTINGS_FILE is created at the group level)
    A_REGULAR_VAR: 'simple value'
  script:
    - cat $A_FILE_VAR # to print the content of the file
    - echo $A_REGULAR_VAR # to print the value of the regular variable

How to use an array type of variable

GitLab lacks a specific array-type variable, but a workaround involves storing multiple values separated by a space within a variable. Subsequently, you can use the shell scripting technique to iterate through these values.

job:
  variables:
    FOLDERS: src test docs
  script:
    - |
      for FOLDER in $FOLDERS
        do
          echo "The path is root/${FOLDER}"
        done

User selection of variable value from a set of options

Starting from version 15.7, GitLab has introduced a new variant of regular variable that allows users to choose a value from a predefined set of options. These values in the options are in a dropdown list in the Run pipeline page. Set a default value of the variable and ensure that this default value is also present in the options list.

variables:
  DEPLOY_ENVIRONMENT:
    value: "SIT"
    options:
      - "SIT"
      - "UAT"
      - "CANARY"
    description: "The deployment target. Set to 'SIT' by default." 

Variable at Instance, Group and Project scopes

Gitlab variables can the defined at different scopes. Variables that are applicable to all pipelines within a GitLab instance can be established at the instance level, ensuring their availability across the entire environment. The instance level variables are configured by the system administrators. In addition to user-defined instance-level variables, GitLab provides a range of predefined instance-level variables that are available to all projects and groups in a GitLab instance.

GitLab allows the organization of projects based on the organizational structure, enabling grouping of projects. Within the same group, a group owner can create a shared set of variables at the group level that are applicable to all projects within that group.

Beyond the shared variables established at the group level, individual projects may require unique variables that remain accessible throughout the pipeline. Project maintainers are eligible to create these variables. Additionally, there might be instances where overriding the value of a group-level variable is necessary; this can be accomplished by adding the same variable in the project’s CI/CD variables with a new value.

Masked and Protected Variables

Variables stored at the instance, group, and project levels may include sensitive information such as passwords, and it’s essential to mask these values to prevent them from being displayed in job logs. In situations where passwords contain special characters that hinder GitLab’s masking functionality, an alternative approach is to store passwords as base64 values, allowing it to be masked in Gitlab.

Certain variables, like user credentials for publishing compiled artifacts to repositories such as JFrog Artifactory or Sonatype Nexus, should only be accessible from protected branches or tags. To enforce this restriction, designate the variable as protected, ensuring it won’t be included in the pipeline execution environment unless triggered from these protected branches or tags.

Definition of variables in .gitlab-ci.yml file (pipeline scope)

The variables defined at the instance, group, projects scopes have constant values. However, if there is a need for the pipeline to run with user-input values on each execution, these variables can be defined directly in the .gitlab-ci.yml file as pipeline variables.

variables:
  DEPLOY_ENVIRONMENT:
    value: "SIT"
    options:
      - "SIT"
      - "UAT"
      - "CANARY"
    description: "The deployment target. Set to 'SIT' by default." 
  ENABLE_DAST:
    value: 'true'
    description: "Enable or disable the dynamic application security testing (DAST)"     

When the runner uses service containers to execute a job, it’s important to note that, by default, these service containers can only access variables stored in the .gitlab-ci.yml file. Variables configured through the GitLab UI are not by default accessible in the docker images or service containers. Consequently, it is necessary to explicitly reassign these variables in your .gitlab-ci.yml file for them to be available in the service containers.

variables:
  REASSIGN_VALUE_IN_YAML_FILE: $VALUE_FROM_UI

Variables in a pipeline job

The pipeline variables are common to all jobs in a pipeline and the values of these variables could be overwritten in any pipeline jobs.

variables:
  OVERWRITE_MY_VAR: 'pipeline value'

test_job:
  stage: test
  variables:
    OVERWRITE_MY_VAR: 'Test Job Value'
  script:
    - echo $OVERWRITE_MY_VAR # to print the overrided value

Variables can be explicitly defined within the scope of a job, and these variables remain isolated, inaccessible to other jobs in the same pipeline. Consider a scenario in your pipeline where a stage is dedicated to code quality analysis using SonarQube. If the intention is to terminate the pipeline upon a quality gate failure and the job is initiated from the default/protected branch, it is possible to create a job-specific boolean variable for SonarQube. This variable can then be utilized to set the value of the SonarQube parameter, specifically sonar.qualitygate.wait.

sonar_scan:
  stage: code_quality
  image:
    name: sonarsource/sonar-scanner-cli:latest
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
    WAIT_FOR_CODE_QUALITY_CHECK: true
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - "$SONAR_USER_HOME/cache"
  rules:
    - if: "$CI_COMMIT_REF_NAME =~ /^feature/" 
      variables:
        WAIT_FOR_CODE_QUALITY_CHECK: false    
  script:
    - sonar-scanner -Dsonar.qualitygate.wait=$WAIT_FOR_CODE_QUALITY_CHECK

Variables as Job artifact

Sometimes, there might be a need to transfer an environment variable generated in one job to the following jobs. In such instances, you can save this variable into a .env file and designate this .env file as a dotenv artifact in the pipeline configuration.

build:
  stage: build
  script:
    - echo "ARTIFACT_VERSION=1.0.0" >> version.env
  artifacts:
    reports:
      dotenv: version.env

test:
  stage: test
  script:
    - echo "$ARTIFACT_VERSION"  # Output is: ''
  needs:
    job: build
    artifacts: false

sonar-scan:
  stage: test
  script:
    - echo "$ARTIFACT_VERSION"  # Output is ''
  dependencies: []

publish:
  stage: publish
  script:
    - echo "$ARTIFACT_VERSION"  # Output is: '1.0.0'

As you can see above, you have the option to use the dependencies or needs keywords to manage the jobs that receive the dotenv artifacts.

How to pass variables to a downstream pipeline

In a complex CICD pipeline, you may have to trigger a child pipeline in the same project or in another project and some of the variables from the parent pipeline needs to be sent to the child pipeline. You can use the variables keyword to pass trigger variables to a downstream pipeline.

variables:
  ARTIFACT_VERSION: "1.0.0"

staging-deployment:
  inherit:
    variables: true
  variables:
    ENVIRONMENT: staging
  stage: deploy
  trigger:
    include:
      - local: path/to/deployment-pipeline.yml

The global variable ARTIFACT_VERSION and the job variable ENVIRONMENT are avilable in every job defined in the downstream pipeline. By using the inherit attribute in job definition, it is possible to stop the global variables reaching the downstream pipeline.

staging-deployment:
  inherit:
    variables: false
  variables:
    ENVIRONMENT: staging
  stage: deploy
  trigger:
    include:
      - local: path/to/deployment-pipeline.yml

Conclusion

GitLab CI/CD variables serve as a potent resource for configuring and tailoring your CI/CD pipelines. Whether defined in your GitLab project settings or within the .gitlab-ci.yml file, these variables play a pivotal role in passing data between jobs, configuring services, and influencing the overall behavior of your pipelines.