AAtsushi's Blog
AzureDevOps

Azure DevOps

→ 日本語版を読む

Overview

I took the following Udemy course as I might need to use Azure DevOps at work.

I'll take notes on the important parts.

Learn DevOps: Docker, Kubernetes, Terraform and Azure DevOps

Since I have some familiarity with Docker and Terraform, I worked on the following sections this time:

  • 6 Getting started with Continuous Integration, Deployment & Delivery
  • 7 Learn Azure DevOps - Continuous Integration, Deployment & Delivery Docker
  • 8 DevOps on Azure AKS, Kubernetes Clusters - Docker, Azure DevOps & Terraform
  • 10 Learn Azure DevOps with Boards and Backlogs

Maybe it's because I skipped a lot, but the overall picture of Azure DevOps was hard to understand in this Udemy course.

Articles to Read Before Watching Udemy

Rather than jumping straight into the Udemy course, first get an overview of Azure DevOps here:

CI/CD Process

Continuous Integration

The CI process is as follows:

  1. Code Commit
  2. Unit Tests
  3. Code Quality
  4. Package
  5. Integration Tests

Continuous Deployment

The CI process plus Deploy and Automated Tests:

  1. Code Commit
  2. Unit Tests
  3. Integration Tests
  4. Package
  5. Deploy
  6. Automated Tests

Continuous Delivery

In addition to the Continuous Deployment process, UAT teams and Testing teams give approval, and deployment is made to staging and production environments.

  1. Code Commit
  2. Unit Tests
  3. Integration Tests
  4. Package
  5. Deploy
  6. Automated Tests
  7. Testing Approval
  8. Deploy Next
  9. ...

Tools Used in CI/CD

  • Code Commit
    • GitHub
  • Unit Tests
    • PyTest
  • Integration Tests
    • Cucumber, Selenium, Protractor
  • Package
    • pip
  • Deploy
    • Jenkins, Azure DevOps
  • Automated Tests (Smoke test, Load test, Performance test)
  • Testing Approval
  • Deploy Next

Pipelines in Azure DevOps

https://stackoverflow.com/questions/58575016/what-is-the-difference-between-pipeline-and-release-pipeline-in-azure-devops

  • In Azure DevOps, there are Pipelines and Releases under Pipelines.

  • Pipelines refers to build pipelines, used to generate artifacts from source code. For example, used to generate Docker images. Unit tests and integration tests are also automated and executed.

  • On the other hand, Releases deploys artifacts generated by the build pipeline to a stage. For desktop applications, for example, this means the installation process.

  • The Azure DevOps Server provides two different types of pipelines to perform build, deployment, testing and further actions.

  • A Build Pipeline is used to generate Artifacts out of Source Code.

  • A Release Pipeline consumes the Artifacts and conducts follow-up actions within a multi-staging system.

Build Pipeline

Build Pipeline Creation Flow

https://learn.microsoft.com/en-us/azure/devops/pipelines/create-first-pipeline?view=azure-devops&tabs=java%2Ctfs-2018-2%2Cbrowser

  1. Create a GitHub repository to use as the Repository
  2. Specify the created repository in Azure DevOps Pipelines
  3. Create the pipeline definition YAML

Build Pipeline Definition

Describe the CI/CD process in the pipeline YAML file:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

The YAML file defines items such as:

  • trigger

    • Trigger for continuous integration
    • In the example above, main is specified for trigger. In this case, this pipeline runs when any change is made to the main branch.

  • pool

    • The pool where this pipeline's jobs run unless otherwise specified
    • The computing used to run the pipeline is called an Agent
    • pool defines the agent pool
    • In the example above, Ubuntu is specified as the VM image to use in the agent pool
    • Agent pools can be configured as per the link above

  • steps

    • The list of steps to execute in this job

  • script

    • Describes the script to be executed

Build Pipeline Structure

Pipelines can define tasks in the following hierarchy.

Tasks can be defined even without stages or jobs being defined.

Pipeline └── Stage └── Job └── Task (Step)

Defining Jobs

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Below, two jobs named Job1 and Job2 have been added.

When run in Azure DevOps Pipeline, the status of each of Job1 and Job2 is displayed.

Each job runs on a different agent (VM). Unless there are dependencies between jobs, each job runs in parallel.

jobs:
- job: Job1
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'
  - script: |
      echo Add other tasks to build, test, and deploy your project.
      echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

- job: Job2
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'

Defining Dependencies Between Jobs

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#dependencies

Dependencies between jobs can be defined by specifying jobs in dependsOn:.

Multiple job dependencies can also be specified.

In reality, you build infrastructure with Terraform first and then install apps. In such cases, jobs are executed in the specified order.

Dependencies between stages can also be defined by specifying stage names in dependsOn:.

jobs:
- job: Job1
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'
  - script: |
      echo Add other tasks to build, test, and deploy your project.
      echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

- job: Job2
  dependsOn: Job1
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'

- job: Job3
  dependsOn: Job1
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'

- job: Job4
  dependsOn:
  - Job2
  - job3
  steps:
  - script: echo Hello, world!
  displayName: 'Run a one-line script'

Defining Stages

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/stages?view=azure-devops&tabs=yaml

Stages can be defined as follows:

stages:
- stage: Build
  jobs:
  - job: FirstJobs
    steps:
    - bash: echo Build,FirstJob
- stage: DevDploy
<omitted>
- stage: QaDevloy
<omitted>
- stage: ProdDeploy
<omitted>

Running this displays the status of each stage in the Azure DevOps pipeline screen.

Defining Steps

https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps?view=azure-pipelines#list-types

Steps support various types in addition to script, such as bash and pwsh. See the link for types.

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: echo This runs in the default shell on any machine
- bash: |
    echo This multiline script always runs in Bash.
    echo Even on Windows machines!
- pwsh: |
    Write-Host "This multiline script always runs in PowerShell Core."
    Write-Host "Even on non-Windows machines!"

Defining Variables

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch

Variables can be defined in the console, pipeline definition, or task scripts.

Defining Variables Within the Pipeline Definition

Variables can be defined within the pipeline definition as follows:

jobs:
- job: B
  dependsOn: A
  variables:
    env: Dev

Variable Scope

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#variable-scopes

Variables can be set at various scopes:

  • At the root level, to make available for all jobs in the pipeline.
  • At the stage level, to make available only in a specific stage.
  • At the job level, to make available only in a specific job.

Predefined Variables

https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

There are variables defined in Azure pipelines as shown above.

For example, to display the name of the built pipeline in the log, use the Build.DefinitionName variable and display it with echo:

stages:
- stage: Build
  jobs:
  - job: FirstJobs
    steps:
    - bash: echo $(Build.DefinitionName)

Viewing Logs

https://learn.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/review-logs?view=azure-devops#view-and-download-logs

Individual logs for each step can be viewed following the above steps.

Task Assistant

https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/yaml-pipeline-editor?view=azure-devops#use-task-assistant

Easily add tasks you want to use.

File Copy

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/copy-files-v2?view=azure-pipelines&tabs=yaml

Files can be copied by defining CopyFiles@2 as a task in steps:

stages:
- stage: Build
  jobs:
  - job: FirstJobs
    steps:
    - bash: echo $(Build.DefinitionName)
    - task: CopyFiles@2
      inputs:
        SourceFolder: '$(Build.SourcesDirectory)'
        Contents: |
          **/*.yaml
          **/*.tf
        TargetFolder: '$(Build.ArtifactStagingDirectory)'

Publishing Build Artifacts

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-build-artifacts-v1?view=azure-pipelines

Sometimes you want to share build results between stages.

In such cases, use task: PublishBuildArtifacts@1:

steps:
- task: CopyFiles@2
  inputs:
    contents: '_buildOutput/**'
    targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts@1
  inputs:
    pathToPublish: $(Build.ArtifactStagingDirectory)
    artifactName: 'drop'
    publishLocation: 'Container'

What Are Build Artifacts?

https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/artifacts-overview?view=azure-devops&tabs=nuget

  • Files generated by the build
  • Examples: .dll, .exe, etc.

Strategy

https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job-strategy?view=azure-pipelines

  • Using a matrix generates copies of a job, each with different inputs.
  • These copies are useful for testing against different configurations or platform versions.

strategy:
    matrix:
      Python35:
        PYTHON_VERSION: '3.5'
      Python36:
        PYTHON_VERSION: '3.6'
      Python37:
        PYTHON_VERSION: '3.7'
    maxParallel: 2

steps:
- script: echo Running on $(PYTHON_VERSION)
  displayName: 'Run a one-line script'
  • This matrix creates 3 jobs: "Build Python35", "Build Python36", and "Build Python37".
  • Within each job, a variable named PYTHON_VERSION can be used.
  • In "Build Python35", the variable is set to "3.5". Similarly, "Build Python36" is set to "3.6".
  • Only 2 jobs run simultaneously.

Deployment Jobs

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops

  • In YAML pipelines, it is recommended to place deployment steps in a special type of job called a deployment job.
  • A deployment job is a collection of steps that run sequentially against an environment.
  • Deployment jobs and regular jobs can coexist in the same stage.
stages:
- stage: DevDeploy
  jobs:
  - deployment: DeployToDevelopment
    environment: Dev
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo deploy
- stage: QADeploy
  jobs:
  - deployment: DeployToQA
    environment: QA
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo deploy

Defining Approvals and Checks

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass#approvals

  • Approval checks can be used to manually control when stages run.
  • For example, creating "Approvals and checks" for an environment causes stage execution to wait until approval is given.
  • This makes it possible to deploy only what has been approved.

Disabling Build Pipelines

https://www.ntweekly.com/2022/01/25/how-to-disable-azure-devops-pipeline-temporarily/

A created pipeline can be disabled following the above steps.

Creating and Using Service Connections

To connect to external services such as DockerHub, create a "service connection" and use it in the pipeline definition.

Creating a Service Connection

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml

Using a Service Connection

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#use-a-service-connection

Building Docker

https://www.udemy.com/course/devops-with-docker-kubernetes-and-azure-devops/learn/lecture/18086393#overview

  1. Create a service connection to DockerHub
  2. Use the service connection in the pipeline
trigger:
- main

resources:
- repo: self

variables:
  tag: '$(Build.BuildId)'

stages:
- stage: Build
  displayName: Build image
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: ubuntu-latest
    steps:
    - task: Docker@2
      displayName: Build an image
      inputs:
        containerRegistry: 'docker-hub-test'
        repository: 'test2023'
        command: 'buildAndPush'
        Dockerfile: '**/Dockerfile'
        tags: '$(tag)'

In the above example, the build ID is specified as the Docker image tag.

Library

In the library, variable groups and secure files can be configured.

Access control for variables and files allows safely storing Azure connection information and other data for use in pipelines.

Pipelines that can access variable groups and secure files can be restricted.

Variable Group

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml

Secure Files

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops

Release Pipeline

https://learn.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops

The release pipeline releases artifacts created by the build pipeline to Azure App Service, etc.

For example, the following release examples exist:

  • Deploy Python or Java applications to Azure App Service
  • Deploy containerized applications to a Kubernetes cluster
  • Deploy to Azure Functions
  • Create Azure resources using ARM templates
  • Create Azure resources using Terraform

Creating Azure Resources with Terraform

https://learn.microsoft.com/en-us/azure/developer/terraform/best-practices-integration-testing

https://www.udemy.com/course/devops-with-docker-kubernetes-and-azure-devops/learn/lecture/18086321?start=180#overview

Implement creating Azure resources using Terraform via Pipelines.

To execute Terraform in Pipelines to create Azure resources, a plugin needs to be enabled in Pipelines to allow handling Terraform CLI (terraform init, terraform apply).

Pipelines are created in the following flow:

  1. Create Terraform code
  2. Create a "service connection" to Azure in Azure DevOps
  3. Enable Azure DevOps plugin
    1. https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks&targetId=4f030a9a-9459-478b-beae-d29469710a6d
    2. https://marketplace.visualstudio.com/items?itemName=charleszipp.azure-pipelines-tasks-terraform
  4. Create Pipelines

To execute Terraform CLI, you need to pass the Azure client ID, client secret, and SSH public key file to the pipeline YAML.

To pass client ID and client secret to Pipelines, use the Variable group in Library of Azure Pipelines.

On the other hand, to pass the SSH public key file, use Secure files in Library of Azure Pipelines.

The pipeline definition that executes terraform init looks like this:

In the following, a .tfstate file is created on Azure Storage Account. Resource groups and storage accounts are also created together.

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: echo Azure Terraform!
  displayName: 'Run a one-line script'

- task: TerraformCLI@0
  inputs:
    command: 'init'
    workingDirectory: '$(System.DefaultWorkingDirectory)/configuration'
    backendType: 'azurerm'
    backendServiceArm: 'Pay-As-You-Go (e595aee6-f9b8-450f-b9d3-73768a2fd47a)'
    ensureBackend: true
    backendAzureRmResourceGroupName: 'terraform-backend-rg'
    backendAzureRmResourceGroupLocation: 'westeurope'
    backendAzureRmStorageAccountName: 'azuredevopsterraformbackend20230311'
    backendAzureRmContainerName: 'dev'
    backendAzureRmKey: 'terraform-dev.tfstate'

- task: TerraformCLI@0
  inputs:
    command: 'apply'
    workingDirectory: '$(System.DefaultWorkingDirectory)/configuration'
    environmentServiceName: 'Pay-As-You-Go (e595aee6-f9b8-450f-b9d3-73768a2fd47a)'
    commandOptions: '-var client_id=$(client_id) -var client_secret=$(client_secret) -var ssh_public_key=$(publickey.secureFilePath)'

Azure DevOps Demo Generator

https://azuredevopsdemogenerator.azurewebsites.net/

https://qiita.com/dz_/items/736bfd7be1478d638518

Can create reference projects with Azure DevOps Demo Generator.

When a reference project is created with Azure DevOps Demo Generator, you can refer to sample Dashboards, Wiki, Boards, Repos, and Pipelines that have already been created.

Boards

Reading the following articles is recommended:

https://qiita.com/mstakaha1113/items/44458046d5a8568559b5

https://qiita.com/mstakaha1113/items/2c857e85ed6203d93028

https://qiita.com/mstakaha1113/items/9e0be0aaa5db89dd5ca9

Sprint

https://www.udemy.com/course/devops-with-docker-kubernetes-and-azure-devops/learn/lecture/18086137#overview

Query

https://www.udemy.com/course/devops-with-docker-kubernetes-and-azure-devops/learn/lecture/18086135#overview

Can create queries to find Work items meeting specified conditions.

Created queries can be saved and searched again with the same conditions.

Repos

https://qiita.com/mstakaha1113/items/e2c6ef2622bc6cc0b0ef

Code repository

Test Plans

https://news.mynavi.jp/techplus/article/zeroazure-30/

← Back to articles