Azure CLI: Azure Monitor Alert 일괄 생성 스크립트 Part.3 az monitor scheduled-query create

Log Query Alert

Azure 서비스의 로그를 수집하고 있는 경우, 대상 로그에 관해 사용자 지정 쿼리를 이용하여 알람 규칙을 만드는 것이 가능하다.

예를 들어서, Virtual machine 의 로그를 수집하고 있다면 디스크의 빈 용량을 확인하는 쿼리를 사용하여 해당 값을 주기적으로 확인하고 몇 % 이하의 경우 알람을 발생시키는 것이 가능하다.

Tip. Azure Portal 에서 수동으로 Custom log search 를 지정하여 알람 규칙을 만드는 경우는 아래를 참고

Azure Portal – Custom log search
Perf
| where ObjectName == "LogicalDisk"
| where CounterName == "% Free Space"
| where Computer == "AZRWIN-1"
| where InstanceName  == "C:"
| summarize arg_max(TimeGenerated, *) by InstanceName
| project CounterValue
Azure Portal – Measurement, Alert logic

관련 포스트

준비 하기

https://github.com/jcy9033/azure-cli/tree/master/azureMonitor/logQueryAlert

참고용 Azure CLI 명령어 소개

Microsoft Learn az monitor scheduled-query link

az monitor scheduled-query create --condition
                                  --name
                                  --resource-group
                                  --scopes
                                  [--action-groups]
                                  [--auto-mitigate {false, true}]
                                  [--check-ws-alerts-storage {false, true}]
                                  [--condition-query]
                                  [--custom-properties]
                                  [--description]
                                  [--disabled {false, true}]
                                  [--evaluation-frequency]
                                  [--location]
                                  [--mad]
                                  [--severity]
                                  [--skip-query-validation {false, true}]
                                  [--tags]
                                  [--target-resource-type]
                                  [--window-size]

CSV 구성 소개

  • AlertRuleName
  • Subscription
  • ResourceGroup
  • Severity
  • Subscription_User
  • ResourceGroup_User
  • ResourceName
  • LogQuery
    : Perf | where CounterName == '% Committed Bytes In Use' | where Computer == 'AZRWIN-1' | summarize arg_max(TimeGenerated, *) by InstanceName | project CounterValue

    Tip. ” ” (Double quotation) 대신 ‘ ‘ (Single quotation)을 사용하지 않으면 에러가 발생하므로 주의.
  • Measure
    : 측정값의 기준 ex) CounterValue
  • Aggregation
  • WindowSize
  • Dimensions
  • Operator
  • Threshold
  • EvaluationFrequency
  • AutoMitigate
  • Location
  • ActionGroupName

Script 구성 소개

CSV 파일 경로 및 가져 오기

# Please enter the file path for the CSV
$csvPath = "CSV File Path"

$csvData = Import-Csv $csvPath -Encoding UTF8

Subscription 확인 하기

몇 번째 행을 처리 중인지 확인하기 위해서 Create No.$i 를 추가했다.

$i = 1

$csvData | ForEach-Object {

  #----------------------------------------- Subscription check
  
  Write-Host "----------# [Create No.$i : $(Get-Date)]"
  
  $TargetSubscription = $_.Subscription
  
  $ActiveSubscription = az account show --query name -o tsv

  if ($ActiveSubscription -eq $TargetSubscription) {
    Write-Host "[Info] Already on the target subscription: $TargetSubscription"
  }
  else {
    az account set --subscription $TargetSubscription
    Write-Host "[Info] Switched to the target subscription: $TargetSubscription"
  }

Action Group 값 가져 오기

  #----------------------------------------- Action Group

  $ActionGroupName = $_.ActionGroupName

  $ActionGroupList = $ActionGroupName -split ';'
  
  Write-Host "[Info] Action group list: $ActionGroupList"

  $ActionGroups = @()
  
  foreach ($ActionGroup in $ActionGroupList) {
    $ActionGroupJson = az graph query -q "Resources | where type == 'microsoft.insights/actiongroups' and name has '$ActionGroup' | project id" | ConvertFrom-Json
  
    if ($ActionGroupJson.data -and $ActionGroupJson.data.Count -gt 0) {
      $ActionGroupId = $ActionGroupJson.data[0].id
      $ActionGroups += "$ActionGroupId"
    }
    else {
      Write-Host "[Warning] No valid action group ID found for $ActionGroup"
    }
  }

  Write-Host "[Info] Action groups parameter: $ActionGroups"

Log Query 변수에 저장하기

  #----------------------------------------- Log query 
  
  $LogQuery = $_.LogQuery
  
  Write-Host "[Info] Condition Log Query: $LogQuery"

Condition String 만들기

  #----------------------------------------- Conditions

  $Condition = "$($_.Aggregation) $($_.Measure) from 'LogQuery' $($_.Operator) $($_.Threshold)"

  Write-Host "[Info] Scheduled query alert condition: $Condition"

Resource id 값 가져오기

  #----------------------------------------- Resource parameters

  $ResourceName = $_.ResourceName
  
  $Scopes = (az resource list --query "[?name=='$ResourceName'].id" --output tsv).Trim()
    
  Write-Host "[Info] Resource scopes: $Scopes"

Log Query Alert 만들기

  #----------------------------------------- Create Scheduled query alert
  
  az monitor scheduled-query create --name $_.AlertRuleName --resource-group $_.ResourceGroup --scopes $Scopes `
    --severity $_.Severity `
    --condition "$Condition" `
    --condition-query LogQuery="$LogQuery" `
    --evaluation-frequency $_.EvaluationFrequency `
    --auto-mitigate $_.AutoMitigate `
    --window-size $_.WindowSize `
    --action-group $ActionGroups

  $i++  
}

Write-Host "----------# [All done: $(Get-Date)]"

Script 실행 결과

PS C:\Users\cchi9\OneDrive\Vscode\azure-cli\azureMonitor\logQueryAlert> .\main.ps1
----------# [Create No.1 : 01/06/2024 14:17:17]
[Info] Already on the target subscription: Azure subscription 2
[Info] Action group list: ag-1 ag-2 ag-3
[Info] Action groups parameter: /subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-1 /subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-2 /subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-3
[Info] Condition Log Query: Perf | where CounterName == '% Committed Bytes In Use' | where Computer == 'AZRWIN-1' | summarize arg_max(TimeGenerated, *) by InstanceName | project CounterValue
[Info] Scheduled query alert condition: avg CounterValue from 'LogQuery' > 90
[Info] Resource scopes: /subscriptions/0b5f5005-c30c-4a28-89c1-9457d0cd5e0f/resourceGroups/system-1/providers/Microsoft.Compute/virtualMachines/AZRWIN-1
{
  "actions": {
    "actionGroups": [
      "/subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-1",
      "/subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-2",
      "/subscriptions/611a7ed8-17fa-480a-901d-d7084803c376/resourceGroups/core-1/providers/microsoft.insights/actiongroups/ag-3"
    ],
    "customProperties": {}
  },
  "autoMitigate": true,
  "checkWorkspaceAlertsStorageConfigured": false,
  "createdWithApiVersion": "2021-08-01",
  "criteria": {
    "allOf": [
      {
        "dimensions": [],
        "failingPeriods": {
          "minFailingPeriodsToAlert": 1,
          "numberOfEvaluationPeriods": 1
        },
        "metricMeasureColumn": "CounterValue",
        "metricName": null,
        "operator": "GreaterThan",
        "query": "Perf | where CounterName == '% Committed Bytes In Use' | where Computer == 'AZRWIN-1' | summarize arg_max(TimeGenerated, *) by InstanceName | project CounterValue",
        "resourceIdColumn": null,
        "threshold": 90.0,
        "timeAggregation": "Average"
      }
    ]
  },
  "description": null,
  "displayName": null,
  "enabled": true,
  "etag": null,
  "evaluationFrequency": "0:05:00",
  "id": "/subscriptions/0b5f5005-c30c-4a28-89c1-9457d0cd5e0f/resourceGroups/system-1/providers/microsoft.insights/scheduledqueryrules/log1-1",
  "isLegacyLogAnalyticsRule": null,
  "isWorkspaceAlertsStorageConfigured": null,
  "kind": null,
  "location": "japaneast",
  "muteActionsDuration": null,
  "name": "log1-1",
  "overrideQueryTimeRange": null,
  "resourceGroup": "system-1",
  "scopes": [
    "/subscriptions/0b5f5005-c30c-4a28-89c1-9457d0cd5e0f/resourceGroups/system-1/providers/Microsoft.Compute/virtualMachines/AZRWIN-1"
  ],
  "severity": 1,
  "skipQueryValidation": false,
  "systemData": {
    "createdAt": "2024-01-06T05:17:23.125963+00:00",
    "createdBy": "xxxxx",
    "createdByType": "User",
    "lastModifiedAt": "2024-01-06T05:17:23.125963+00:00",
    "lastModifiedBy": "xxxxx",
    "lastModifiedByType": "User"
  },
  "tags": null,
  "targetResourceTypes": null,
  "type": "Microsoft.Insights/scheduledQueryRules",
  "windowSize": "0:05:00"
}
----------# [Create No.2 : 01/06/2024 14:17:29]
[Info] Already on the target subscription: Azure subscription 2
[Info] Action group list: ag-1 ag-2 ag-3

...Skip

----------# [All done: 01/06/2024 14:17:49]