Add a skill for LLMs on how to get the azp logs for analysis (#86765)

.claude/skills/azp-logs/SKILL.md

@@ -0,0 +1,99 @@
+---
+description: Download Azure Pipelines CI logs for analysis
+argument-hint: <pr_number|build_id|build_url>
+allowed-tools: [Bash(gh pr view:*), Bash(gh pr checks:*), Bash(ls:*), Read, Grep]
+---
+
+Azure Pipelines Logs Downloader
+================================
+
+Download Azure Pipelines CI logs for analyzing test failures and CI issues.
+
+**IMPORTANT**: Always ask the user before downloading logs. The download may take 5-10 minutes (or longer for large CI runs)
+depending on the number of jobs and log size.
+
+Usage
+-----
+
+```bash
+/azp-logs <pr_number|build_id|build_url>
+```
+
+Arguments
+---------
+
+- `pr_number`: GitHub PR number (will extract build ID from latest CI run)
+- `build_id`: Azure Pipelines build ID (numeric)
+- `build_url`: Full Azure Pipelines URL (e.g., <https://dev.azure.com/ansible/ansible/_build/results?buildId=12345>)
+
+Implementation
+--------------
+
+This command uses the existing `hacking/azp/download.py` script to download CI logs.
+
+**Before running**: Always confirm with the user before downloading logs. Inform them that:
+- The download may take 5-10 minutes for a full CI run (potentially longer for very large runs)
+- Logs will be saved to a directory named after the build ID
+- The download size can be 10-50MB depending on the number of jobs
+
+Process Steps
+-------------
+
+1. **Ask user for confirmation**: Explain what will be downloaded and estimated time
+
+2. **Determine build ID**:
+   - If given a PR number: Use `gh pr checks <number>` to get the Azure Pipelines URL
+   - If given a URL: Pass it directly to download.py (it extracts buildId automatically)
+   - If given a build ID: Use directly
+
+3. **Download logs**: Run `hacking/azp/download.py` with appropriate flags:
+
+   ```bash
+   ./hacking/azp/download.py <build_id_or_url> --console-logs -v
+   ```
+
+4. **Analyze logs**: After download completes, examine logs in `<build_id>/` directory:
+   - Grep for common failure patterns: `FAILED`, `ERROR`, `Traceback`
+   - Focus on logs from failed jobs (check job names)
+   - Compare with ansibot comments for context
+
+Download Script Options
+-----------------------
+
+The `hacking/azp/download.py` script supports:
+- `--console-logs`: Download console logs (recommended for CI failure analysis)
+- `--artifacts`: Download test artifacts
+- `--run-metadata`: Download run metadata JSON
+- `--all`: Download everything
+- `--match-job-name <regex>`: Filter to specific jobs
+- `--match-artifact-name <regex>`: Filter to specific artifacts
+- `-v, --verbose`: Show what is being downloaded
+- `-t, --test`: Dry run (show what would be downloaded)
+
+For most CI failure analysis, use `--console-logs` to get the log files.
+
+Common Analysis Patterns
+------------------------
+
+After downloading logs to `<build_id>/` directory:
+
+```bash
+# Find all errors and failures
+grep -r "FAILED\|ERROR\|Traceback" <build_id>/
+
+# Find specific test failures
+grep -r "FAILED test" <build_id>/
+
+# Find sanity test failures
+grep -r "The test" <build_id>/ | grep -i "failed"
+
+# List all downloaded log files
+ls -lh <build_id>/
+```
+
+Notes
+-----
+
+- Logs are downloaded to a directory named after the build ID
+- Console logs are named after the job hierarchy (e.g., "Job Name Stage Name.log")
+- The Ansible project is public, so no authentication is required

AGENTS.md

@@ -42,6 +42,7 @@ gh pr view <number> --comments                            # Check for ansibot CI
 gh pr checks <number>                                     # Get Azure Pipelines URLs
 gh pr checkout <number>                                   # Switch to PR branch
 gh pr diff <number>                                       # See all changes
+/azp-logs <number>                                        # Download CI logs for PR
 ```
 
 **Container Selection:**
@@ -151,6 +152,43 @@ This shows:
 4. For sanity test failures, the error messages usually indicate exactly what needs to be fixed
 5. For test failures, run the same tests locally using `ansible-test` to reproduce and debug
 
+**5. Downloading Azure Pipelines logs for analysis:**
+
+When CI failures need deeper investigation beyond what's visible in ansibot comments or the web UI, use the `/azp-logs` skill:
+
+```bash
+# Download logs using PR number (automatically finds latest build)
+/azp-logs <pr_number>
+
+# Or use build ID directly from gh pr checks output
+/azp-logs <build_id>
+
+# Or use the full Azure Pipelines URL
+/azp-logs https://dev.azure.com/ansible/ansible/_build/results?buildId=12345
+```
+
+The skill uses `hacking/azp/download.py` to download console logs into a directory named after the build ID.
+
+**After downloading, analyze the logs:**
+- Grep for common failure patterns: `grep -r "FAILED\|ERROR\|Traceback" <build_id>/`
+- Focus on logs from failed jobs identified in `gh pr checks` output
+- Compare error messages with ansibot comments to get full context
+- Sanity test failures usually have clear error messages with file:line references
+- Integration/unit test failures may require examining full test output and tracebacks
+
+**Advanced usage:**
+The download script supports filtering and customization:
+
+```bash
+# Download only logs matching specific job names
+./hacking/azp/download.py <build_id> --console-logs --match-job-name "Sanity.*"
+
+# Download artifacts and metadata too
+./hacking/azp/download.py <build_id> --all
+```
+
+See `.claude/skills/azp-logs/SKILL.md` for complete documentation.
+
 ## PR Review Guidelines
 
 ### PR Review Checklist

hacking/azp/download.py

@@ -27,21 +27,16 @@ import json
 import os
 import re
 import io
+import shutil
 import zipfile
-
+import urllib.request
-import requests
+import urllib.error
 
 try:
     import argcomplete
 except ImportError:
     argcomplete = None
 
-# Following changes should be made to improve the overall style:
-# TODO use new style formatting method.
-# TODO use requests session.
-# TODO type hints.
-# TODO pathlib.
-
 
 def main():
     """Main program body."""
@@ -134,9 +129,8 @@ def download_run(args):
 
     if args.run_metadata:
         run_url = 'https://dev.azure.com/ansible/ansible/_apis/pipelines/%s/runs/%s?api-version=6.0-preview.1' % (args.pipeline_id, args.run)
-        run_info_response = requests.get(run_url)
+        with urllib.request.urlopen(run_url) as run_info_response:
-        run_info_response.raise_for_status()
+            run = json.load(run_info_response)
-        run = run_info_response.json()
 
         path = os.path.join(output_dir, 'run.json')
         contents = json.dumps(run, sort_keys=True, indent=4)
@@ -148,9 +142,8 @@ def download_run(args):
             with open(path, 'w') as metadata_fd:
                 metadata_fd.write(contents)
 
-    timeline_response = requests.get('https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/timeline?api-version=6.0' % args.run)
+    with urllib.request.urlopen('https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/timeline?api-version=6.0' % args.run) as timeline_response:
-    timeline_response.raise_for_status()
+        timeline = json.load(timeline_response)
-    timeline = timeline_response.json()
     roots = set()
     by_id = {}
     children_of = {}
@@ -185,18 +178,21 @@ def download_run(args):
 
     if args.artifacts:
         artifact_list_url = 'https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/artifacts?api-version=6.0' % args.run
-        artifact_list_response = requests.get(artifact_list_url)
+        with urllib.request.urlopen(artifact_list_url) as artifact_list_response:
-        artifact_list_response.raise_for_status()
+            artifact_list = json.load(artifact_list_response)
-        for artifact in artifact_list_response.json()['value']:
+
+        for artifact in artifact_list['value']:
             if artifact['source'] not in allowed or not args.match_artifact_name.match(artifact['name']):
                 continue
             if args.verbose:
                 print('%s/%s' % (output_dir, artifact['name']))
             if not args.test:
-                response = requests.get(artifact['resource']['downloadUrl'])
+                with urllib.request.urlopen(artifact['resource']['downloadUrl']) as response:
-                response.raise_for_status()
+                    with io.BytesIO() as buffer:
-                archive = zipfile.ZipFile(io.BytesIO(response.content))
+                        shutil.copyfileobj(response, buffer)
-                archive.extractall(path=output_dir)
+                        buffer.seek(0)
+                        with zipfile.ZipFile(buffer) as archive:
+                            archive.extractall(path=output_dir)
 
     if args.console_logs:
         for r in timeline['records']:
@@ -220,9 +216,9 @@ def download_run(args):
             if args.verbose:
                 print(log_path)
             if not args.test:
-                log = requests.get(r['log']['url'])
+                with urllib.request.urlopen(r['log']['url']) as log:
-                log.raise_for_status()
+                    with open(log_path, 'wb') as log_file:
-                open(log_path, 'wb').write(log.content)
+                        shutil.copyfileobj(log, log_file)
 
 
 if __name__ == '__main__':

test/sanity/ignore.txt

@@ -1,4 +1,5 @@
 .claude/commands/review.md pymarkdown!skip  # Claude Code command with YAML frontmatter
+.claude/skills/azp-logs/SKILL.md pymarkdown!skip  # Claude Code skill with YAML frontmatter
 .github/ISSUE_TEMPLATE/internal_issue.md pymarkdown!skip
 lib/ansible/_internal/_wrapt.py black!skip  # vendored code
 lib/ansible/config/base.yml no-unwanted-files