Skip to content

Commit c1143e9

Browse files
YunchuWangvrdmr
YunchuWang
and
vrdmr
authored
Add performance profiling pipeline to perf tests (#1392)
* create py spy pipeline * fix * setup profiling pipeline * update pipeline name * fix * fix * Update profiling.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf_tests.Dockerfile * Update perf_tests.Dockerfile * Update perf_tests.Dockerfile * Update perf_tests.Dockerfile * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf_tests.Dockerfile * Update perf-testing-setup.yml * Update perf-testing-setup.yml * logging * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf_tests.Dockerfile * Update perf_tests.Dockerfile * Update perf-testing-setup.yml * Update SyncHttpTriggerHelloWorld.js * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * fix * rename * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * clean up * fix feedback * add newline * fix feedback * fix feedback * fix * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update .github/workflows/perf-testing-setup.yml Co-authored-by: Varad Meru <[email protected]> * include br name * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf_tests.Dockerfile * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml * Update perf-testing-setup.yml --------- Co-authored-by: Varad Meru <[email protected]>
1 parent 69f7363 commit c1143e9

File tree

6 files changed

+86
-69
lines changed

6 files changed

+86
-69
lines changed

‎.ci/perf_tests/dockerfiles/perf_tests.Dockerfile‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
ARG PYTHON_VERSION=3.8
1+
ARG PYTHON_VERSION=3.11
22

33
FROM mcr.microsoft.com/azure-functions/python:4-python$PYTHON_VERSION
44

@@ -15,7 +15,8 @@ ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
1515
FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED=1
1616

1717
RUN apt-get --quiet update && \
18-
apt-get install --quiet -y git && \
18+
apt-get install --quiet -y git procps && \
19+
# Procps is required for displaying worker and profiling processes info
1920
cd /home && \
2021
git clone https://github.com/vrdmr/AzFunctionsPythonPerformance.git && \
2122
mkdir -p /home/site/wwwroot/ && \

‎.ci/perf_tests/dockerfiles/perf_tests_3.6.Dockerfile‎

Lines changed: 0 additions & 25 deletions
This file was deleted.

‎.ci/perf_tests/dockerfiles/perf_tests_3.7.Dockerfile‎

Lines changed: 0 additions & 25 deletions
This file was deleted.

‎.ci/perf_tests/dockerfiles/perf_tests_3.9.Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,7 +194,8 @@ VOLUME ["/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_work
194194
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61 && \
195195
echo "deb https://dl.bintray.com/loadimpact/deb stable main" | tee -a /etc/apt/sources.list && \
196196
apt-get update && \
197-
apt-get install -y git k6 && \
197+
apt-get install -y git k6 procps && \
198+
# Procps is required for displaying worker and profiling processes info
198199
cd /home && \
199200
git clone https://github.com/vrdmr/AzFunctionsPythonPerformance.git && \
200201
mkdir -p /home/site/wwwroot/ && \

‎.ci/perf_tests/k6scripts/SyncHttpTriggerHelloWorld.js‎ renamed to ‎.ci/perf_tests/k6scripts/SyncHelloWorld.js‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ export let options = {
3535

3636
// Main function
3737
export default function () {
38-
let response = http.get(`${PROTOCOL}://${HOSTNAME}:${PORT}/api/SyncHttpTriggerHelloWorld`);
38+
let response = http.get(`${PROTOCOL}://${HOSTNAME}:${PORT}/api/SyncHelloWorld`);
3939

4040
// check() returns false if any of the specified conditions fail
4141
let checkRes = check(response, {
@@ -45,4 +45,4 @@ export default function () {
4545

4646
// We reverse the check() result since we want to count the failures
4747
failureRate.add(!checkRes);
48-
}
48+
}

‎.github/workflows/perf-testing-setup.yml‎

Lines changed: 79 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,37 @@ name: Throughput testing workflow
22

33
on:
44
workflow_dispatch:
5-
pull_request:
6-
branches: [ dev ]
7-
5+
inputs:
6+
profiling_sampling_rate:
7+
description: 'Profiling sampling rate (tps)'
8+
required: false
9+
default: '500'
10+
test_to_run:
11+
description: 'List of perf tests to run'
12+
required: false
13+
default: SyncHelloWorld
14+
issue_comment:
15+
types: [created]
816
env:
917
TESTS_DIR_PATH: ".ci/perf_tests/k6scripts/"
10-
PYTHON_VERSION: "3.8"
18+
PYTHON_VERSION: "3.10"
19+
PYTHON_FUNCTION_PROFILING_STORAGE_ACCT: "azpyfuncpipelinestorage"
1120
PORT: 8000
1221

1322
jobs:
1423
build:
24+
if: ${{ github.event_name == 'workflow_dispatch' || github.event.issue.pull_request && contains(github.event.comment.body, '/profile') }}
1525
runs-on: ubuntu-latest
1626
strategy:
1727
fail-fast: false
1828
matrix:
19-
test_to_run: [ SyncHttpTriggerHelloWorld ]
29+
test_to_run: ['${{ github.event.inputs.test_to_run }}']
2030
steps:
2131
- uses: actions/checkout@v2
2232
- name: Set up Python ${{ env.PYTHON_VERSION }}
2333
uses: actions/setup-python@v2
2434
with:
2535
python-version: ${{ env.PYTHON_VERSION }}
26-
- name: Set up Dotnet 2.x
27-
uses: actions/setup-dotnet@v1
28-
with:
29-
dotnet-version: '3.1.405'
30-
- name: Set up Dotnet 6.x
31-
uses: actions/setup-dotnet@v1
32-
with:
33-
dotnet-version: '6.x'
3436
- name: Setup k6 for throughput testing
3537
run: |
3638
cd $GITHUB_WORKSPACE
@@ -47,13 +49,76 @@ jobs:
4749
python setup.py extension
4850
- name: Build and Run the Docker image
4951
run: |
52+
echo "Building Docker image with Python version ${{ env.PYTHON_VERSION }}"
5053
docker build --build-arg PYTHON_VERSION=${{ env.PYTHON_VERSION }} --file .ci/perf_tests/dockerfiles/perf_tests.Dockerfile --tag perfimage:latest .
51-
docker run -d --shm-size="2g" --env FUNCTIONS_WORKER_RUNTIME_VERSION=${{ env.PYTHON_VERSION }} -p ${PORT}:80 -v $GITHUB_WORKSPACE/azure_functions_worker:/azure-functions-host/workers/python/${{ env.PYTHON_VERSION }}/LINUX/X64/azure_functions_worker perfimage:latest
54+
55+
echo "Running Docker container..."
56+
container_id=$(docker run -d --privileged --env FUNCTIONS_WORKER_RUNTIME_VERSION=${{ env.PYTHON_VERSION }} -p ${PORT}:80 -v $GITHUB_WORKSPACE/azure_functions_worker:/azure-functions-host/workers/python/${{ env.PYTHON_VERSION }}/LINUX/X64/azure_functions_worker perfimage:latest)
5257
sleep 10 # host needs some time to start.
58+
echo "Container ID is $container_id"
59+
60+
echo "Fetching Docker container logs..."
61+
docker logs $container_id
62+
worker_pid=$(docker exec $container_id sh -c "ps aux | grep '[p]ython'" | awk '{print $2}')
63+
echo "Python worker process id is $worker_pid"
64+
65+
echo "container_id=$container_id" >> $GITHUB_ENV
66+
echo "worker_pid=$worker_pid" >> $GITHUB_ENV
67+
5368
- name: Validate if the functions are now running
5469
run: |
5570
curl --get http://localhost:${PORT}/api/${{ matrix.test_to_run }}
71+
72+
- name: Start py-spy in the background
73+
run: |
74+
docker exec $container_id sh -c "pip install py-spy"
75+
docker exec $container_id sh -c "mkdir /home/profiling_reports"
76+
profiling_sampling_rate=${{ github.event.inputs.profiling_sampling_rate }}
77+
# report_name="${GITHUB_REF#refs/heads/}_${{ github.run_id }}.svg"
78+
report_name="${{ github.run_id }}.svg"
79+
docker exec -d $container_id sh -c "RUST_BACKTRACE=1 py-spy record -p $worker_pid -o /home/profiling_reports/$report_name -f flamegraph --idle --nonblocking --rate $profiling_sampling_rate > /home/site/wwwroot/py-spy.log 2>&1 &"
80+
sleep 2 # Give it a moment to start
81+
py_spy_id=$(docker exec $container_id sh -c "ps aux | grep '[p]y-spy record'" | awk '{print $2}')
82+
83+
echo "py_spy_id=$py_spy_id" >> $GITHUB_ENV
84+
echo "report_name=$report_name" >> $GITHUB_ENV
85+
5686
- name: Run Throughput tests
5787
run: |
5888
chmod 755 .ci/perf_tests/run-perftests.sh
5989
.ci/perf_tests/run-perftests.sh localhost $PORT ${{ env.TESTS_DIR_PATH }} ${{ matrix.test_to_run }}
90+
91+
- name: Stop profiling and generate report
92+
run: |
93+
echo "Tests completed, terminating py-spy..."
94+
docker exec $container_id cat /home/site/wwwroot/py-spy.log
95+
docker exec $container_id sh -c "kill -2 $py_spy_id"
96+
sleep 2
97+
98+
mkdir profiling_reports
99+
chmod 777 profiling_reports
100+
docker cp $container_id:/home/profiling_reports/$report_name profiling_reports
101+
102+
- name: Upload SVG to Azure Blob Storage
103+
uses: bacongobbler/[email protected]
104+
with:
105+
source_dir: 'profiling_reports' # Directory containing the $report_name file
106+
container_name: 'profiling'
107+
connection_string: ${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}
108+
sync: 'false'
109+
110+
- name: Output Blob URL
111+
run: |
112+
blob_url="https://${{ env.PYTHON_FUNCTION_PROFILING_STORAGE_ACCT }}.blob.core.windows.net/profiling/${{ env.report_name }}"
113+
echo "You can view the Blob at: $blob_url"
114+
115+
- name: Upload profiling result to artifact
116+
uses: actions/upload-artifact@v2
117+
with:
118+
name: py-spy-output
119+
path: 'profiling_reports/${{ env.report_name }}'
120+
121+
- name: Create Artifact Link
122+
run: |
123+
echo "You can download the SVG artifact from the Actions run page."
124+
echo "Link to the Actions run page: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"

0 commit comments

Comments
 (0)