Skip to content

Commit 9e9118c

Browse files
jblakleyjblakley
authored
Merge pull request #6 from jblakley/agw-orc8r
Pre-1.9 merge of magma-deployer
2 parents 12f4d31 + 6d29d52 commit 9e9118c

File tree

353 files changed

+40032
-57
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

353 files changed

+40032
-57
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/.project

‎README.md‎

Lines changed: 39 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,39 @@
1-
# magma-deployer
2-
3-
Docs: https://magma.github.io/magma/docs/next/orc8r/deploy_using_ansible
4-
5-
Quick Install:
6-
```bash
7-
sudo bash -c "$(curl -sL https://github.com/magma/magma-deployer/raw/main/deploy-orc8r.sh)"
8-
```
9-
10-
Switch to `magma` user after deployment has finsished:
11-
```bash
12-
sudo su - magma
13-
```
14-
15-
Once all pods are ready, setup NMS login:
16-
```bash
17-
cd ~/magma-deployer
18-
ansible-playbook config-orc8r.yml
19-
```
20-
21-
You can get your `rootCA.pem` file from the following location:
22-
```bash
23-
cat ~/magma-deployer/secrets/rootCA.pem
24-
```
1+
# Magma-Deployer
2+
Is a Magma project to simplify deployment of the components of the Magma Platform. The deployable components include the *Orchestrator aka Orc8r*, the *Access Gateway aka AGW*, the *Federation Gateway aka FEG (future)*, and the *Domain Proxy aka DP (future)*. There are multiple deployment methods and target environments possible for these components. At this time, magma-deployer enables the following deployments:
3+
4+
| Component | Version | Host | Deployment Model |
5+
| --------- | ------- | ---------------------------------------------------- | ------------------------------------------------------------ |
6+
| Orc8r | v1.9 | Ubuntu 22.04 Bare Metal or Virtual Machine | Ansible playbook(s) and helm into K8s cluster |
7+
| AGW | v1.9 | Ubuntu 20.04 Bare Metal or Virtual Machine (2 NICs) | Ansible playbook(s) and docker-compose into docker containers |
8+
9+
## Out of Scope (at this time)
10+
11+
* Although other deployment models exist (e.g., Terraform, AWS-specific, Vagrant VMs, non-containerized AGW, k8s AGW), they are not currently implemented in magma-deployer.
12+
* Magma-deployer does not currently support FEG or DP deployments
13+
* Magma-deployer ends when the AGW successfully connects to the Orc8r. Follow on tasks of adding eNodeBs and gNodeBs, subscribers, configuring for 5G and connecting UEs are considered post-deployment steps.
14+
15+
## Basic Workflow
16+
17+
A minimal new deployment begins with an Orc8r and a single AGW. To have a full standalone network, an eNodeB (LTE) or gNodeB (5G) and a compatible UE and SIM are needed. The steps in the deployment are:
18+
19+
1. Deploy Orc8r using orc8r-deployer.
20+
2. Deploy an AGW using agw-deployer. You will need information from step #1 during step #2 and beyond.
21+
3. Connect the AGW to the Orc8r
22+
23+
----
24+
25+
*Magma-deployer ends here.*
26+
27+
-----
28+
29+
4. [optional] Configure the network for 5G
30+
5. Connect eNodeB or gNodeB to AGW and provision in Orc8r
31+
6. Provision a UE and SIM for the network
32+
7. Provision a subscriber in Orc8r
33+
8. Connect the UE to the network
34+
35+
Now, head to orc8r-deployer. <link>
36+
37+
## References
38+
39+
TBD

‎agw-deployer/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/downloads
2+
ansible_collections

‎agw-deployer/README.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# An AGW Deployment Recipe
2+
3+
This recipe deploys a standalone Magma AGW. It assumes that the orc8r-deployer or equivalent has already been run and the orc8r is accessible from the AGW machine.
4+
5+
DISCLAIMER: As with many deployment recipes, successful execution of the recipe is dependent on adjusting it to the specifics of a given environment. There is no guarantee this recipe can work "out of the box" in an arbitrary environment. Familiarity with Linux, Docker, Magma, IP Networking, and Ansible are likely to be needed to assure successful completion.
6+
7+
The recipe consists of the following primary steps:
8+
9+
1. Bootstrapping the initial environment
10+
2. Deploy the Access Gateway
11+
3. Connect the AGW to the Orc8r
12+
13+
14+
### Prerequisites
15+
Ubuntu 20.04 system with >100GB disk.
16+
17+
## Preparing your environment
18+
19+
You will need a physical system running Ubuntu 20.04 to deploy the **Magma Access Gateway (AGW)**. The AGW requires two physical ethernet network interface cards. It is possible to run this same recipe in a virtual machine and this method has been tested in KVM/QEMU virtual machines. This recipe does not cover preparing a virtual machine.) This recipe assumes a bare metal install of the AGW. The AGW deployment uses docker and docker-compose deployment on the baremetal AGW system.
20+
21+
This recipe deploys a 5G network using Magma v1.9.
22+
23+
On the AGW System, set the environment variable `RECIPE_HOME` to the full pathname of the recipe folder (e.g,. `export RECIPE_HOME=/home/ubuntu/<repository>/magma-deployer/agw-deployer`).
24+
25+
## Deploy the AGW
26+
Deployment of the AGW involves:
27+
28+
1. Configuring deployment specific environment variables and installing prerequisites (`.env`, `bootstrap.sh`, reboot)
29+
2. Setting up the AGW network configuration (`agwc-networking` playbook)
30+
3. AGW docker-compose deployment (`agwc1` playbook, reboot, `agcw2` playbook)
31+
32+
The recipe is based on [this](https://magma.github.io/magma/docs/next/lte/deploy_install_docker) deployment guide.
33+
34+
### Initial configuration and installation of prerequistes
35+
36+
```
37+
$ export RECIPE_HOME=<THIS DIRECTORY>
38+
$ cd $RECIPE_HOME/bootstrap
39+
$ cp template.env .env
40+
$ vim .env
41+
```
42+
43+
Edit the variables in .env to your preferred values. Then run:
44+
45+
```
46+
$ bash bootstrap.sh
47+
```
48+
#### Notes on the `.env` variables:
49+
- MAGMA_DN is the domain for your Orc8r
50+
- PRIVATE_KEY is used for ansible hosts
51+
- ROOT_CA_PATH is the directory that contains your Orc8r's `rootCA.pem`
52+
- At this writing:
53+
- `DOCKER_IMAGE_VERSION=20.10.21-0ubuntu1~20.04.2`
54+
- `DOCKER_COMPOSE_VERSION="v2.17.2"`
55+
56+
Reboot and test that docker works correctly (e.g., `docker ps` should respond with no containers running). You may want to to inspect the `$RECIPE_HOME/ansible/hosts.yml` file to validate the configuration set up by `bootstrap.sh`.
57+
58+
`bootstrap.sh` runs an ansible playbook called `deploy-common-system.yml`. If you run into issues during this phase, you may need rerun this playbook.
59+
```
60+
$ cd $RECIPE_HOME/ansible
61+
$ ansible-playbook deploy-common-system.yml -K
62+
```
63+
64+
### AGW network configuration
65+
66+
This will set up the AGW eth0 and eth1 interfaces
67+
68+
```
69+
$ cd $RECIPE_HOME/ansible
70+
$ ansible-playbook deploy-agwc-networking.yml -K
71+
$ ip a
72+
```
73+
Verify that network for eth0 and eth1 are correct. You should be able to ping the Orc8r over eth0 and, if you have one, your gNB over eth1.
74+
75+
#### Notes on network configuration
76+
- Your network renderer should be set to `NetworkManager` (check: `/etc/netplan/00-installer-config.yaml`)
77+
- You may have issues with connectivity if this stage fails. Try to have direct console access during this playbook's execution
78+
- Configuration of the AGW's ethernet networking has been one of the more problematic parts of bringing up an AGW. The playbook sets up the networking prior to actually deploying the gateway to prevent some issues that arise during. However, the deployment may impact some of the configuration.
79+
- If you find that your network names are changing after reboots, make sure you do [this](https://askubuntu.com/questions/1255823/network-interface-names-change-every-reboot) Your grub configuration is wrong. Make it like this:
80+
81+
```
82+
$ sudo vim /etc/default/grub
83+
...
84+
# GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
85+
GRUB_CMDLINE_LINUX=""
86+
...
87+
$ sudo update-grub
88+
89+
$ reboot
90+
```
91+
92+
### AGW docker-compose deployment (Part 1 and Part 2)
93+
94+
The docker-compose version of the AGW will be deployed in two stages.
95+
96+
Stage one sets up many of the agwc parameters and files and runs `agw_install_docker.sh`. This script clones magma and configures it for use. It installs the OpenVSwitch used by magma which requires a reboot.
97+
98+
```
99+
$ cd $RECIPE_HOME/ansible
100+
$ ansible-playbook deploy-agwc1.yml -K
101+
```
102+
Reboot.
103+
104+
Stage two completes the configuration and brings up the AGW containers. After this is complete,
105+
106+
```
107+
$ cd $RECIPE_HOME/ansible
108+
$ ansible-playbook deploy-agwc2.yml -K
109+
$ docker ps
110+
```
111+
112+
All AGW containers should be running and showing healthy. The playbook will print the information needed to provision the AGW in the Orc8r. You can do that provisioning at this point. If you lose the info:
113+
114+
```
115+
$ docker exec magmad show_gateway_info.py
116+
```
117+
Use this to provision the AGW in the Orc8r.
118+
On the Orc8r NMS, navigate to `Equipment->Gateways" from the left navigation bar, hit "Add New" on the upper right, and fill out the multi-step modal form. Use the secrets from above for the "Hardware UUID" and "Challenge Key" fields.
119+
120+
For now, you won't have any eNodeB's to select in the eNodeB dropdown under the "Ran" tab. This is OK, we'll get back to this in a later step.
121+
122+
At this point, you can validate the connection between your AGW and Orchestrator:
123+
124+
After the provisioning, restart the AGW services.
125+
126+
```
127+
$ cd /var/opt/magma/docker
128+
$ sudo docker compose --compatibility up -d --force-recreate
129+
```
130+
At this point, you can validate the connection between your AGW and Orchestrator.
131+
132+
The magma documentation says to run:
133+
```
134+
sudo docker exec magmad checkin_cli.py
135+
```
136+
137+
to verify connectivity, however, as of this writing, there is a bug in the containerized version that will give this error even when you are connected to the Orc8r:
138+
```
139+
1. -- Testing TCP connection to controller.orc8r.magma18.livingedgelab.org:443 --
140+
2. -- Testing Certificate --
141+
3. -- Testing SSL --
142+
4. -- Creating direct cloud checkin --
143+
144+
> Error: <_MultiThreadedRendezvous of RPC that terminated with:
145+
status = StatusCode.UNAVAILABLE
146+
details = "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"
147+
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2022-10-21T19:23:22.773234625+02:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNAVAILABLE: Socket closed {created_time:"2022-10-21T19:23:22.773231265+02:00", grpc_status:14}]}"
148+
```
149+
Two currently more reliable ways to validate Orc8r connection are:
150+
```
151+
$ sudo docker exec magmad cat /var/log/syslog|grep heart
152+
```
153+
154+
Which should show multiple lines of:
155+
```
156+
Oct 21 13:33:43 agw-p18-2 eba229d6ac98[780]: INFO:root:[SyncRPC] Got heartBeat from cloud
157+
```
158+
159+
And, from the NMS console in Orc8r, see if the AGW has checked in recently. Sometimes, this method will indicate a bad state even when all is OK, though. C.f.:
160+
161+
![image](https://github.com/user-attachments/assets/dd11f37e-c9f7-4fd2-8334-d0a3138b6545)
162+
163+
At this point, you should have a working dockerized AGW connected to the Orc8r. You can check the overall operation of the AGW.
164+
```
165+
$ docker ps
166+
```
167+
Should show something like this:
168+
```
169+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
170+
171+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
172+
49fa338525dd linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_c:v1.9 "sh -c 'mkdir -p /va…" 2 hours ago Up 2 hours (healthy) sessiond
173+
20204d9636e0 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_c:v1.9 "sh -c '/usr/local/b…" 2 hours ago Up 2 hours (healthy) oai_mme
174+
0af7a08a2489 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) state
175+
975a3a3af345 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) policydb
176+
19a416e835ec linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) directoryd
177+
49372d59d9d8 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "bash -c '/usr/bin/o…" 2 hours ago Up 2 hours (healthy) pipelined
178+
64990c44a1da linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_c:v1.9 "/usr/local/bin/sctpd" 2 hours ago Up 2 hours sctpd
179+
bd0e794c9a3c linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) subscriberdb
180+
a21bdf353a1b linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) eventd
181+
434910ae0121 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) redirectd
182+
20636cdd3a41 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "sh -c '/usr/local/b…" 2 hours ago Up 2 hours (healthy) control_proxy
183+
68143b9804e6 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/bin/bash -c '/usr/…" 2 hours ago Up 2 hours (healthy) redis
184+
854259a6b479 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/bin/bash -c '/usr/…" 2 hours ago Up 2 hours (healthy) td-agent-bit
185+
db4a62105283 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) health
186+
fad91576d6be linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_c:v1.9 "/usr/local/bin/conn…" 2 hours ago Up 2 hours (healthy) connectiond
187+
4a704bbe52ea linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "sh -c 'sleep 5 && /…" 2 hours ago Up 2 hours (healthy) mobilityd
188+
dc15610a0c3f linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/bin/bash -c '\n /u…" 2 hours ago Up 2 hours magmad
189+
cf8dd09aeac8 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) ctraced
190+
1aa4cde13cd2 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) enodebd
191+
a215379edf82 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) smsd
192+
f3fd137e4652 linuxfoundation.jfrog.io/magma-docker-agw-test/agw_gateway_python:v1.9 "/usr/bin/env python…" 2 hours ago Up 2 hours (healthy) monitord
193+
194+
```
195+
196+
## Other tools, tips, debugging suggestions
197+
198+
# Notes to be dealt with later
199+
200+
# TODO
201+

‎agw-deployer/ansible/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/hosts.yml
2+
/env2jinja2.log

‎agw-deployer/ansible/LICENSE‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
BSD 3-Clause License
2+
3+
Copyright (c) 2022, Shubham Tatvamasi
4+
All rights reserved.
5+
6+
Redistribution and use in source and binary forms, with or without
7+
modification, are permitted provided that the following conditions are met:
8+
9+
1. Redistributions of source code must retain the above copyright notice, this
10+
list of conditions and the following disclaimer.
11+
12+
2. Redistributions in binary form must reproduce the above copyright notice,
13+
this list of conditions and the following disclaimer in the documentation
14+
and/or other materials provided with the distribution.
15+
16+
3. Neither the name of the copyright holder nor the names of its
17+
contributors may be used to endorse or promote products derived from
18+
this software without specific prior written permission.
19+
20+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
23+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
24+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
25+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
26+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
27+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎ansible.cfg‎ renamed to ‎agw-deployer/ansible/ansible.cfg‎

File renamed without changes.

‎agw-deployer/ansible/collections/requirements.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
collections:
2+
- name: community.crypto
3+
- name: community.docker
4+
- name: cloud.common
5+
- name: kubernetes.core
6+
- name: ansible.posix

‎config-orc8r.yml‎ renamed to ‎agw-deployer/ansible/config-orc8r.yml‎

File renamed without changes.

‎agw-deployer/ansible/deploy-agwc-networking.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
# Must run common-system first
3+
- name: Configure Magma AGW Networking
4+
hosts: all
5+
roles:
6+
- role: agwc-config-networking

0 commit comments

Comments
 (0)