MOC VLAN Distribution¶
This document lays out how the VLANs are distributed between our various clusters.
It also talks (briefly) about how the hardware, and the infrastructure services we are running.
There are (or will be) environment specific documents describing the networking setup with VLAN, network and host information.
oVirt¶
We have a 3 node oVirt setup for hosting crucial services like DHCP, DNS, FreeIPA, Foreman etc.
All these services are highly available VMs.
Configuration of oVirt nodes:
| Name | RAM | Cores/threads | Storage |
|---|---|---|---|
| ov1.massopen.cloud | 384 GB | 24/48 | 10X1TB |
| ov2.massopen.cloud | 384 GB | 20/40 | 10X1TB |
| ov3.massopen.cloud | 384 GB | 20/40 | 10X1TB |
Services that will be hosted on the oVirt setup
FreeIPA server (one will be duplicated at BU),
Foreman (might be multiple for different env)
RHEL repo server, just 1 for all environemtns. Public IP with limited access.
HIL server (maybe multiple for different environments)
Gateway/router for HIL/BMI services (again, could be multiple)
HIL client (again, could be multiple)
DNS servers for massopen.cloud, openapp.cloud, mocapps.cloud
1 Management instance and 2 DNS servers for massopen.cloud
same thing for the other 2 domains.
a total of 6 VMs
Additional internal DNS
Kaizen’s RadosGW will be here.
cacti - for statistics
Nagios - for monitoring.
Other ssh gateways.
IPMI gateways.
Each environment gets around 300 to 400 VLANs which will leave room for expansion for more environments.
Below is the detailed distribution of the VLANs.
MOC VLANs¶
VLANs common to all environments:
| VLAN(s) | Description | Subnet(s) |
|---|---|---|
| 1 to 200 | Reserved by University's IS&T for providing public IPs | - |
| 127 | NEU Public IP for infrastructure | 129.10.5.0/24 |
| 3800 | For connecting environments, but we don't know about bandwidth, or the route | - |
| 3801-3803 | CSAIL Floating IPs | 128.31.XX.YY/22 |
Kaizen¶
Hardware Summary
25 openstack computes (Dell IBs)
3 oVirt nodes (Dell IBs)
48 Cisco nodes. (Will be available for Elastic HW)
3+2 openstack controllers (Dell SBs)
8 ceph OSDs (right now, we can change this).
64 Dell SB nodes that can be elastic with 2 drives each which we can redistribute. Extra 5 drives for spares. Total Elastic Nodes = 48 Cisco + 64 Dell SB = 112 nodes.
VLANs
VLAN range : 201 to 700 (total 399 VLANs)
201 to 600 - 10G networks
601 to 700 for IPMI/management on 1G networks.
| VLAN | Description | Subnet |
|---|---|---|
| 127 | Openstack public facing services (horizon, keystone) | 129.10.5.0/24 |
| 201 | Foreman Provisioning. SNMP for OpenStack and Ceph | 172.16.0.0/19 |
| 202 | OpenStack internal API) | 172.16.32.0/19 |
| 203 | OpenStack tenant network | 172.16.64.0/19 |
| 204 | Intranet (routable to internet). SNMP for client nodes. | 172.16.96.0/19 |
| 205 | Gluster/VM migration - oVirt | 192.168.0.0/24 |
| 206 | OStack isolation native vlan for trunk only ports | N/A |
| 207 | For OCT/UMass Switch Management | 10.1.0.0/24 |
| 208 | ESI control plane | -- |
| 209 | Openshift Internal (Baremetal 4.x) | -- |
| 210 | NFS for zero cluster | 10.253.0.0/23 |
| 211 | New England Storage Exchange (NESE) | 10.120.0.0/22 |
| 212 | Openshift Staging Internal (ocp-staging) |
-- |
| 213 | Ceph Cluster iSCSI | 10.21.0.0/22 |
| 249 | Ceph Cluster (internal) | 192.168.40.0/23 |
| 250 | Ceph public (for clients) | 192.168.0.0/19 |
| 259 | Staging - Foreman | 172.17.0.0/19 |
| 270 | Staging - Internal API | To be decided |
| 271 | Staging - Tenant Network | To be decided |
| 272 | Staging - Public Network | To be decided |
| 273 | Staging - OS Stack isolation for trunk only ports | To be decided |
| 290 | OKD internal network | 10.5.0.0/24 |
| 911 | For OCT/UMass nodes IPMI | 10.2.0.0/19 |
| 912 | For OCT/UMass nodes IPMI for operate first | 10.3.0.0/19 |
| 913 | For OKD nodes in OCT4 IPMI | 10.4.0.0/19 |
| 3802 | OpenStack Tenant Floating IPs | 128.31.24.0/22 |
IPs in VLAN 204 will be assigned based on rack and unit number, rest will be regular DHCP. Eg; a node in cage 3 unit 20 will get an ip like 172.16.96+3.20
VLANs 251 to 600 will be reserved for HIL and BMI. If each tenant needs at least 5 VLANs for their project in HIL, then with approximately 350 VLANs we can support a decent number of tenants with room for expansion.
Kaizen Floating IP Split:
3802 (Kzn) is 128.31.24.0/22 (i.e., 2K IP)
Start 128.31.24.129 - End 128.31.27.254
Kzn split
Old: 128.31.24.129,128.31.25.255
New: 128.31.26.0,128.31.27.254
Engage1¶
Hardware Summary
12 OpenStack computes right now. 3 nodes with GPUs will be moved to Kaizen so we’ll have 9 OpenStack computes.
2 OpenStack Controllers
3 monitors and 10 OSDs.
4 services nodes. And some more extra nodes.
VLANs
VLAN Range: 701 to 1000 (total 299 VLANs)
701 to 900 - 10G networks.
901 to 1000 for IPMI/management on 1G networks.
| VLAN | Description | Subnet |
|---|---|---|
| 10 | Openstack public facing services (horizon, keystone) | DHCP from CSAIL 128.52.60.0/22 |
| 701 | Foreman Provisioning. SNMP for OpenStack and Ceph | 172.17.0.0/19 |
| 702 | OpenStack internal API) | 172.17.32.0/19 |
| 703 | OpenStack tenant network | 172.17.64.0/19 |
| 704 | Intranet (routable to internet). SNMP for client nodes. | 172.17.96.0/19 |
| 749 | Ceph Cluster (internal) | -- |
| 750 | Ceph public (for clients) | -- |
| 3801 | OpenStack Tenant Floating IPs | 128.31.20.0/22 |
VLAN 751 to 1000 will be reserved for HIL and BMI.
Kumo¶
Hardware Summary
16 Dell Blades
1 kumo-storage hosting all services.
1 kumo-services, and 1 kumo-emergency.
VLANs
VLAN Range: 1001 to 1300 (total 299 VLANs)
1001 to 1200 for 10G stuff
1201 to 1300 for IPMI/Management on 1G networks.
| VLAN | Description | Subnet |
|---|---|---|
| 105 | Public IPs from BU | 192.12.185.0/24 |
| 1004 | Intranet (routable to internet). SNMP for client nodes. | 172.18.0.0/19 |
| 3808 | Public IPs | 128.31.28.0/24 |
Rest of the VLANs will be reserved for HIL in this environment.
This leaves us with VLANs 1300 to 4094 for expansion in the future
NERC¶
| VLAN | Description | Subnet |
|---|---|---|
| 2476 | NERC-Admin network 1->eth0:provisioning network(vlan 100)-deploying images/DHCP> PAT out | 10.255.0.0/24 |
| 2472 | eth0.101: internal api (vlan 101) - internal to node | 172.18.0.0/23 |
| 2473 | eth0.102: tenant private (vlan 102) - VMs sit here, private network/VXLAN | 172.18.4.0/23 |
| 2477 | NERC - OBM/MGMT Network 1 -> eth1: management/ipmi - undercloud to DRAC/BMC 10.255 | 10.255.1.0/24 |
| 2478 | eth2: storage (vlan 103) - ceph/jumboframe? 10.255 | 10.255.2.0/24 |
| 2471 | *eth3: external network (vlan 200) - public API / endpoints | 140.247.152.0/27 |
| 2470 | *eth4: floating ip (vlan 201) - tenant public IPs | 140.247.152.128/25 |
undercloud needs these - 2476, 2477 (edited) NERC checking with nNck see if it’s possible get 2476 to be a 192.168.24.0/24 CIDR (edited)
OCT/cloudlab¶
Cloudlab side’s dataplane is fully dynamic, they’d block out MOC vlans to avoid conflict, and agree on a range. The proposed range right now is < 1024 for cloudlab, > 1024 for OCT (with cloudlab blocking MOC existing vlan IDs).
| VLAN | Description | Subnet |
|---|---|---|
| 84 | cloudlab-1 UMass Public IPs | 198.22.255.0/24 |
MOC connections to OCT/Umass¶
| Description | MOC switchport | OCT/Umass switchport |
|---|---|---|
| 40G cable for data plane | Cage 15 Unit 43 Brocade (Fo 2/0/52) (Port Channel 2) | OCT-HUB-2 (Port Channel 11, Fo 1/27/1) |
| 40G cable for data plane | Cage 15 Unit 42 Brocade (Fo 1/0/52) (Port Channel 2) | OCT-HUB-1 (Port Channel 11, Fo 1/27/1) |
| 1G cable for IPMI network (disabled) | Cage 15 Unit 44 port 45 | - |
While the 1G connection still exists, we have disabled it since we can reach the OCT IPMI network over out data network.