Attack of the ZTP Server

Sub title: Simplified Aristra Zero Touch switch deployment

DISCLAIMER: You can only trust a mad scientist so much! 

________________________________________________________________

Seems like I’m going a mile a minute here, but there is a great urgency to get this information out to as many people as possible, and as soon as possible. If you are building spine/leaf data centers with Arista switches, how would you like to provide a baseline configuration on a 4 spine and 8 leaf network and never touch one console cable?  You can get started just by scanning the barcode on the outside of the box!

ZTP is the nirvana state that all of us who have been doing this for a while dream of (while we’re not dreaming of other things!).  ZTP means you can pull the switch out of the box, plug in a power cable and connect the management port to the network and the switch will configure itself. It’s a neat trick that can save you a lot of keystrokes. There are several different ways of doing this and they are documented on the interwebs. No need to hash that out here. Let’s set up a simple lab and I’ll show you the way we do it here in the laboratory!

cvp

Above is a diagram of my lab. I am running 10x Arista vEOS switches in vagrant on a really big Windows 10 PC. All of them are set to zerothouch enable by default. This means if they can’t find a “startup-config file in flash, they will start broadcasting DHCP request out all ports.

You don’t need all 10, one or two will do, but I like to show off every now and then.

The ZTP server that we will implement will have a DHCP scope enabled and Option 67 as well to tell the switch who to “talk” to, to get is configuration, and possibly software as well.

First thing first! You will need to get my repo off of github at this location: https://github.com/xod442/aristaZTP

From a shell prompt on your linux workstation,

sudo git clone https://github.com/xod442/aristaZTP.git

You will see a newly created directory called aristaZTP.

cvp

I’m using the Ubuntu Mate distribution. It is probably one of the most “windows like” versions of linux and comes with “caja” a file explorer. Snap up a copy here: https://ubuntu-mate.org/download/

HIP TIP OF THE WEEK!

In Ubuntu’s command shell, to avoid typing sudo in front of just about every command you can “sudo –s” and once you enter the password, you will be at a “hashtag” prompt…..sudo no more!

Once you’re at the “#” prompt, type caja and spawn a very powerful file browser.

cvp

Now let me explain the different files, the README.md has some interesting tid bits and should be read at you leisure. There is a tftpboot directory, this is where we keep our csv variables and the configuration template. There is an interfaces file in my screenshot but these are not the droids you are looking for…….Finally, the setup.sh is a shell script that will configure the entire ZTP server from scratch. You just have to change a few things.

Setup.sh

cvp

You’re looking at the first of three sections in the setup.sh file. Take a close look at what has a # in front of it. In the DHCP section the #apt-get install isc-dhcp-server is commented out. If you want to install it, and you do, get rid of the “#” in front of the command and it will change color.

Look a little more at the things you need to change.

Subnet 172.20.0.0 netmask 255.255.255.0. You might want to change this to match your own lab addressing. Take caution when working with multiple DHCP servers on the same network. Change the DHCP range and the ‘option tftp-server-name “172.20.0.1”;’ to match the eth0 address of your ZTP server. A quick ifconfig should reveal the address you will need.

Next the option for the boot file. Option bootfile-name “boot.py” This is the file that you want the switch to start processing when it boots. This could be the switches startup-config, but you would need to do static DHCP entries and have a boot file for every switch! No such shenanigans here! Our file has the name boot.py which means it’s a python script and when the switch downloads the boot.py file it will start configuring the switch for you.

Let’s look at the other sections of the setup.sh file.

There are two more sections in the script. One to setup the tftp server and the other sets up a ftp server but we don’t really need it for our demonstration, but what the heck, fire it up anyway!

cvp

PAY CLOSE ATTENTION to the directories listed in the tftp server sections. These must point to where you have the tftpboot directory saved. Pay attention! Change them to match your environment.

Once you made the necessary changes to the setup.sh file, run it # ./setup.sh…………..didn’t run? Try change the permissions of the file like this # chmod 775 setup.sh…….try it again.

Now the DHCP and TFTP should be up and running but you will need to CHMOD the dhcpd.leases file. It’s a pain in the butt and I have not figured out a way to not have to do this but it only needs done when you boot up the ZTP server.

cvp

OK, just a couple other things to do for successful booting of our switches. The one thing you absolutely need to know is the MAC address of all the switches. You will need this information and should be easily obtained without having to console into each switch. Barcode lables on the boxes, Perhaps they are listed on your PO?

Once we have the MAC addresses, we can pair it up with some repetitive information and just about anything you would like to see configured in the “base configuration”. I keep things like the real IP address I want the switch to have and the Cloud Vision Portal credentials. You can maintain this information in an Excel spreadsheet or you can use “switchdb” A flask application I wrote for just such things. You can find switchdb here: https://github.com/xod442/scriptsonly/tree/master/switchdb

Hint, Hint! install switchdb on the ZTP server…easy peazy!

Once we have all of our MAC addresses in the CSV file, you will need to store the CSV file in the tftpboot directory with the name of “varMatrix.csv”. It should look something like this.

cvp

The last piece of this puzzle is the template.txt file. It is used to create the switches configuration. In this file, everywhere we have a dollar sign “$” it “pairs” a variable in the CSV file.

cvp

You can modify this template file to add anything else that might be a common configuration for all switches. Don’t get too crazy, the next blog post will cover Ansible and we will really start deploying configurations then

NOTE: Once the switches boot with this template, they will be ready to import into Cloud Vision Portal and further configurations can be deployed there!

The real “magic” behind this ZTP process is the boot.py file. I had to do some interesting things to make this work but the process is fairly simple. Here’s the process.

Switch boots and gets DHCP and location of the boot.py file, downloads and executes it

boot.py copies over the varMatrix.csv file and the template.txt file to the switches flash.

boot.py opens both files for reading.

boot.py finds the MAC address of the local switch and compares it to the MAC field in the CSV.

If there is a match, the dollar sign variables in the template are replaced with the CSV variables

boot.py saves the result to the switches flash drive and the switch reloads

The switches will be ready to be managed after they complete their reload. There are a couple different directions we can go from here. We can import the switches into CVP check out this blog post for more information. http://techworldwookie.com/?p=205

Next up: Whipping these switches into shape with Ansible and Docker!

Beginner’s guide to the Universe and Cloud Vision Portal Episode 2

Part Two: Get that switch in the game!
DISCLAIMER: You can only trust a mad scientist so much!
_____________________________________________________________________
Thanks for hanging in there with me on this long blog post. We are going to go a little deep in building the configets for our spine leaf network using Arista Cloud Vision Portal (CVP). The CVP server ships as an OVA and with a couple of clicks we can have it up and running on our ESXi host. You’re going to need about 10 Gig of RAM to get this going. At a bare minimum I have had it running with 7 Gig, but I do not recommend it. The first step is to login with the cvpadmin user and run the setup script. You will be prompted to change the “Root” password, no weak stuff here….I tried.
cvp
After we accomplish this we need to run the setup script. I selected “s” for standalone mode. Answer the prompts according to your own environment.
cvpmain
Select “r” and “a” for apply and watch the boot process. I had a lot of issues with the nic coming up with the correct address. If this happens to you, quit the script with a “q” and login as “root”, remember, you just set the password for the root user. Once the network connectivity issues are resolved run the standalone script once more. It will take about 15 minuets for the CVP server to boot.

Look in /etc/sysconfig/network-scripts/ifcfg-eth0 for any mis-configurations. I had issues with the NTP server as well so I just used the “IP” address instead of the DNS name. All good, let’s move on. Point your favorite web browser to the address of the CVP server and you should be greeted with the login screen.
cvpmain
We are going to login to CVP with the “cvpadmin” user. At the first login attempt you will use cvpadmin as the userid and cvpadmin for the password as well. You will be prompted to change the password for cvpadmin.
cvpmain
Success!
musthave

CVP Main Menu

If you read part one of this blog post you should be a little familiar with navigating around CVP. The main menu is your starting point. Let’s verify our containers are just the way we want them using the Network Provisioning page. Arrange them to your liking. We will start with at least one of our switches are in the inventory. We’ll use it’s configuration to build our configlets.

Each container in the tree can host a particular part of the overall switch configuration, a configlet. As we traverse down the tree to the switch we pick up configlets and stitch them together to make up the overall configuration.

In the blog we will create all the configlets for our two spine/two leaf network.
musthave
If we take a closer look at our switch that we have added to the inventory, we see it is yellow and if we hover on the image we will get a pop-up that tells us the devices configuration is out of sync. This means that the configuration of the switch does not match what are configlets “think” it should be.
tree
The process starts by “right-clicking” the device and selecting “manage-configet” from the network provisioning screen.
tree
On this screen you will select “validate” from the bottom of the page and CVP will pull the configuration from the switch and compare it to what the configurations look like from cascading through the network provision tree.
add
Seems we have some differences between the two! If you see items marked in red in the running configuration window to the right, this indicates that you do not have a configlet for that part of the configuration and you will need to create it.
add
In the running configuration window, highlight the entire configuration and paste it into an Excel spreadsheet. It’s a little weird, but believe me, it makes it easier to see what’s going on.
OK, at this point we are going to start rearranging some of the configuration statements. We will group them into common items that can be found on all switches and unique items that only are available on the switch, things like router-id, loopback address, etc, etc.
add
Once we get our configuration statements arranged the way you want them, go back the CVP screen and let’s add a new configlet. Pull down the plus “+” sign menu and select Configlets.
add
Give the configlet a title and copy and paste the configuration statements from the excel SS into the configlet editor. Save.
add
Here we have copied out the rest of the unique information for the other switches to help build the configlets for all the devices in our network. Obviously, this won’t scale but it is a good way to learn what configlets are and help you keep track of what is going on as you’re just starting out.
add
Looking good, I think we can move on! Configlets done!
add

Assigning configlets to containers

We need to assign the common configlets to a top level container. From the network provision screen right-click on your top most container and manage configlets. Select the common configlets and assign them to this container.
short
Now, right-click on the switch and assign the configlets to the individual switch.
short
The unique configlets will be at the left and you will need to place a check mark on which ones you want to add. Don’t worry if the common configlets show up on the right. They are supposed to. Select validate from the bottom of the screen.
short
The main goal of this exercise is to get the red indicators in the running configuration window to no longer appear. You either make another configlet or remove the configuration from the switch.
short
It is possible to use the Reconcile at the bottom of the screen and this will clean up the remaining red items as well. I don’t really recommend this, it is up to you.
short
For me it is difficult to tell if these will be common across all switches or not. I just keep making configets until all the red is gone. As shown in the following image.
short

TASKS

Now once we save the validation screen we will see back in the network provision screen that something new has appeared. A small letter “T” in a yellow circle tells us that we have tasks that need to be pushed.
short
At the top left corner of the screen get to the short cut menu and navigate to the tasks page and have a look.
short
On the far right of the following image, we see the tasks are pending.
short
Just above the tasks pending notification there is a small circle icon with an arrowhead inside. Click on this to start deploying the task(s). At this point they are only staged.
short
If the deployment is successful you can look at the network provision screen and see the color of the switch icon has changed to purple.
short
Now to finish this up we will add the other three switches and with any luck we will automatically configure the switches common elements.
short
Don’t forget to add the unique configlets to the new switches and deploy the new tasks
short
We have now completed this blog post on configlets and how to use them to configure our network devices. If we add more switches to other containers we will have to repeat this process. One of the benefits of doing our configurations in this format, is we can add ACL’s to the sector 5 container and we don’t need to be too worried about what configlets are in the others.
Next up, we will look at writing a API script to log into CVP and back these configlets up! Stay Tuned.