cisco – Page 4 – Write Mem

Creating a Bootable ISO from Cisco Non-Bootable Updates

There’s plenty of great posts out there on how to make bootable CUCM/Collaboration media with paid GUI software such as UltraISO or free command line utilities like cdrtools. Inspired by those bloggers, I wanted to share a method to create bootable media that features the best of both worlds: open source utilities with easy to use interfaces, all for the low low price of free.

Software you’ll need:

7-Zip / cdrtfe

First, find an existing bootable CUCM ISO (a Red Hat or CentOS live cd will do the trick as well) and open it in 7zip.

Find the isolinux directory, highlight it, and click Extract. Choose where to save the files.

Next, take your non-bootable update dvd from Cisco and extract the contents with 7zip by right clicking the ISO -> 7-Zip -> Extract to….

Copy the isolinux directory you extracted earlier to the newly extracted Cisco ISO folder, overwriting any duplicate files.

Now open cdrtfe. Under the Data Disc tab click Options.

Under ISO image, select Use image, and browse to a location to store the finished ISO. Check Create image only, do not burn. Click Ok.

Next, click the File System button. Check the following options: Create boot disc, No disc emulation, and Create boot info table. Under the Boot image field, browse to the copied isolinux directory of your extracted ISO folder and select isolinux.bin. Click Ok.

In file explorer, browse to your extracted ISO folder and copy over all the files and folders into the cdrfte window.

Click Start to create your ISO. Once completed you should have a fully bootable ISO!

Troubleshooting TFTP Issues with Cisco Unified Real-Time Monitoring Tool (RTMT)

I’ve recently began studying for my CCNA Collaboration exam and to help with my studies I’ve built a small collaboration lab. Nothing too fancy, a 2811 with PVDMs and FXO/FXS cards, 2 3750 PoE switches, and 2 Cisco 7960 VOIP phones. My lab server has a dedicated NIC which connects to the lab network and hosts a domain controller, workstation, and CUCM 11.5 virtual machines.

All this was working well until I configured the switches to have a separate dedicated voice VLAN. The phones started having issues contacting CUCM and downloading new configuration files or firmware. If I moved the phones back to the same VLAN as CUCM the phones would work properly. Sounded like a TFTP issue to me, and here are the steps I followed to resolve the issue.

TLDR; ip tftp source-interface

First, I setup a trace to capture TFTP logs from my CUCM server.

From the CUCM Console: select Cisco Unified Serviceability from the Navigation drop down, and click Go.

On the Cisco Unified Serviceability page select Trace -> Configuration

Select your CUCM Server running the TFTP service for Server then select CM Services under Service Group.
For Service select Cisco Tftp then click Go.
Be sure Trace On is selected then change the Trace Level to Detailed.
Click Save.

With the detailed TFTP trace enabled, I tried resetting the phones to duplicate the issue. Once I verified the issue was still occurring it was time to grab the trace files.

To download the trace files you’ll need to use the Cisco Unified Real-Time Monitoring Tool which comes bundled with CUCM. To download the tool, follow these steps.

From the CUCM Administration page navigate to Application -> Plugins.

Near the bottom of the plugins page find the download link for Cisco Unified Real-Time Monitoring tools.
Download the tool for your appropriate os and install.

When you run the program it will ask you for the IP of the CUCM server and the GUI username/password.

Select Trace & Log Central. Then double click Collect Files.

Scroll down to find Cisco Tftp and select either All Servers or an individual server.

Select your download options and click Finish.

You should now have a wealth of information to dive into to try and troubleshoot your issue. As I was perusing the TFTP trace logs a few lines popped out at me.

It would appear that the phone is contacting the TFTP server, but its IP address was not part of my voice VLAN. In fact, the source IP was the IP of the outside interface of my 2811!

After some quick google’ing I came across this article on the ip tftp source-interface command. Because my router was sourcing TFTP traffic from its outside interface, CUCM was not able to route the traffic back to 7960 phone.

Simply adding the ip tftp source-interface command followed by the VLAN my CUCM server resided in resolved the issue and my phones began registering again.

COLLAB-SW-01(config)#ip tftp source-interface vlan 103
COLLAB-SW-01(config)#end
COLLAB-SW-01#wr
Building configuration...
[OK]
COLLAB-SW-01#

Zero Downtime ASA Upgrade (ASDM)

In the same vein as my last post, this update will take you through the steps of performing a zero downtime upgrade on a HA pair of ASA 5525-X’s, this time via the ASDM GUI.

Take a look here for Cisco’s official documentation, which will be the blueprint for our guide.

First, back up your configuration by going to Tools -> Backup Configurations

Download your updated ASDM and ASA software from Cisco.com
Click Tools -> Upgrade Software from Local Computer

We’ll update ASDM first, select ASDM from the drop down box and click Browse Local Files. Navigate to where you downloaded your images and select the new ASDM image then click Upload Image.

You will receive a prompt asking to set this image as the ASDM image. Click No for now. If we select Yes you will not be able to use ASDM to connect to the secondary unit and upload the new images in the later steps.

Now let’s update the ASA software. Again, click Tools -> Upgrade Software from Local Computer. Select ASA from the drop down menu and click Browse Local Files. Navigate to where you downloaded your images and select the new ASA image then click Upload Image. Click yes on the prompt.

Open a new ASDM window and connect to the standby ASA IP address. Follow the same steps as above to upload the new ASDM and ASA images. Close the ASDM window for the secondary unit when finished.
On the Primary ASDM window click Configuration -> Device Management -> expand System Image/Configuration -> click Boot Image/Configuration. Under ASDM Image File Configuration click Browse Flash and select our new ASDM image we uploaded earlier. Save the configuration.

We can now reload the secondary unit to start the upgrade process. Click Monitoring -> Properties -> Failover -> Status. Click Reload Standby.

Refresh the Failover Status page until the Secondary unit moves from a failed state to standby ready. Make note of the sw rev field. It should reflect the new ASA software version.

Force the active unit to failover to standby by clicking the Make Standby button on the Failover Status page. Refresh the Failover Status page to verify the Secondary unit is now the active unit. Now click Reload Standby to reboot the primary/standby unit and complete the upgrade.

Exit ASDM and relaunch. You should now see the updated ASDM and ASA software versions under Device Information.

Zero Downtime ASA Upgrade (CLI)

So you have stateful failover configured on your pair of Cisco ASAs and need to upgrade ASDM or the os? Maybe a critical security vulnerability was discovered with the software and you need to upgrade them ASAP. With stateful failover, we can perform a zero downtime upgrade on our ASAs to minimize end user disruption. Below are the steps I used to upgrade a pair of ASA 5525-X’s using the command line interface. You can find Cisco’s documentation for upgrading an Active/Standby Failover Configuration here.

First, back up your current configuration!

ciscoasas# copy running-config tftp://10.1.1.10/ciscoasa-config.txt

Source filename [running-config]?

Address or name of remote host [10.1.1.10]?

Destination filename [ciscoasa-config.txt]?
Cryptochecksum: 495070f8 a3de0875 adfc16d6 18c04464
!!
4996 bytes copied in 0.660 secs

Download your updated ASDM and ASA software from Cisco.com
Copy new ASA operating system to active and standby units.

ciscoasa# copy tftp://10.1.1.10/asa952-smp-k8.bin disk0:/asa952-smp-k8.bin

Address or name of remote host [10.1.1.10]?

Source filename [asa952-smp-k8.bin]?

Destination filename [asa952-smp-k8.bin]?

Accessing tftp://10.1.1.10/asa952-smp-k8.bin...
!!!!!!!
Writing file disk0:/asa952-smp-k8.bin...
!!!!!!!
82593792 bytes copied in 211.300 secs (391439 bytes/sec)

ciscoasa# failover exec mate copy /noconfirm tftp://10.1.1.10/asa952-smp-k8.$

Accessing tftp://10.1.1.10/asa952-smp-k8.bin...
!!!!!!!
Writing file disk0:/asa952-smp-k8.bin...
!!!!!!!
82593792 bytes copied in 192.570 secs (430176 bytes/sec)

Copy new ASDM software to active and standby units.

ciscoasa# copy tftp://10.1.1.10/asdm-752.bin disk0:/asdm-752.bin

Address or name of remote host [10.1.1.10]?

Source filename [asdm-752.bin]?

Destination filename [asdm-752.bin]?

Accessing tftp://10.1.1.10/asdm-752.bin...
!!!!!!!
Writing file disk0:/asdm-752.bin...
!!!!!!!
25627616 bytes copied in 57.570 secs (449607 bytes/sec)

ciscoasa# failover exec mate copy /noconfirm tftp://10.1.1.10/asdm-752.bin d$

Accessing tftp://10.1.1.10/asdm-752.bin...
!!!!!!!
Writing file disk0:/asdm-752.bin...
!!!!!!!
25627616 bytes copied in 58.190 secs (441855 bytes/sec)

Verify the current boot images, making note of the boot order. Remove the current image and set the config to boot from the newly uploaded image followed by the old image as a back up.

ciscoasa# sh running-config boot system
boot system disk0:/asa922-4-smp-k8.bin
ciscoasa# conf t
ciscoasa(config)# no boot system disk0:/asa922-4-smp-k8.bin
ciscoasa(config)# boot system disk0:/asa952-smp-k8.bin
ciscoasa(config)# boot system disk0:/asa922-4-smp-k8.bin
ciscoasa(config)# end
ciscoasa# sh running-config boot system
boot system disk0:/asa952-smp-k8.bin
boot system disk0:/asa922-4-smp-k8.bin

Configure the ASA to use the new ASDM image

ciscoasa(config)# asdm image disk0:/asdm-752.bin

Save your configuration and reload the secondary standby unit.

ciscoasa# wr
Building configuration...
Cryptochecksum: c4c58b75 ea6ca884 235cc59e 05d6f114

5030 bytes copied in 0.670 secs
[OK]
ciscoasa# failover reload-standby
ciscoasa#
************WARNING****WARNING****WARNING********************************
   Mate version 9.5(2) is not identical with ours 9.2(2)4
************WARNING****WARNING****WARNING********************************
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate

Verify the standby unit is back up and running the new software version.

ciscoasa# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: FO GigabitEthernet0/7 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 216 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.2(2)4, Mate 9.5(2)
Last Failover at: 03:14:15 EDT Apr 22 2016
        This host: Primary - Active
                Active time: 605706 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.2(2)4) status (Up Sys)
                  Interface Inside (10.1.1.1): Normal (Monitored)
                  Interface management (192.168.1.1): No Link (Waiting)
                slot 1: SFR5525 hw/sw rev (N/A/5.3.1-152) status (Up/Up)
                  ASA FirePOWER, 5.3.1-152, Up
        Other host: Secondary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.5(2)) status (Up Sys)
                  Interface Inside (10.1.1.2): Normal (Monitored)
                  Interface management (0.0.0.0): No Link (Waiting)
                slot 1: SFR5525 hw/sw rev (N/A/) status (Init/Up)

Make the active primary unit the new standy unit

ciscoasa# no failover active

You may need to re-establish your SSH connection. Log back into the ASA and verify that the secondary unit is now the active unit. Reload the primary/standby unit, wait a few minutes and verify that both units are now running identical code.

ciscoasa# failover reload-standby
ciscoasa# Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ciscoasa# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: FO GigabitEthernet0/7 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 216 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.5(2), Mate 9.5(2)
Last Failover at: 10:34:53 EDT Apr 29 2016
        This host: Secondary - Active
                Active time: 429 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.5(2)) status (Up Sys)
                  Interface Inside (10.1.1.1): Normal (Monitored)
                  Interface management (192.168.1.1): No Link (Waiting)
                slot 1: SFR5525 hw/sw rev (N/A/5.3.1-152) status (Up/Up)
                  ASA FirePOWER, 5.3.1-152, Up, (Monitored)
        Other host: Primary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.5(2)) status (Up Sys)
                  Interface Inside (10.1.1.2): Normal (Monitored)
                  Interface management (0.0.0.0): No Link (Waiting)
                slot 1: SFR5525 hw/sw rev (N/A/) status (Init/Up)

Stateful Failover Logical Update Statistics
        Link : FO GigabitEthernet0/7 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         101        0          97         0
        sys cmd         93         0          92         0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        1          0          0          0
        UDP conn        0          0          0          0
        ARP tbl         5          0          4          0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKEv1 SA    0          0          0          0
        VPN IKEv1 P2    0          0          0          0
        VPN IKEv2 SA    0          0          0          0
        VPN IKEv2 P2    0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        SIP Tx  0          0          0          0
        SIP Pinhole     0          0          0          0
        Route Session   1          0          0          0
        Router ID       0          0          0          0
        User-Identity   1          0          1          0
        CTS SGTNAME     0          0          0          0
        CTS PAC         0          0          0          0
        TrustSec-SXP    0          0          0          0
        IPv6 Route      0          0          0          0
        STS Table       0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       12      336
        Xmit Q:         0       29      300
ciscoasa#

And there you have it. If you’d like, after monitoring and verification, you can remove the old ASA and ASDM images from the boot order commands as well as the disk to keep things clean.

ciscoasa# delete /replicate disk0:/asa922-4-smp-k8.bin

Delete filename [asa922-4-smp-k8.bin]?

Delete disk0:/asa922-4-smp-k8.bin? [confirm]

ciscoasa# delete /replicate disk0:/asdm-7221.bin

Delete filename [asdm-7221.bin]?

Delete disk0:/asdm-7221.bin? [confirm]
ciscoasa# sh run boot
boot system disk0:/asa952-smp-k8.bin
boot system disk0:/asa922-4-smp-k8.bin
ciscoasa# conf t
ciscoasa(config)# no boot system disk0:/asa922-4-smp-k8.bin
ciscoasa(config)# end
ciscoasa# sh run boot
boot system disk0:/asa952-smp-k8.bin
ciscoasa#

Configuring Stateful Failover on a Cisco ASA HA Pair

The ASA, Cisco’s Adaptive Security Appliance, has been around for over 15 years and has since become an ubiquitous network security solution, securing networks the world over.

Because it is such a critical device in our networks, it has become best practice to deploy these security appliances in a resilient and highly available configuration.

Currently, Cisco supports Active/Active as well as Active/Standby failover. This article contains a simple example of how to configure Active/Standby stateful high availability on a pair of Cisco ASAs, where one unit acts as the primary ASA and a standby unit becomes active once a failover has occurred. When stateful failover is enabled, connection states are continuously passed between the active and standby units keeping session information available to the new active unit. Please note that both ASAs must be running identical hardware and software versions.

Detailed instructions from Cisco on how to configure Failover can be found here: http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/ha_failover.html

Each interface on the primary ASA will need an additional “standby” IP address, for example:

interface GigabitEthernet0/0
description Inside Interface
nameif Inside
security-level 100
ip address 10.1.1.1 255.255.255.0 standby 10.1.1.2

Specify failover interface on the primary ASA

ciscoasa(config)# failover lan interface failover GigabitEthernet0/7

Configure failover link IP address

ciscoasa(config)# failover interface ip failover 192.168.1.1 255.255.255.0 standby 192.168.1.2

Configure shared failover key

ciscoasa(config)# failover key PASSWORD123

Configure ASA as primary

ciscoasa(config)# failover lan unit primary

Enable stateful failover

ciscoasa(config)# failover link failover GigabitEthernet0/7

Enable failover

ciscoasa(config)# failover

On the secondary ASA we’ll need to do some similar configuration:

ciscoasa(config)# failover
ciscoasa(config)# failover lan unit secondary
ciscoasa(config)# failover lan interface failover GigabitEthernet0/7
INFO: Non-failover interface config is cleared on GigabitEthernet0/7 and its sub-interfaces
ciscoasa(config)# failover link failover GigabitEthernet0/7
ciscoasa(config)# failover interface ip failover 192.168.1.1 255.255.255.0 standby 192.168.1.2
ciscoasa(config)# failover key PASSWORD123

If everything is configured properly you should see some console output regarding configuration replication.

Detected an Active mate
Beginning configuration replication from mate.
End Configuration Replication to mate

Now verify the status of failover with a show failover:

ciscoasa# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/7 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 216 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.2(2)4, Mate 9.2(2)4
Last Failover at: 03:14:15 EDT Apr 22 2016
 This host: Primary - Active
 Active time: 469208 (sec)
 slot 0: ASA5525 hw/sw rev (1.0/9.2(2)4) status (Up Sys)
 Interface Inside (10.88.31.3): Normal (Monitored)
 Interface management (192.168.1.1): No Link (Waiting)
 slot 1: SFR5525 hw/sw rev (N/A/5.3.1-152) status (Up/Up)
 ASA FirePOWER, 5.3.1-152, Up
 Other host: Secondary - Standby Ready
 Active time: 2808 (sec)
 slot 0: ASA5525 hw/sw rev (1.0/9.2(2)4) status (Up Sys)
 Interface Inside (10.88.31.4): Normal (Monitored)
 Interface management (0.0.0.0): No Link (Waiting)
 slot 1: SFR5525 hw/sw rev (N/A/5.3.1-152) status (Up/Up)
 ASA FirePOWER, 5.3.1-152, Up

Stateful Failover Logical Update Statistics
 Link : FO GigabitEthernet0/7 (up)
 Stateful Obj xmit xerr rcv rerr
 General 119141 0 62582 0
 sys cmd 62578 0 62578 0
 up time 0 0 0 0
 RPC services 0 0 0 0
 TCP conn 7 0 0 0
 UDP conn 0 0 0 0
 ARP tbl 56548 0 3 0
 Xlate_Timeout 0 0 0 0
 IPv6 ND tbl 0 0 0 0
 VPN IKEv1 SA 0 0 0 0
 VPN IKEv1 P2 0 0 0 0
 VPN IKEv2 SA 0 0 0 0
 VPN IKEv2 P2 0 0 0 0
 VPN CTCP upd 0 0 0 0
 VPN SDI upd 0 0 0 0
 VPN DHCP upd 0 0 0 0
 SIP Session 0 0 0 0
 Route Session 8 0 0 0
 Router ID 0 0 0 0
 User-Identity 0 0 1 0
 CTS SGTNAME 0 0 0 0
 CTS PAC 0 0 0 0
 TrustSec-SXP 0 0 0 0
 IPv6 Route 0 0 0 0
 STS Table 0 0 0 0

 Logical Update Queue Information
 Cur Max Total
 Recv Q: 0 10 62681
 Xmit Q: 0 1 373300