An Element represents the Device-under-Test. The Element could be any network device – a swtich, router, hub, work-station, etc. that is being tested. The Element must offer an interface (telnet, ssh) through which the tests can be conducted.
Test Department usually has several teams of Test Engineers working on the qualification of network device software as well as hardware. Often multiple testbeds are used to speed up the process of qualifying a network software or hardware release. It becomes necessary to be able to replicate a particular test condition across multiple testbeds. This is particularly true when the Test and Development teams work together to resolve any problems found. For this to happen, it is important that the testbeds are ‘similar’. The exact implication of ‘similarity’ depends on the type of network device and the testcase and cannot be generalized. However, we can consider an example. If the testcase requires that the number of packets received by a network interface be equal to the number of packets transmitted by another network interface, then in order to verify this testcase we need a network device with two network interfaces with a traffic generator connected to one interface and a traffic analyzer connected to the other interface. For the purposes of this testcase, all testbeds that include at least one such network device are considered to be ‘siimilar’.
A common problem that haunts the testcase development process is ensuring testcase portability. Let us consider an example. If a testcase includes a command to check the incoming packet count on a port, say ‘GigabitEthernet 1/0’, that testcase would not be useful on a network device where the port ‘GigabitEthernet 1/0’ does not exist. This is because the command to check the incoming packet count will fail due to the absence of the port ‘GigabitEthernet 1/0’. Usually, to circumvent this problem, testcases are developed using a scripting language where the port name (here – ‘GigabitEthernet 1/0’) is converted into a variable, say ‘ifname’. A separate file contains the mapping between the variable ‘ifname’ and the actual value ‘GigabitEthernet 1/0’. Let us call this file ‘Properties file’. It is clear that we need one Properties file per testbed. This Properties file contains the mapping between the variables used in the script and the actual value for those variables for the specific testbed to which the Properties file belongs. A testcase that does not include any testbed specific names or addresses and that can be executed on ‘similar testbeds’ is considered to be a portable testcase. In other words, ‘Portable Testcases’ can be executed without any modification on ‘similar testbeds’.
A Property is just a name-value pair. Let us go back to the example we discussed in a previous section – using ‘ifname’ to represent the interface name ‘GigabitEthernet 1/0’ for a particular network device. In this case, (ifname, GigabitEthernet 1/0) form a Property. A sample list of properties for a particular network device follows.
How do we decide to create a Property for a particular attribute (router id, interface name, etc) of a network device? Usually, it is not necessary to create a Property for each attribute of a network device. The list of attributes we should create a Property for is decided by the testcase that is to be executed on that network device.
A Property-set is a group of properties. Usually each testbed has its own Property-set. A Property-set makes it easier to associate multiple properties with a testbed. In the table below, a Property-set ‘testbed1PropertySet’ groups together multiple properties that describe a particular testbed.
A Property-set makes it easier to create portable testcases as we will see in the following sections.
Publishing your work
Zipline supports a distributed mode of operation wherein you work on your Test Automation in your own Zipline Environment and later, when you are ready, you can choose to publish your work for use by your team members.
You can create Testbeds and Property-Sets and later publish them so that others can view and make use of them. Only you can edit or delete your Testbed & Property-Set configurations.
DUT Version is simply the network device software or hardware revision number. You may specifiy a DUT Version during Testcase Replay in order to track your Testcase results across multiple software or hardware versions.
Zipline allows you to log your Testcase Replay sessions. You can select the location where the log files generated by Zipline should be stored. Zipline automatically generates a unique log file name based on the Testcase name and the Testcase Replay timestamp.
How often have you run an overnight regression test and discovered the next day morning that somewhere during the regression, a network device experienced an unexpected system error? Because your testcases were not looking for this system error, they would have continued their execution uninterrupted. The network device may have lost important information about the root cause of the system error.
Zipline allows you to configure a set of text patterns that qualify as a system error for your network devices. Upon encountering any of these text patterns at any time in the command output, Zipline takes the necessary configured action – which may be to pause the testcase execution.
Test Execution Delay
Consider a scenario where you use Zipline to create a Testcase where you capture about 15 commands from a telnet session with the network device. You would have typed the commands, reviewed the output and then decided the next command to type in. Let us assume the telnet session lasted about 10 minutes during which you executed 15 commands. Zipline would have populated your testcase with these commands.
You have a choice as you decide to replay this testcase using Zipline.
|Delay Mode||Conditional Action|
|Immediate||Execute the commands as soon as possible|
|Captured Delay||Execute the commands exactly as they were captured with the typing delay as well as inter-command waits|
You can either select ‘Immediate’ mode where the commands are executed as soon as possible by Zipline or select ‘Captured Delay’ where the commands are executed exactly as they were captured.
Test Failure Handling
A Testcase may include multiple test conditions. Testcase Replays can be configured to pause execution if a test condition fails.
There are two different controls that allow different levels of granularity. A global switch pauses testcase execution upon encountering any test condition failure. Whereas, Conditional Actions offer a more granular approach wherein you can select on a test-by-test basis whether you want to pause the testcase execution if that test fails.
Test Resources are the components that are necessary in order to capture a new testcase or replay an existing testcase. Test Resources consist of a Testcase or a Test-group and a Testbed.
Zipline allows you to replay a testcase on a testbed. During the Testcase Replay process, Zipline sends the commands from the testcase to the respective network devices. Zipline also keeps track of the output sent back by the network device. If you configured any tests, or included a Tracker, Runner or a Repeat-Group in the testcase, those are executed as well. Zipline stores information about the Testcase Replay in order to allow you to retrieve the details later.
A Test-group is a collection of Testcases. You can create a Test-group by adding one or more Testcases to it. A Test-group makes it easy to execute multiple (possibly related) testcases. Zipline executes the testcases in the order in which they appear in the Test-group. You can even include a Repeat Count with the Testcase that you add to a Test-group. The Repeat Count specifies how many times the Testcase should be executed.
A Test Task consists of a Test-group and a date/time when the Test-group should be executed. To add a Test Task, you identify a Test-group and specify the date and time when the Test-group should be executed. Zipline keeps track of the Test Tasks and executes them in order as their specified date/time arrives.
Zipline Scheduler manages multiple Test Tasks. You can add, remove, modify Test Tasks using the Zipline Scheduler controls. You can pause, stop and resume the Zipline Scheduler. In case a particular Test Task runs into the scheduled date/time for other Test Tasks, the Zipline Scheduler delays execution for the other Test Tasks for as long as it takes for the current Test Task to complete.
Zipline Scheduler makes it very easy for you to run your nightly and long-term regression tests.
A Testcase consists of multiple commands. A Testcase Scope is simply a contiguous group of commands with at least two commands or more. Usually a Testcase Scope is defined in order to track the value of a pre-specified test parameter or repeatedly validate a particular test condition.
Let us consider a testcase that verifies basic network interface functionality. It enables a network interface, transmits and receives traffic through the interface and then disables the interface.
- Login to the network device
- Configure the network interface
- Enable the network interface
- Transmit traffic through the interface
- Receive traffic through the interface
- Perform other interface related tests
- Disable network interface
- Logout from the network device
Say we configure a Testcase Scope – ‘InterfaceEnabledScope’. We begin the scope with the command that enables the interface and end it with the command that disables the interface. With this scope, we can conduct some additional tests that would otherwise be difficult to configure. For starters, we can track the inbound and outbound packet rate on the interface being tested using a ‘Tracker’. Zipline automatically begins tracking the inbound and outbound packet rate as soon as the ‘InterfaceEnabledScope’ becomes active. We can even repeated ping a remote host connected via this interface for as long as the scope is active using a ‘Runner’. Zipline will know when to initiate the Runner and when to disable it.
Zipline makes the process of developing a testcase incredibly easy. Usually, you would either have to manually type the commands and the desired output using a scripting language. If you desired to develop a ‘Portable Testcase’, you would be required to add the complexity of using variables and tracking them throughout the testcase. With Zipline, all you need to develop a testcase is just manually execute the commands on the network device using the Zipline IDE. Using the ‘Capture’ process, Zipline automatically captures the relevant information and populates the testcase for you. To create a ‘Portable Testcase’ is only an incremental effort. While you type in your commands on the network interface session, popup the Properties list using the Ctrl-Space shortcut and select the value that you intend to use in the command. With this additional step, the testcases you capture are automatically ‘portable’ without you having to worry about using and tracking variables.
Zipline allows you to export your testcase into a file. The file contains all the necessary details about the testcase. This file can later be imported into a Zipline installation.
During testcase execution, there may be some information from the command output that you wish to preserve for future use during the testcase. Let us consider a simple example. Suppose a network device receives data traffic on one network interface and transmits that data traffic out of another network interface. To ensure that all data packets received by the network device were transmitted out, it would be necessary to store the number of packets received (in a Test Parameter – say ‘numPktsReceived’), the number of packets transmitted (in another Test Parameter – say ‘numPktsTransmitted’) and then compare the two. Test Parameters allow you to save dynamic infomation during a testcase execution. The saved information can either be locally validated by Zipline (e.g. numPktsReceived == numPktsTransmitted) or sent to an external script for more complex validation.
Zipline allows you to import testcases from a file into your Zipline installation. This comes in handy especially when you work in groups and want to execute or modify a testcase developed elsewhere.
A Test Category groups together testcases of similar nature. You can define a hierarchy of Test Categories to organize the testcases as you wish.
Publishing your work
Suppose you have been working on a Testcase that your team members would like to access. Once you have completed working on the Testcase, you can choose to Publish it. Once published, the Testcase becomes available for use by your team members. However, only you can make changes or delete the Testcase.
Output based Tests
An Output based Test validates certain condition in the output received from the network device (DUT), usually in response to a command typed by the user.
Router2# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 192.168.45.1 1 FULL/DR 00:00:36 10.0.0.1 Ethernet0
Consider the above command and output taken from a user session. In this particular example, to verify that the protocol is operational, we need to check the protocol state (FULL/DR). So this test qualifies as an output based test that uses a part of the command output to validate a test condition.
Parameter based Tests
A Parameter based test uses parameters saved during the testcase execution to validate a test condition. Parameter based Tests can perform basic math and string operations within Zipline. Alternatively, the saved parameters can be sent to an external script for validation. More about this in a later section that deals with Parameter based Test workflow.
Consider the following command and output grabbed from a user session.
Router# show interfaces gigabitethernet 0/0 GigabitEthernet0/0 is up, line protocol is up Hardware is 82543 (Livengood), address is 00d0.ffb6.4c00 (bia 00d0.ffb6.4c00) Internet address is 10.1.1.3/8 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex mode, link type is autonegotiation, media type is SX output flow-control is on, input flow-control is on ARP type:ARPA, ARP Timeout 04:00:00 Last input 00:00:04, output 00:00:03, output hang never Last clearing of "show interface" counters never Queueing strategy:fifo Output queue 0/40, 0 drops; input queue 0/75, 0 drops 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 2252 packets input, 135120 bytes, 0 no buffer Received 2252 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 2631 packets output, 268395 bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
Router# show interfaces gigabitethernet 0/1 GigabitEthernet0/1 is up, line protocol is up Hardware is 82543 (Livengood), address is 00d0.ffb6.4c01 (bia 00d0.ffb6.4c01) Internet address is 126.96.36.199/8 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex mode, link type is autonegotiation, media type is SX output flow-control is on, input flow-control is on ARP type:ARPA, ARP Timeout 04:00:00 Last input 00:00:03, output 00:00:03, output hang never Last clearing of "show interface" counters never Queueing strategy:fifo Output queue 0/40, 0 drops; input queue 0/75, 0 drops 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 2631 packets input, 268395 bytes, 0 no buffer Received 2631 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 2252 packets output, 135120 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
To validate the condition that the number of packets received on GigabitEthernet 0/0 is the same as the number of packets transmitted on GigabitEthernet 0/1, we would need to save the ‘packets input’ value from the first command (‘show interface GigabitEthernet 0/0’) and the ‘packets output’ value from the second command (‘show interface GigabitEthernet0/1’) and then configure a Parameter based Test to compare the two saved parameters.
Zipline supports conditional actions that can be taken based on the outcome of test conditions specified by you. For each possible outcome of a test (Pass/Fail), a particular action can be designated.
|Test to check if protocol is enabled||If Test passes, Jump to Label: ProtocolEnabled
If Test fails, continue execution
Various Conditional Actions are supported
- Pause Testcase execution
- Jump to a specified label
- Continue Testcase execution
Suppose a testcase includes commands to enable an interface, perform some tests and then disable the interface. Normally, these commands would be executed once in the sequence in which they appear in the testcase and then we would move on with the following commands in the testcase. What if you wanted to repeat these commands 10 times? Of course, you could configure these commands 10 times during the testcase development stage, but that would be cumbersome to say the least. Morevover, if you wanted to repeat these commands 1000 times, configuring these commands manually is hardly an option anymore.
Repeat Group allows you to specify that a contiguous set of commands should be repeated for a specified number of times before the testcase moves on to the following commands.
Often you need to execute a particular command or a set of commands repeatedly over the course of a testcase execution. Generally these are diagnostic commands, that look at some system parameters – such as packet queue size or CPU usage. Rather than polluting the testcase with repeated instances of these commands, you would prefer running them over a separate session. If you were developing scripts for your tests, you would probably write a small script where you execute these commands in a repeated manner at a particular frequency. If you are using Zipline you simply create a Runner.
A Runner executes one or more commands repeatedly at a user-specified frequency over the course of a Test Scope. A Runner can even perform a tests during each iteration if you configure it to do so.
Let us consider the OSPF example again. Following is grabbed from a user session.
Router2# show processes cpu CPU utilization for five seconds: 8%/4%; one minute: 6%; five minutes: 5% PID Runtime(uS) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 384 32789 11 0.00% 0.00% 0.00% 0 Load Meter 2 2752 1179 2334 0.73% 1.06% 0.29% 0 Exec 3 318592 5273 60419 0.00% 0.15% 0.17% 0 Check heaps 4 4 1 4000 0.00% 0.00% 0.00% 0 Pool Manager 5 6472 6568 985 0.00% 0.00% 0.00% 0 ARP Input 6 10892 9461 1151 0.00% 0.00% 0.00% 0 IP Input
You may want to have ensure that the ‘five minutes’ CPU usage never exceeds 20%. A Runner can repeatedly execute this command and test for the CPU usage. If you also wanted to plot the ‘five minutes’ CPU usage, you could add a Tracker for this command.
Like Trackers, Runners run out-of-band. Runners open a separate session to the network device and do not use the session that is used for commands included in the Testcase.
You may want to be able to track a particular value found in a command output over the course of a Test Scope or the entire Testcase. There are various scenarios where such capability can provide valuable insights. Let us consider OSPF as an example. Following is grabbed from a user session.
Router2# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 192.168.45.1 1 FULL/DR 00:00:36 10.0.0.1 Ethernet0
The ‘Dead Time’ usually counts down from (00:00:40) and is reset every time a hello packet is received from the Neighbor (192.168.45.1 in this case). Usually OSPF hello packets are sent every 10 seconds. So if there is not packet loss anywhere, the Dead Time should fluctuate between 00:00:30 and 00:00:40 under normal circumstances.
Rather than verify the ‘Dead Time’ every so often using an Output based Test, you can make use of a Tracker. Tracker is a specialized tool that repeats a command at a user specified frequency over the course of a Test Scope and captures the value of the desired parameter (a sample) from the command output. Additionally, a Tracker plots the sample values to allow you to quickly identify any unusual behavior. In a plot for the above example, if you notice the Dead Time dips below 00:00:20 too many times, clearly there is a problem that is leading to hello packets being lost.
Trackers run out-of-band. In other words, Trackers open a separate session to the network device. They do not use the session that is used for commands included in the Testcase.
Zipline automatically creates a testcase for you as you execute your tests by typing commands into the telnet or ssh session with the network device. Zipline captures all relevant information to populate the testcase with the commands you type. These commands that are automatically inserted into the testcase are termed ‘Captured Commands’.
Description allows you to add more information to a command in the testcase. This information can be very useful in helping you and your team manage the testcase.
|Enable Interface||Interface connecting the two routers|
|Delay: 30 seconds|
|Send Traffic||This traffic follows the routed path through the neighbouring router|
Certain commands require a specific amount of time after their execution for the desired effect to take place. This is particularly true for commands that enable certain protocols. Consider OSPF. After enabling OSPF on two adjacent network devices, we may have to wait for a few seconds before we can expect them to form an adjacency. If we continue testcase execution without waiting, it is likely that the next command(s) may either fail or not produce the desired effect. To insert delays to accomodate such conditions, Zipline offers the Delay Command. A Delay command can be inserted in a testcase either before or after any command. You can specify the number of seconds for which to wait before continuing with the testcase execution. When the testcase execution reaches the Delay Command, it waits for the specified time interval before proceeding with subsequent commands.
|Delay: 30 seconds|
A Pause command pauses the execution of the commands in a testcase. This command can be useful when you would like to check the system state or perform other actions before continuing with the testcase execution. You can resume testcase execution that has been paused using a Pause Command.
A Stop Command stops the execution of the testcase. It can be useful if you want to execute only a part of a testcase. Insert the Stop Command in the testcase wherever you would like to stop the execution of the commands.
Zipline captures commands directly from user sessions and inserts these captured commands into the testcase. Often, you may want to insert a command somewhere in the testcase after the testcase has already been created based on the user session. You do not need to start another session to insert the command. Zipline allows you to insert a User-specified Command anywhere in the testcase. You need to specify the command as well as the associated output when you insert a User-specified Command in a testcase.
A Label identifies a particular command in a testcase. A Label by itself does not alter the testcase execution in any way. However, a Label can be used with a Jump Command to modify the testcase execution sequence.
|Test to check if protocol is enabled||If Test passes, Jump to Label: ProtocolEnabled|
External Script Command
External Script command allows you to call an external script at any point during testcase execution. You can specify the script path, any prefix and/or suffix arguments that the script may need as well as any parameters that you have saved during testcase execution.
Normally, a testcase executes the commands in the order in which they are captured or configured by the user. However, you can choose to alter the sequence in which the commands are executed using a Jump Command. Identify the destination command by associating it with a Label. Insert a Jump Command wherever desired and associate that Jump Command with the Label that you just defined. When the testcase reaches the Jump Command, it will jump to the command identified by the associated Label and continue its execution.
Zipline also allows you to configure a conditional Jump Command. Whether or not the Jump Command is executed depends on the outcome (Pass or Fail) of a Test condition. Let us consider a simple example. You would like to configure a protocol – say OSPF on a network device only if it not already configured. You can create a Test condition that checks for OSPF being configured – either by checking the running configuration or using OSPF specific commands. If the Test determines that the protocol is not configured, you can continue the script execution along its normal course where it configures the protocol. If the Test determines that the protocol has already been configured, you can skip the following commands where we configure the protocol and jump to the subsequent commands.
|Test to check if protocol is enabled||If Test passes, Jump to Label: ProtocolEnabled|
A Skip Command allows you to skip the next command in the sequence of commands. You can use the Skip Command to temporarily prevent a command from executing. This can prove to be useful when you are altering a testcase or creating a new testcase.