6. Zipline IDE

6. Zipline IDE

Logging In

Double-click the Zipline icon on the Desktop or select Zipline from the list of installed applications.

Specify your username and password. If you are running a trial version of Zipline, the default credentials are:

  • Username: user
  • Password: password

Specify the license key that has been provided to you.

Zipline can be used in two different modes

  • Internal Database Mode
  • External Database Mode

Make your selection using the drop-down menu depending on the installation type. Check with the system adminstrator if you are unsure. If Zipline has been installed in the ‘External Database Mode’, you will need to provide the external database server details.

Zipline offers various interfaces through which you can execute your testcases. Apart from the default graphical user interface, Zipline supports a command-line based telnet interface, web-based REST application programming interface as well as a Python Interpreter interface. If you would like to enable these interfaces, select the checkbox labeled ‘Start Zipline Server’.

Please note that you need to make these choices only once – the first time you start Zipline. The values are cached for future uses of Zipline.

User Interface Overview

Zipline graphical user interface is divided into seven different sections.

Settings

The Settings area allows you to carry out the following operations.

  • Enable/Disable logging functionality
  • Manage DUT versions that help you track & categorize your test activity
  • Manage System Errors that you want to be on the lookout for as you validate your DUTs

Testcase

You can manage your testcases in the Testcase area. It offers the following operations

  • Import/Export Testcases
  • Delete, Publish & Categorize Testcases

Capture

You will be spending a lot of time in the Capture area if your job is to develop testcases. The Capture area offers the following operations:

  • Create or Edit Testcases

Testbed

The Testbed area is where you manage your testbeds and associated property-set files. Please refer to the Zipline Nomenclature section to understand how testbeds and property-sets are connected. Here are the operations you can perform in the Testbed area:

  • Create, Edit, Delete testbeds
  • Create, Edit, Delete testbed elements or DUTs
  • Create, Edit, Delete property-sets
  • Create, Edit, Delete properties
  • Associate a testbed with a property-set

Replay

The Replay area is probably where you will spend most of your time as you execute testcases you developed on the DUTs. Here are the operations you can perform in the Replay area:

  • Replay Testcases & Test-groups
  • View logs for replay sessions initiated from the Telnet server interface, Web inteface or the Python Interpreter interface

Report

The Report area is where you access information about past testcase or test-group executions. You can search through the recorded test logs based on the testcase/test-group/testbed name, DUT version, test result or the dates when the tests were executed. Here are the operations you can perform in the Replay area:

  • Search for Replay reports based on various search criteria
  • Select a particular replay session and view its details

Test Scheduler

The Test Scheduler area allows you to schedule and execute tests at a particular time in future. A common usecase for this functionality is if you would like to run some tests on a testbed over the weekend or at a particular time when you get access to the testbed. Here are the operations you can perform in the Test Scheduler area:

  • Create, Edit, Delete a test-group
  • Schedule, Edit, Delete a Replay Task by specifying the test resources and the scheduled time
  • Manage (Pause, Resume) the test scheduler

Create Property Set

Navigate to the Properties section in the Testbed area to manage property-sets.

Click on Create Property Set to create a new property-set. Specify the desired property-set name.

The configured property-set shows up in the list. We can add properties to this property-set.

Add Properties

Select the desired property-set to which you want to add a new property. Click on Create Property button. Sepcify the property name and the value.

The property should show up in the list of properties for the selected property-set.

Create Testbed

Click on Create Testbed in the Testbed area to create a new testbed. Specify a name for the testbed.

The testbed should get created and be available in the Testbed area.

For this example, we would like to associate the testbed ‘simpleTestbed’ with a property-set ‘simplePropertySet’ that we just created.

Click the button Link Property Set to complete the association. The testbed is now associated with the property-set.

Add Elements

Testbed is composed of network devices or elements referred to as DUTs. Select the testbed to which you wish to add an element. Click on Create Element in the Testbed area. You need to specify the details about the element that you want to add to the selected testbed. You can either type in the actual IP address of the element or use a property from the property-set associated with the testbed. By using a property name as the element IP address, your testbed configuration becomes logical. It is not restricted to a particular set of DUTs. In order to point your testbed configuration to a different physical testbed, simply associated the corresponding property-set file with the testbed configuration.

Specify the necessary details in the Element dialog. In this case, the element does not have a username configured. So we leave the corresponding text fields empty.

Navigate to the Advanced tab in the Element dialog. Specify the fields that are relevant to the element. In most cases, you will need to specify a comma-separated list of prompts that are supported by the element. You do not need to specify the entire prompt string patterns in the list. Just specify enough characters of the prompt that can unambiguously identify it.

Verify that you see the element in the list of elements for the selected testbed.

Create Testcase

Navigate to the Capture area to begin capturing a testcase.

Click on Select Test Resources. You can either select an existing testcase to modify it or create a new testcase using this wizard. Select a testbed, a test category and specify a name for the testcase that you want to create.

Because we are creating a new testcase, we do not see any data at this point. To begin populating the testcase, we need to start the capture process. To do that, click on Start Capture.

This view lists all available elements in the testbed that we have selected. Click on the element that you would like to conduct your tests on and click on Start Session to initiate a session to that element. Based on the element configuration, either a telnet or a ssh session will be initiated with the element selected.

Zipline embeds the telnet/ssh session within its graphical user interface. Type in your commands as normally would and conduct your tests on the DUT.

Once you are done conducting the necessary tests on the DUT, click on Stop Capture to terminate your session and process the commands that you captured. You can review the commands that you typed in a table.

You may want to add some more commands to an existing testcase at a later point. To do that, open the testcase in the Capture area and click on Start Capture as we discussed earlier. Zipline offers a choice to either append the new commands at the end of the existing set of commands or insert them new somewhere within the existing set.

A portable testcase is one that can be executed on different similar testbeds. The testcase does not include any values that are specific to a particular testbed (such as an IP Address, or an interface name). Instead the testcase uses property values from the property-set associated with the testbed that is being used.

To make use of properties instead of the actual values of IP Addresses, interface names and other identifiers, use the Ctrl-Space shortcut. An example of the sequence of keystrokes that you type in is: ‘ip address ‘. You will be presented with a popup that lists the available properties for the selected testbed. Select the property that is applicable to the current command and hit enter. You can navigate through the list of properties using the up-arrow or the down-arrow keys on your keyboard. To select a particular property, hit Enter on your keyboard or double-click the left-button on the mouse.

Even though you selected a property name (e.g. if1_addr) while typing in the command, Zipline will show you the actual value (e.g. 192.168.1.6) being used with the command in the displayed tables as well as when you replay the testcase on the selected testbed. If you reopen this testcase with a different testbed, Zipline will replace the property name with the appropriate property value for the new testbed. This is where the testcase portability becomes evident.

You can edit previously captured commands using the Edit Command shortcut available in the toolbar or in the right-click menu options. Let us edit the command that we just inserted into our testcase. Since we used a property name instead of the actual IP address, we will see the property name as a part of the command in the Edit Command wizard.

If you would like to specify a property name instead of a literal identifier in any command after it has already been captured, Edit Command wizard allows you to make the necessary change. Replace the literal identifier with the property name wrapped in curly braces.

A testcase without any tests would not be very useful. There are two types of tests that Zipline supports.

  • Command output based test
  • Parameter based test

Let us see how we can add an command output based test to a testcase. Double click on the command to view the corresponding output in the Command Output pane. Select the particular line of interest from the command output. Click Add Test to configure a test for the selected line.

The selected command output line is segmented into various sections that can individually either be matched or ignored during the test validation. A test can be made to be portable across testbeds. We need to replace any literal identifier (e.g. IP address, interface name) with the property name just as we did in the case of commands. Click on the output segment that you would like to replace with a property name and use the Ctrl-Space shortcut. From the list of properties that pops up, select the one applicable to the current output segment.

Zipline allows you to configure Conditional Actions based on the outcome of the test during testcase replay. For each possible test outcome (Pass, Fail, Error), you can select the action that should be performed. A common usecase is to pause the testcase when a particular test fails. Alternatively, you can jump to a specific label/command in the testcase or simply continue with the testcase execution at the next command (default action).

The configured command output based test should show up along with its parent commmand in the list of commands.

You may want to add a new command to an existing testcase. There are two ways you can do that.

You can initiate a new Capture session and capture one or more commands the same way as you would with a new testcase. You have a choice to either insert the captured command somewhere within the testcase or append them to the existing set of commands in the testcase.

You can insert one command at a time to a testcase by using the Insert Command wizard from the right-click menu or from the toolbar. When using the Insert Command wizard, you will need to specify the command, the expected command output and the testbed element on which the command should be executed during testcase replay.

For command portability, you can make use of a property name instead of a literal identifier in the Insert Command wizard.

The new command should show up in the list of commands for the testcase.

Replay Settings

To begin testcase replay, navigate to the Replay area. Zipline allows you to configure settings related to testcase replay. Replay Settings wizard can be accessed by clicking Replay Settings in the Replay area. You can choose to execute the testcase commands at the same pace at which they were typed in during the capture process or execute the testcase commands as fast as possible while interacting with the DUT.

You can decide to pause or continue the testcase execution when a failure is encountered. Failure can occur due to a test configured in the testcase or when a system error is encountered at anytime during the testcase.

Replay Testcase

Select the test resources (testcase or test-group and the testbed) and click on Start Replay to begin executing the commands on the testbed.

The currently executing command is highlighted with gray background. After a test is executed, it is highlighted either with green or red background depending on whether the test execution passed or failed. The live replay session is displayed using two tabs in the lower portion of the Replay area. The Foreground tab displays the commands from the testcase being replayed on the DUTs. The Background tab runs over a separate session and replays the Trackers & Runners configured with the testcase.

Each tab in the Replay area displays information relevant to the current testcase replay session. The Output Tests tab displays the results of the output based tests configured for the testcase.

Zipline also displays details about the test after it has been executed. Click on the View Test Details button to view the details about the test. The relevant output lines are highlighted in green if they match the test conditions configured.

The test has been reconfigured such that it fails during testcase execution. Accordingly, the test is now highlighted in red.

The Test Details shows that the output line does not match the test condition anymore. The last segment for the output line ‘inet 192.168.1.6 netmask 255.255.255.0 ..’ did not match the test condition and hence is no longer marked in green.

View Test Report

Navigate to the Report area to view details about testcase executions based on your specified search criteria. You can search for testcase replay details based on the following items:

  • Testcase or test-group name
  • Testbed name
  • Test Result
  • DUT Version on which the testcase was executed
  • Date range during which the testcase was executed

Click on Search to search for testcase executions that satisfy the criteria that you specify. The matching testcase executions are listed in a table. You can view details about a particular testcase execution by selecting it from the list and clicking on View Details.

The details about the selected testcase execution are displayed using the same layout and format as that of a live testcase replay session executed from the Replay area. The output for each command executed is displayed in the Command Output tab.

Test results and test details are displayed just as we discussed in the Replay section. The test output is highlighted to point out the output segments that matched the test conditions that those that did not match.

You can even view the replay session logs from the past testcase executions in the Foreground and the Background tabs.

Import/Export Testcases

Navigate to the Testcase area to manage testcases. In order to export a testcase, select a testcase from the list and click on Export.

Specify the directory path and a file-name for the exported testcase.

You will be notified when Zipline is done exporting the testcase to the specified location.

To import a testcase into Zipline, click on Import and select the .ntt file that contains the testcase that you want to import.

Zipline imports the testcase, the testbed and the propert-set details from the .ntt file. It displays a notification after the import process is complete.

In case a testcase, testbed or property-set with the same name exists, Zipline appends a suffix ‘_Imported_’ to the imported testcase, testbed or property-set. The in the suffix is used if necessary to create a unique name.

Repeat Groups

To repeat a set of commands for a specified number of times, Zipline offers a tool called Repeat-Groups. Navigate to the Capture area and open the testcase that you want to add a Repeat-Group to. Go to the Repeat-Group tab.

Click on the Create Repeat-Group button to trigger the Repeat-Group wizard. Specify the range of commands that you want to include in the Repeat-Group configuration. Also specify the number of times the commands in the range selected should be repeated.

The newly configured Repeat-Group will show up in the list. You can view, edit or delete the Repeat-Group using the toolbar options or the right-click menu.

Parameter Tests

Zipline allows two type of tests to be configured.

  • Command output based test
  • Parameter based test

A Parameter based test allows you to save specific bits of information (parameters) from the command output as you go through the testcase replay. At a later point, when you have the necessary data gathered from previous command executions, you can ask Zipline to put these bits together and test for certain condition. A common usecase would be to store inbound packet count on one interface (call it ‘inputCount’), the outbound packet count on another interface (call it ‘outputCount’) and then compare the two to be equal (inputCount == outputCount).

To begin configuring a parameter test, identify the parameters that you would like to save from the command ouput. As an example, you may want to save the maximum ping response time from a ping command output.

Zipline performs default segmentation of the output line that you selected. In this case, the first attempt at segmentation is not granular enough for us to pick the maximum ping response time from the command output. So we add forward-slash – ‘/’ to the list of delimiters that Zipline uses and click Apply to take another stab at segmenting the output line to grab the parameter of our interest.

Now we can individually identify the segment that contains the information we need – the maximum ping response time. In this case, it happens to be 70.046. Using the drop down menu under each segment, we instruct Zipline to save the maximum ping response time as a parameter. We name this parameter maxPingResponseTime. Also, in order to Zipline to uniquely identify the output line containing the desired segment, we ‘match’ some of the other segments.

The parameter shows up in the list of parameters for this testcase.

To use this parameter, we select a command and use the Add Parameter Test option from the right-click menu. For obvious reasons, the command where we intend to add the parameter test should be the one where the desired parameter has been saved or any other command after that one.

Zipline allows you to perform the following operations with the saved parameters.

  • Mathematical Operation
  • String Operation
  • External Script Processing

Basic mathematical and string comparison operators are available in the drop down menu in the Parameter Test wizard.

We configure a very simple parameter test that checks to see if the maximum ping response time is less than 100. The test is marked as having passed if this condition is true. If we had saved a couple of string parameters and wanted to compare them for string equality, we could do that as well using the Parameter Test wizard.

Just as with output based tests, we can configure test result based Conditional Actions. We can either continue test execution, pause it or jump to another label/command.

The newly configured parameter test shows up along side the command in the command list.

Test Scope

A test scope is a contiguous range of commands. A common usecase is to configure a scope spanning a range of commands where certain conditions holds true. A scope is considered to be active when Zipline is execcuting a command that falls on or within the range identified by the scope. Configuring a scope is fairly straightforward. Navigate to the Capture area and select the testcase for which you want to configure a test scope. Click on Create Scope to trigger the wizard.

Pick a name for the scope. Identify the commands where the scope starts and ends.

The scope shows up in the list of scopes defined for this testcase. You can configure multiple overlapping scopes.

Add Trackers

Trackers and Runners are tools that depend on a scope configuration.

A Tracker tracks certain numerical parameters – such as the OSPF Dead Interval, an interface’s input packet rate, etc. Because such parameters may be meaningful only when certain conditions are met (such as OSPF being enabled), we use scope to decide when and for how long should a Tracker attempt to track the parameter value. For example, tracking OSPF Dead Interval is meaningful only as long as OSPF is configured on the testbed. So a scope that spans only those commands during which OSPF is configured on the testbed should first be defined.

Then, navigate to the Tracker tab in the Capture area to create a new tracker. Click on Create Tracker to start configuring a tracker.

Pick a name for the Tracker. Identify the scope during which the tracker should be activated. Additionally, pick a sampling interval. The sampling interval (in seconds) decides how often the specified parameter would be sampled as long as the associated scope remains active.

We configured the tracker. But we are only half way through. We need to tell Zipline what parameter to track.

You can either manually specify a new command and the corresponding output (‘Add new command’ toolbar button) or select a command from the command list table and add it to the tracker (‘Add selected command’ toolbar option).

The selected command now shows up alongside the tracker configuration. You can add only one command to a tracker configuration.

Double click the command to view the command output. Select the output line that contains the parameter value that you want to sample during the testcase execution. Click on Configure Tracker button in the toolbar.

Tracker configuration is somewhat similar to test configuration. In a tracker configuration you are required to identify the parameter that you want to sample during testcase execution. You can specify an appropriate delimiter to segment the output line in such a way that the parameter that you want to sample can be separated from the other segments in the line. Once you separate the desired parameter, instruct Zipline to sample it during testcase execution. You do that by selecting ‘Track’ option from the corresponding drop-down menu..

Add Runners

A Runner is another tool that is associated with a test scope. A Runner is used to instruct Zipline to run one or more commands at a desired frequency in a separate telnet/ssh session to the DUT. Common usecases could be flapping BGP routes every few minutes to test for memory leaks, or running ping tests to check for round-trip delays. To configure a runner, navigate to the Capture area and select the Runner tab. Click on the Create Runner button to trigger the wizard.

Specify a name for the runner. Select the scope that you want to associate with this runner. The runner is triggered only when the associated scope is active. Select a frequency to specify how often the runner commands should be executed.

Pick a name for the runner, associate it with a scope and specify a frequency with which Zipline should scheduler the runner commands. The newly configured runner is added to the Runner tab.

Just as with trackers, you can choose to add a new command to a runner (‘Add new command’ toolbar option) or select an existing command from the command list (‘Add selected command’ toolbar option) and add it to the runner.

You should be able to view the command that you added to the runner configuration. Unlike a tracker, you can add multiple commands to a runner configuration.

Optionally you can add a test condition to a command in the runner configuration. This test is executed every time the associated command is executed by the runner. To add a test to a runner command, double click the command to view the command output. Select the output line where you wish to add a test and click on the Add Test toolbar option.

Adding a test to a runner command is very similar to adding a test to a testcase command. Identify that output line segments that should be matched or ignored by the test. Optionally, configure test status based Conditional Actions for this particular test.

The configured test appears alongside the runner command in the list.

Replay Testcase (again)

We added a bunch of features to our testcase over the last few sections. Navigate to the Replay area to execute the testcase. Select the test resources and click Start Replay to continue. You can view the test execution logs in the Foreground and Background tabs. Once the test execution completes, access the tabs in the lower half of the Replay area to view details about the test execution. The Trackers tab shows the number of times the selected parameter value was successfully sampled during the period that the associated scope was active. The samples are stored and available to view in the form of a chart.

Trackers

Select a tracker from the list and click on View Chart to view the line chart formed using the samples collected by the tracker.

Runners

The Runners tab shows the number of times the test that we had configured was executed during the period when the associated scope was active. The number of times the test ran, passed, failed or ended up with an error can be determined from the table.

Saved Parameters

The Saved Parameters tab shows the value of the parameter as captured by Zipline during the test execution process.

Parameter Tests

The Parameter Tests tab shows whether the configured parameter test passed or failed. In addition, the table also lists the condition specified by you when you configured the parameter test as well as the actual mathematical or string operation that was evaluated.

Test Scope

The Test Scope tab lists the test scopes configured by you for the testcase being executed.

Repeat Groups

The Repeat Groups tab lists the repeat groups configured for the testcase. The table includes details about the commands included in the repeat-group and the number of times the repeat-group was executed.

Command Output

Double click on any command to view the output captured for that command during the testcase execution.

Session Logs

The Foreground and Background tabs show the session logs that were captured during the course of testcase replay.

View Test Report (again)

The Report area also includes details about all aspects of testcase execution.

Specify the testcase and the testbed to view all previous testcase executions.

Select the testcase execution that you are interested in and click View Details.

Navigate through the tabs to view all aspects of the testcase execution – just as we did in the Replay area.

Trackers

Runners

Saved Parameters

Parameter Tests

Test Scopes

Repeat Groups

Command Output

Session Logs