[06/03/2015] - Minying

Overview

Tempest is a set of integration tests that runs against Openstack cluster(s). The main components are API tests, Unit tests, Scenario tests, Stress tests and Third Party API tests

API tests are running all the api calls of the environment both positively (expect success) and negatively (expect failure). This is the most basic and important tests of Tempest, because Scenario and Stress tests are calling the same functions as the API tests just in a different manner.

Unit tests are tests that run against the tempest code itself. This is for running Tempest against the Harvard production environment.

Scenario tests are “throughout path” tests of OpenStack usage. i.e. use cases

Stress tests are use to detect condition bugs which will not be easily found during fictional tests but will be encountered by users in large deployments. User who’s running this test must have access to the controller, because the tests look into controller’s log file.

Third Party API tests to test non OpenStack native APIs

How to run Tempest

Make sure you have access to the back door and you have proxychains set up correctly To run Tempest Without VPN access against the production environment. You have to do double ssh tunnel to our back door and run the tests with proxychains.

ssh -L 5000:140.247.152.201:22 username@140.247.152.200

open a new window and run

ssh username@localhost -p 5000

and then open another new window, go to tempest home directory and run

# for all the tests
proxychains ./run_tempest.sh 

# for all api tests in nova
proxychains ./run_tempest.sh tempest.api.nova

# for a specific tests such as test_rebuild_server test in nova api calls
proxychains ./run_tempest.sh -d tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server

Expected Results

Unfortunately, there are a number of tests in Tempest that are expected to fail due to the way our production environment is set up or bugs exists in OpenStack (that they unwilling to fix, or fixed in release later than icehouse).

API tests

Keystone [identity]

All pass

Nova [compute]

All pass except

  • test_allocate_floating_ip_from_nonexistent_pool
    • Because floating ip pool here is hard-coded in to satisfy other tests

Neutron [network]

All pass except

  • test_create_security_group_rule_with_invalid_ports
    • This test expects Neutron to retun 400 because it’s trying to create security group rule with invalid port_range_min=none (). But our environment let the request pass and return a 200 success.
  • test_delete_external_networks_with_floating_ip
    • This test deletes the external network with floating ip attach to it. The first response is NeutronError: NetworkInUse: Unable to complete operation on network [] There are one or more ports still in use on the network. However Neutron actually behaves correctly (can successfully delete the network) with the error message
    • After the first response(NetworkInUse), Neutron will schedule to delete the port/floating ip that is attached to the network and then delete the network again. This time we will be able to get 204 (successfully delete). But Tempest already got the first response and throw the error exit at this point.

Cinder [volume]

All pass except

  • test_volume_crud_with_volume_type_and_extra_specs
    • We don’t have crud

Sahara [data-processing]

All fail, Sahara currently not in the production environment

Swift [object-storage]

Almost all fail, because we are using Ceph, and common header like “content-length” is not available to tempest

Heat [orchestration]

  • setUpClass(tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON)
    • we are using Ceph

Scenario tests

see https://bugs.launchpad.net/tempest/+bug/1298472

Stress tests

I’m still trying to figure this out, their documentation seems to be outdated…..

Unit tests

All pass

Third Party API tests

Ignore