[06/03/2015] - Minying
Overview¶
Tempest is a set of integration tests that runs against Openstack cluster(s). The main components are API tests, Unit tests, Scenario tests, Stress tests and Third Party API tests
API tests are running all the api calls of the environment both positively (expect success) and negatively (expect failure). This is the most basic and important tests of Tempest, because Scenario and Stress tests are calling the same functions as the API tests just in a different manner.
Unit tests are tests that run against the tempest code itself. This is for running Tempest against the Harvard production environment.
Scenario tests are “throughout path” tests of OpenStack usage. i.e. use cases
Stress tests are use to detect condition bugs which will not be easily found during fictional tests but will be encountered by users in large deployments. User who’s running this test must have access to the controller, because the tests look into controller’s log file.
Third Party API tests to test non OpenStack native APIs
How to run Tempest¶
Make sure you have access to the back door and you have proxychains set up correctly To run Tempest Without VPN access against the production environment. You have to do double ssh tunnel to our back door and run the tests with proxychains.
ssh -L 5000:140.247.152.201:22 username@140.247.152.200
open a new window and run
ssh username@localhost -p 5000
and then open another new window, go to tempest home directory and run
# for all the tests
proxychains ./run_tempest.sh
# for all api tests in nova
proxychains ./run_tempest.sh tempest.api.nova
# for a specific tests such as test_rebuild_server test in nova api calls
proxychains ./run_tempest.sh -d tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server
Expected Results¶
Unfortunately, there are a number of tests in Tempest that are expected to fail due to the way our production environment is set up or bugs exists in OpenStack (that they unwilling to fix, or fixed in release later than icehouse).
API tests¶
Keystone [identity]
All pass
Nova [compute]
All pass except
test_allocate_floating_ip_from_nonexistent_pool
- Because floating ip pool here is hard-coded in to satisfy other tests
Neutron [network]
All pass except
test_create_security_group_rule_with_invalid_ports
- This test expects Neutron to retun 400 because it’s trying to create security group rule with invalid port_range_min=none (). But our environment let the request pass and return a 200 success.
test_delete_external_networks_with_floating_ip
- This test deletes the external network with floating ip attach to it. The first response is NeutronError: NetworkInUse: Unable to complete operation on network [] There are one or more ports still in use on the network. However Neutron actually behaves correctly (can successfully delete the network) with the error message
- After the first response(NetworkInUse), Neutron will schedule to delete the port/floating ip that is attached to the network and then delete the network again. This time we will be able to get 204 (successfully delete). But Tempest already got the first response and throw the error exit at this point.
Cinder [volume]
All pass except
test_volume_crud_with_volume_type_and_extra_specs
- We don’t have crud
Sahara [data-processing]
All fail, Sahara currently not in the production environment
Swift [object-storage]
Almost all fail, because we are using Ceph, and common header like “content-length” is not available to tempest
Heat [orchestration]
setUpClass(tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON)
- we are using Ceph
Scenario tests¶
see https://bugs.launchpad.net/tempest/+bug/1298472
Stress tests¶
I’m still trying to figure this out, their documentation seems to be outdated…..
Unit tests¶
All pass
Third Party API tests¶
Ignore