Cluster Tests, Unit Tests 1. Purpose of this talk - 2 min, 2 min total 1. A "Testing 102": instead of introducing what tests are and why you should write tests, this will get you comfortable with the different layers of test protection. 2. Target audience: someone who has worked in a shared code base before and contributed to unit/end-to-end tests, but hasn’t had the responsibility of setting them up. 2. Introduce our example service: Pwitter, a twitter clone in Python - 5 min, 7 min total 3. API is consumed by website, mobile app, etc 4. HTTP POST to ‘/username/tweets’ endpoint to send a tweet (auth required) 5. HTTP GET to ‘/username/tweets’ endpoint to view that person’s tweets 6. HTTP DELETE to ‘/username/tweets’ endpoint to delete the tweet (auth required) 7. What do we need to test? 1. Outward behavior: Make sure the service works, component pieces are talking to each other. These are cluster tests. 2. Internals: Make sure every line of code is doing what it’s intended to do. These are unit tests. 3. Coverage Philosophy Comparison 8. Cluster Tests - 7 min, 14 min total 3. Let’s say our service is made up of 2 components: controller and DB. What are the failure scenarios? 1. bug in controller 2. bug in DB (assume already tested) 3. failure of controller machine 4. failure of DB machine 5. connection error between controller and DB 4. We want to make sure posting a tweet works, "thread the needle". 5. Approach to writing: Simulate the action from user’s point of view. 6. Writer visibility (to the service internals): 0%, black box 7. Coverage: 6. One test that threads the needle once is almost sufficient. Covering all use-cases is preferred. 7. For example for Pwitter, you’d want to test GET, POST, DELETE all do expected behavior. If you only were able to write one for GET, you’d know that the controller and DB are still talking to each other and that’s better than nothing, but you should still write tests for POST and DELETE. 8. However, don’t let the perfect be the enemy of the good. In the case of a more complex service, don’t have to hunt down every test case out or let missing ones stop you from releasing basic E2E tests. 8. Rule of thumb: 9. Should be able to change service internal architecture without breaking tests. (Service architecture change example). 10. If one component of your service fails or loses connection, tests should fail. 9. (Code snippet example) 9. Unit Tests - 5 min, 19 min total 10. Looking at the internals of our service, we want to make sure every line of code is bug-free. 11. Coverage: Every possible code path. When there are "if" statements, test true/false cases. When there are “for”/”while” loop, go through the loop. Test caught exceptions. Test data input possible range beginning, end, edge cases. 12. Writer visibility: 100%. 13. Rule of thumb: 11. Tests are so thorough and explicit that if you make functional changes, you know there are tests you have to find for it and fix. 12. Tests are so thorough that if you introduce even a typo into the code, it would be caught. 14. (Code snippet example) 4. Practical Considerations - 5 min, 24 min total 10. Where to write tests 15. Cluster tests are simulating being a client from the outside. Tests should be on a completely separate machine from your code base, call out over network. 16. Unit tests need to be easily accessible to developers as they write the software, keep them in the same code base. Usually there’s a "test" folder. 11. When to run them 17. Cluster tests are checking for the health of the system; if they fail, it might be because a box broke or a connection failed-- something exogenous to the software code. Run them continuously. 18. Unit tests make sure you’re not introducing bugs into your software. Run them before you merge in new code. (Also individual developers run them during development). 12. Data housekeeping 19. Since cluster tests are affecting the state of the world, will need to clean up after yourself, ie delete the tweet after you create it so you can run your test again. 13. (Code example, at the end of test for making a tweet, clean up with an HTTP DELETE) 20. Unit tests are run on their own, often set up and tear down their own version of the "world" and use fixtures. 5. Review and conclusion - 2 min, 26 min total