Weekend Project: Integration Testing for TSM
by, March 21st, 2016 at 03:02 AM (5584 Views)
It's been a while since I've posted here, but this weekend I've been working on a project which I've wanted to tackle for a while. Mainly, I just wanted to figure out how doable it would be, and use it as an excuse to learn more about Python and automation framework design. The project was to "bot" TSM, in the name of automated integration testing. Fair warning: this is going to be a long post.
What does this mean?
Well, as you may know, the TradeSkillMaster addon is massive, with tens of thousands of lines of code. As a general rule, we do some level of testing on a new feature or change before releasing it, but bugs slip through the cracks, and there's no way to be 100% confident that a change isn't going to cause a bug in some unrelated area of the addon. Integration testing is generally a great way to make sure the basic functions are working as expected, and that there aren't any obvious breakages before a new release goes out to the world. So, my project this weekend was to write an integration testing harness and framework for TSM, with the end-goal of being able to run a comprehensive suite of tests before any new release in a completely automated fashion. I also wanted to learn as much as possible from the project, so didn't spend much time looking for existing WoW-automation solutions, but instead went for a homegrown approach.
The first step was to setup a test environment. I created a separate WoW account and character, made a copy of my WoW folder, and tweaked all the game settings to make sure things ran as reliably as possible (lowest graphics settings, block trades and guild invites, etc). I also decided on Python (3.4.3) as my language of choice (running on Windows 10) since I'm fairly comfortable with it and it has an abundance of libraries and Google-able information.
Now that I was all setup, I started out with automating the login process. The first challenge I ran into was programmatically determining what screen of the login process I was on at any given time (email/password entry, character selection, loading, etc). After a bit of Google'ing, I found this blog post which describes a simple image hashing approach which allows the use of the Hamming distance algorithm to determine the similarity of images. I implemented the process as described, but ran into issues with it not giving enough resolution and often providing inaccurate results for which image in my list of hashes was closest to the real one. So, I made some tweaks including using color instead of grayscale and using a larger image (and resulting hash). I ended up with the following, which seems to work well enough for me:
I am using Python's Pillow library for grabbing the screen image and processing it. Each hash I'm comparing against has a unique threshold (the login screen has a lot of animation in the background which warrants a larger threshold for example). I also had to do a bit of cropping to get just the area of the WoW window I wanted (the actual game screen). After some more searching, I found some examples of how to use the win32 APIs to send mouse and keyboard events to the game, which I successfully implemented, and I used Python's subprocess module to be able to launch WoW from my script. With all of these pieces in place, I could now automatically go from WoW not running at all to my character logged in and sitting in Orgrimmar (where I left him).Code:img = img.resize((32, 32), Image.ANTIALIAS) data = img.getdata() avg = tuple(sum(x[i] for x in data) / len(data) for i in range(0, 3)) hash_str = '' for i in range(0, len(data), 2): value = 0 for j in range(3): if data[i][j] > avg[j]: value += 1 << j if data[i+1][j] > avg[j]: value += 1 << j hash_str += '%01x' % value return hash_str
I'm going a bit out of order from how I actually did things, but the next thing I'd like to touch on is how I got consistent screen images once I was logged into the game, even though the game world is constantly changing. I wanted to be able to sit by the auction house to test the AH features of TSM, but didn't want random other players messing up my image hashes. After a lot of experimentation, I finally stumbled across a solution. In my addon code, I created a frame at the lowest possible frame level of "BACKGROUND" and made it completely black and opaque. I also created a new "Empty" chat tab which had all chat sources disabled. As you can see, this fixes the problem of changes in the game world throwing off my image hashes. However, there are still a few things that aren't constant (like the minimap and clock), and flash (like the player portrait and chat frame tab highlight). So, I simply hid them, leaving only the chat box (which I needed to keep). Of course, the TSM windows show above this black frame, so there's no problem there. This gives me an extremely consistent environment for comparing images, to the point where a slight change in text will cause image hashes to no longer match (which in this case is a good thing).
Now, my Python script could send commands to the addon via slash commands, but there was no way for the addon to send data back to the Python script. This part is what I spent a big chunk of my time on, and which I personally found the most enjoyable (I like networking - don't hate!). The TSM Desktop Application gets data out of the addon through the saved variables, but these are only updated upon logging out (or reloading); way too slow for this. A few years ago, I wrote a small addon and companion application (link just for reference - it's long been abandoned) for opening URLs in your browser from the game. The way the addon sent data to the companion application was to change the color of a few specific pixels. Data could then be encoded in these pixels and read by the companion app. I used the same approach for this project, but needed to send a lot more data with perfect reliability. Each pixel is an individual button, with the data being encoded in the background color. Unfortunately, it's really hard to draw single-pixel things from addons (every other black pixel is actually 2 pixels wide - everything is intentionally 3 pixels tall). After trying a lot of different things, the scheme that ended up working was to use the just the red and green channels to transmit data (1 byte each = 2 bytes per pixel) and use the blue channel to indicate the "type" of the pixel, with the type of consecutive data pixels alternating in a checkerboard pattern. This way, no matter how big the pixels turned out to be in my image in Python, I could simply look at the blue channel for the edge between two pixels. I also had a single clock pixel, which had a completely separate blue channel "type". This blown-up image shows an example of what these data pixels look like. I ended up with 10 rows of 1000 pixels each, for a data rate of 20kB per frame (I wanted everything to fit into a single message and TSM group exports can be tens of thousands of characters).
So, I could now reliably log into WoW, send slash commands to control the addon, and get data back from the addon in real-time. Now I was ready to actually do some testing! I went through a half dozen iterations on how I wanted the test cases to be structured, how they should interact with the test harness, and generally trying to increase the level of abstraction. What I finally ended up with as far as test structure is concerned is something very similar to Python's own unittest framework. The simplest test case is just 4 lines of code:
The idea that tests should be easy to write was critically important. So, I tried to make them as straight-forward and simple as possible and do as much of the complex stuff within the test harness (under the hood) as possible. As an example, here's an actual test I wrote which verifies that group importing, group exporting, subgroups, and group filtering is all working.Code:from lib import TestCase class MyTest(TestCase): def test_basic_math(self): self.assert_equal(1 + 1, 2)
I can now easily and quickly write very high-level integration tests for TSM. My future plans for this project are to continue to add functionality to the test harness to enable more tests, speed things up by removing blind delays and instead using image-based indicators, and move the whole thing to a separate system so it can be run remotely (and automatically).
NECESSARY DISCLAIMER: I know this is technically against the ToS. I've purposely left out the details of how I'm sending inputs to the game and how I'm modifying TSM to enable this (there are deliberate road-blocks to automation within TSM's code). I'm also not releasing the source code for this (although if you're a fellow developer and want to discuss it, I'd be happy to do so either here or via IRC).