View RSS Feed

//no comment

Improving TSM's Scan Speeds

Rate this Entry
I think it's about time I do a blog post about TSM directly. The process of scanning the AH is fundamental to the addon. Even though the general idea for how to scan the auction house has never changed (and a lot of the code structure is identical to QA3 even), the code behind this process has been tweaked and improved countless times. First, let's have a (relatively) plain English look at how scanning the auction house works.

1. Figure out what we're scanning and organize it into auction house queries. These queries include search term (usually name of the item), level range, rarity, item type, etc (all the filters you can set on the "browse" tab of the AH).
2. Request the first query.
3. Wait for the AH page to be loaded fully (seller names can take some time). This could be anywhere from a couple tenths of a second, to a couple seconds.
4. Scan the page and store the data.
5. If there's more pages of results for this query, request the next page and go back to step 3.
6. If there's more queries, request next query and go back to step 3.
7. We're done!

Hopefully that all sounds pretty simple and straightforward at this point. I've left out most of the technical details, but everything described has been fully optimized. So, how do we make this faster?

Well, in v1.0 of TSM_Auctioning, the common search term feature was introduced. This allows you to change how the addon does step 1 of the process. For example, if you have a group of glyphs, you can tell the addon to use a "glyph of" search term rather than searching for each individual glyph. As an example, if you had 80 different glyphs in your bags, that would normally result in 80 separate queries (and we'll assume a total of 80 pages of auctions being scanned - 1 per glyph). If we assume there's only 40 pages of results for a "glyph of" search term, we just cut our scan time in half! Essentially, we're taking advantage of the fact that there's typically less than 50 auctions (1 page of results) for each individual glyph on the AH. One thing to note is that TSM is smart enough to not use the common search term if it'll result in more pages being scanned (ie if you only have 2 glyphs to scan). This is a great speedup, however it relies on the user setting it up properly for maximum benefit and the assumption that there's less than 50 auctions per individual item. Also, it takes an AH query to check if using common search terms is faster or not.

In v2.0 of TSM, the addon itself will dynamically and automatically determine common search terms on ALL auction house queries (ie for Shopping too instead of just Auctioning). This means that the user will no longer have to tell the addon which common search terms to use, and the addon will be smart enough to figure out the optimal search terms. How does the addon do this? Magic. Without getting too technical, it takes the list of items the module (Auctioning / Shopping) wants to search for and creates search terms out of the greatest common substrings of items which share at least the first word of their name. So, given a list of glyphs, it'll generate a "glyph of" search term since all item names in that list start with "glyph of". Once we've got these search terms generated, this gives us the full benefit from using common search terms and results in optimal speed-up. As with the v1.0 method, it does take time to determine whether these search terms will save us time or not, but this time is relatively insignificant when compared to the speed-up we're potentially getting. Reducing this amount of time and improving the intelligence of the code that determines the common search terms is something that will be pursed further in the future, specifically having the addon learn from past performance of search terms it generates.

Another speed-up that's been around for a while (I don't think it was v1.0, but some v1.x release shortly thereafter) in TSM_Shopping helps out with Dealfinding and Vendor searches where we only care about items below some price per item. Put simply, we'll sort the auctions by buyout, and then stop scanning pages once we're over the maximum price we care about (and at the maximum stack size). So, if we're looking for GIO below 2g per item, we won't scan the pages full of >100g stacks.

So, why is scanning still so slow? Well, Blizzard only gives us data so fast, so we're left with taking advantage of other tricks to try and reduce the number of pages we're actually scanning. However, the speed at which Blizzard gives us the data will always be the limiting factor and there's not much we can do about this. As you can see, most of the speed-up efforts are focused on step 1 of the process.

Updated March 31st, 2013 at 02:39 PM by Sapu94

World of Warcraft , Development


  1. theatermusic87's Avatar
    How ironic that i find this AFTER i posted anyways looks fantastic and extremely well summarized