View RSS Feed

//no comment

AuctionDB: Under the Hood - Part 1

Rating: 2 votes, 3.00 average.
TSM_AuctionDB is the third most popular of the TSM modules (Crafting is #1 and Auctioning is #2). The vast majority of TSM users rely on it as their primary source of pricing data; whether they are viewing its data in the tooltips of items, or using as part of a price source (or custom price) for one of the other modules. As with ItemTracker and WoWuction, the AuctionDB module does not provide any significant user interface, and for those who use the TSM Application, runs completely in the background. Not much has changed from a user's perspective over the past couple of years. However, a lot has been going on under the hood, especially recently, to make it faster, smaller, and more reliable than ever.

First, a bit of context. My general design philosophy with TSM (and other projects) is that speed is extremely important. This is especially true when looking at the time it takes to load / save TSM (and its modules). Not only does TSM need to process a TON of data (think about all the calculations Auctioning is doing just to post a single item - HINT: it's a lot, especially when you factor in complex custom prices), but the goal of any software is to hide the computational complexity and make it so the user never notices it. Similarly, nobody likes to wait forever for their addons to load when they log in, or saved when they log out.

A few weeks ago I was looking through some code, and decided to measure how much of the ~10 seconds it takes me to "/reload" (I have an SSD and run with very few other addons) was coming from TSM_AuctionDB. I'd done this measurement before, but not in a while, and also wanted to see what could be done about the freezing people were experiencing from AuctionDB importing a bunch of data from the app all at once (10+ scans at a time) upon loaded. I'll come back to importing from the app in another part. What I realized when I measured the typical load/save time for AuctionDB was that it was accounting for about 1.5 out of my 10 seconds of reload time. Not particularly good, especially when you scale that up to people on slower, non-SSD setups. So why was it taking so long?

Well, AuctionDB doesn't use the same format for storing its data during normal use (in memory) as it does in the saved variables. If it did, this would make the saved variables file gigantic (~5MB per server). It uses a custom-designed compression scheme to convert the database into a very long string, which takes up about a tenth the amount of space. This saves on loading / saving time of the saved variables file itself (which importantly isn't counted in the 1.5 seconds I measured). However, this compression does require a lot of processing, and this is what AuctionDB was spending 1.5 seconds doing: compressing and then decompressing the entire database for a given factionrealm.

So, how can this 1.5 seconds be reduced? Well, in the next part, I'll walk through how the it was reduced to under 1/50th of a second; that's a 75x speedup! Check back tomorrow for part 2.

Part 2 can be found here:

Updated November 19th, 2013 at 11:17 PM by Sapu94

World of Warcraft , Development


  1. Kathroman's Avatar