View RSS Feed

//no comment

AuctionDB: Under the Hood - Part 2

Rate this Entry
This is part 2. If you haven't already, you can read part 1 here.

So how do we get the 1.5 seconds of /reload time down? Well, I'd already spent a lot of time back in TSM 1.x optimizing the crap out of the compression / decompression algorithms. Any further optimizations that could be done there would have a very minimal impact if any, so that wasn't an option. How else can we get the times down? Well, the answer is to do the decompression and compression on-demand for each individual item, rather than all at once when the addon is loaded / unloaded. This is essentially removing the 1.5 seconds worth of processing from the load bar, and adding a little extra work the first time each item's data is being requested.

But wait, doesn't this extra work just move the slowness from the load bar to the general usage of AuctionDB? One might think so, but it's important to consider that there's thousands of different items in AuctionDB's database, and you generally only care about a small subset of these (maybe a couple hundred) throughout a session, and only a dozen or so at any one point in time. Because of this, the extra processing to decompress each item's data the first time it's accessed is not noticeable, and we don't have to do any computation for the majority of the items which we don't care about the data for.

At the same time, AuctionDB will also store an up-to-date compressed version of each item's data so that when we go to save the database, we don't have to do anything except grab all these already compressed versions of the data and stick them together. This means any time the data is changed for an item (ie a new scan was performed), we do a tiny bit of computation to re-compress that data, in addition to updating the un-compressed values as well always have. This does add to the time it takes to process scan data, but we don't care about this as much since scans aren't that frequent, and they are expected to take a non-zero amount of time. If the app is being used to do the scans, we do care about this because it's being done upon logging into the game, but I'll get back to the app in a later part.

So, I partially lied before. There are cases where we do care about every single item. Specifically, for the vendor search in TSM_Shopping which looks at the min buyout of every single item. To avoid this having to decompress a ton of data all at once, when you first log in, AuctionDB kicks off some background processing which slowly goes through all the items and decompressing them. It is doing the processing slow enough to not take up any significant CPU time, which usually means it takes about 15 seconds to complete all the items from the time you first log in. After that's done, no extra work is needed for accessing any item, including those not accessed during that 15 seconds and for accessing all the items for a vendor search. This background processing creates a lot of garbage in memory, which I'll discuss further in another part.

To summarize, AuctionDB takes the decompression processing and does it as the data is needed (as well as in the background on its own). The addon is also storing an up-to-date compressed version of the data at all times, so the amount of processing that has to be done during a /reload has been all but eliminated. There is still a small bit of work to split apart the long string of data into individual items (login) and then stick them back together (logout) , which takes about 1/50th of a second: a negligible amount of time. There's one problem with this though. If you import data from the app, it requires that everything be decompressed first so that it can recalculate the market value with the new scan data included, which ruins this entire plan of decompressing on-demand. In the next part, I'll discuss how AuctionDB gets around this problem. See you tomorrow!