View RSS Feed

//no comment

AuctionDB: Under the Hood - Part 3

Rate this Entry
This is part 3. If you haven't already, go back and read part 1 and part 2.

We have now eliminated the 1.5 seconds of load time by loading the data slowly in the background as well as on-demand when it is requested. There's still a problem though. Importing data from the app also adds to the load time of the addon, and can't happen without first decompressing all the data. How do we get around this issue? The answer is that both problems can be solved the same way.

Remember that, in the last part, I mentioned that AuctionDB slowly goes through all the items and processes them in the background, rather than always waiting for each item's data to be requested. What we can do is, perform all the importing for each item at the same time this background processing does the decompression for the item. So, not only are we doing the decompression a little bit at a time, in the background, we are also doing the importing of data from the TSM app in the same way.

This method does still require us to do a bit of processing upon loading which will actually copy the raw data from the app import into the saved variables, from where it's then importing via the background processing. Why do we copy it to the saved variables? Imagine what happens if you /reload or logout before this background processing is complete. You wouldn't want to lose all the data which has not yet been imported, so we store the non-imported data in the saved variables file so that we can continue next time the addon is loaded if we're interrupted. Saving the non-imported data in the saved variables will potentially increase the saved variables file if not very much of it gets processed, but this is not as big of a concern as loading times and in-game CPU usage.

So, what about memory usage? First of all, memory usage doesn't matter until you run out. Every once in a while, WoW's garbage collector will run and clean up unused memory, typically once every 5-10 seconds on average. If an addon used 40MB of memory in one function, and then didn't need any of that memory anymore, it will still show that the addon is using that much memory until the garbage collector runs and cleans it up. Memory usage primarily becomes an issue if the garbage collector is forced to run constantly because an addon is generating a ton of "garbage" memory. The background thread that AuctionDB is now running will generate a significant amount of memory usage, which is why you'll notice AuctionDB's memory usage going up fairly quickly when you log in. Once the background processing is done and the garbage collector cleans up the unused memory, AuctionDB will settle back down to a more reasonable memory footprint (15MB - about the same as TSM_WoWuction for me).

There are a few other recent changes to AuctionDB. For starters, we'll now more aggressively clean up data that's older than 14 days (the max AuctionDB considers when determining market value). Also, AuctionDB will consolidate scans wherever possible via a running average mechanism which greatly reduces the amount of data stored for people who scan a lot (with the TSM app for example).

As a result of all these new changes, AuctionDB is now faster, smaller, and more reliable than ever. I hope you've enjoyed this behind the scenes look at AuctionDB. Feel free to leave a comment on any part which you have a question on.