View RSS Feed

//no comment

Automatic Group Import Links

Rate this Entry
I recently made a preview post for a new feature that was just released with r237 of the beta version of the TSM Application. There will be another update (r238) shortly with some tweaks to the format of the group import links. We are still working on the web interface for creating these links, but I thought I'd give some technical details about how it all works (as of r238).

From a high level, when you click on a URL, your operating system knows that the TSM Application can handle the URL (the TSM App registers itself as being able to do so when you first run it). On windows, it'll then launch a second copy of the application and pass in the url, which is then sent the already running instance of the app before the second instance quits (only one TSM app can be running at a time). On Mac, it'll send a notification to the already running application, with the URL contained within the notification information. On either operating system, if the app isn't currently running, a new instance of the app will be started, and it'll essentially notify itself of the URL and take the appropriate action. For group imports, this action is to fetch the actual import string from the TSM servers, and then place that into a specific file within TSM which will be read next time you log in or /reload.

Now let's get into the format of the URL itself. First of all, there were a few requirements when designing the format:
  1. The protocol (the part of the URL where "http" usually goes) had to be unique, but easy to recognize as TSM.
  2. The URL had to be reasonably short and could not contain any special characters.
  3. The method of creating the URL had to be secure so that there's no chance of malicious parties creating their own URLs which could potentially do bad things.
  4. Overall, it had to have built-in support for future expansion. For example, there needed to be support for a variety of different types of links. Importing groups is just the first of many potential uses for these URLs.

With that in mind, here's what I came up with: tsmapp://8110aed5837dc60436386d396c666c6e
NOTE: This is the URL I used in the preview video. Feel free to try it if you have the latest alpha of TSM and r238 or higher of the app.

First of all, the entire URL is in hex, which satisfies the "no special characters" requirement. It's arguably not short, but the important thing here is that for group imports it is of fixed length. So, no matter how many group import URLs are created, they'll all be the same length. Also, the protcol is "tsmapp" which is unique and easily identifiable as being TSM. So far, this satisfies the first two requirements.

Let's dig a little deeper into what the jumbled mess of hex characters actually means. There are two parts to the URL. The first 8 bytes (16 hex characters) is the header, and the rest is the data (in hex). In this case, the header is 8110aed5837dc604 and the data is 36386d396c666c6e. Let's look at the header first. It has the following fields of various different lengths which add up to a total 64 bits:
  • Type - This is an enum value which differentiates different types of URLs. There are currently two different types. Type 0 is used for testing and simply prints out the data to the event log. Type 1 is the group import type, which the type of this example URL. This field satisfies requirement #4, and there are already plans for additional types being added (stay tuned).
  • Length - This is the length of the data in bytes. In this case, and in the case of all group import URLs, the data is 16 hex characters long, which is 8 bytes.
  • Version - The version of the format of the URL. This field allows us to change the format and potentially provide backward compatibly in the future. We're currently on version 2.
  • Magic - This is simply a fixed number that'll always be constant. This is used internally for basic validation.
  • Hash - I won't go into the details of how this is calculated, but this is a secure hash of the entire URL. This field satisfies the security requirements. To calculate this hash requires information / keys which we are keeping private. This way, no third party can crack the format of the URLs and create their own malicious links. If any part of the URL is changed, the hash will be completely different and the TSM app will be able to tell that the hash is incorrect and will ignore the URL completely.

The "data" is up to 128 bytes (256 hex characters) in length (guess how long that header field is ). As I mentioned, all group import URLs currently contain 8 bytes of data (16 hex characters). This corresponds to 8 base-36 (0-9a-z) characters. In the case of this example, if you pass the data through a hex/ascii converter, you'll see that the data is 68m9lfln. This corresponds to a unique ID in a database on the TSM server which is storing all the group imports. The reason that the group import string itself isn't contained directly in the URL is that it would make the URL extremely long. So, there's an intermediate ID which the app can then use to get the full import string from the TSM servers.

One last fun fact that I'll end with is that the scripts which support this on the TSM servers are all in PHP, but manipulating the bits in the header in PHP would have been a nightmare, so the PHP script is actually calling a small C++ program to do all the low-level bit manipulation work.

Updated July 5th, 2014 at 04:29 PM by Sapu94

World of Warcraft , Development