Register
  • Consortium Total Goblin System

    Hey folks.

    This was originally going to be called the Consortium Notification System, and it was supposed to have been completed a few weeks ago. Hopefully you'll see why I decided to change the name AND why it's taken a bit longer to develop. As this is an initial version, I'm sure there will be lots of bugs and features that need to be implemented. Please check out the "known issues" and "planned features" after the initial guide for more information.

    Overview of the CTGS:
    When development initially began of an alternative TUJ-inspired notification system, I was met with several quick hurdles. The biggest and most crippling was the fact that my personal shared hosting plan didn't offer significant enough resources to process more than 8-12 realms at a time. A far cry from the nearly 500 US and EU realms, I scrapped the idea altogether for a few days thinking it couldn't be done. In a separate discussion, I was reminded of WAMP, and things took a turn for the better again. Once I realized the potential of having an external database on the same machine as your WoW/addons/spreadsheets, etc, I knew this could be much, much bigger than just a notification system. So here we are.

    Pros:
    Flexible - Your data for Your realm the way You want it. Also, the only storage limitation is the capacity of your own local machine, so there is plenty of room for expansion here.
    Integration - The development platform offers many options for getting data in and out of your existing resources, including in-game addon data.
    Accessible - Run the application on your terms. Coding is completely open-source, so you're free to add/remove or modify anything that doesn't meet your needs.

    Cons:
    WAMP server requires a not-insignificant amount of default space (somewhere in the realm of 250MB)
    Does require the machine it is installed on to be ON and also actively connected to the internet in order to be most useful.
    Updates and Changes will require manual patching and upgrading. I will try and make this as painless as possible for everyone, and hopefully you're all at least "used" to this process given the nature of WoW addons, but still a drawback compared to a 100% "online" application.

    Alright, let's get to the good stuff

    Current Version:
    version 1.0.0
    link: ctgs_v1_0_0.zip

    Installation Guide
    (NOTE: this installation process HAS NOT been extensively tested on multiple platforms and OS'es, so please be sure to include as much information about your system in any bug reports as possible)

    1. Download WampServer
    For more information about, and to download the WAMP files, visit Install PHP 5 Apache MySQL on Windows : WampServer.
    *** Important - this application's install procedure was specifically designed under default wampserver conditions. While their are many alternative options (LAMP, XAMPP, MAMP, etc.), I'm 99% certain the application WILL NOT install correctly under any of those, so if you are absolutely unable to run WampServer, please PM so I can work with your situation.

    2. Install WampServer
    Follow the default installation settings. When you get to the following screen, ignore your impulse to throw custom values in here. We'll change this later, but since there are additional settings we need to adjust, it make sense to do them all at once. If you do anything at all, remove both values altogether and attempt to proceed with empty fields.



    3. Download Sendmail
    One of the issues I first ran into with WAMP was the fact that the default PHP mail() function doesn't work. This was obviously a huge problem for a notification system, but I found a suitable workaround. NOTE: It is my understanding that XAMPP actually comes included with Sendmail, but since I had already gone to all the trouble of finding and configuring the WAMP/Sendmail solution, I stubbornly stuck with it.

    Sendmail Link: sendmail

    4. Unzip and Install the Sendmail files.
    Copy the entire "Sendmail" folder to your Wamp's /bin folder. Default should be C:\wamp\bin\. The resulting sendmail.exe file should end up being located at C:\wamp\bin\sendmail\sendmail.exe - if it isn't, you'll likely run into problems.

    5. PHP Settings
    Launch WampServer. When wampserver is online, the icon should be green in the system tray, as such:



    I did experience some conflict when using both Skype and wampserver simultaneously. I was able to remedy this by changing my Skype settings to ensure Skype no longer communicated via port 80. See below.



    Next, we need to enable a couple of important PHP extensions. Click on the wampserver system tray icon, then > PHP > PHP extensions.



    The 2 you'll need to enable are CURL and OPENSSL.





    You'll also need to restart the server in order for the extensions to be activated. Click the "restart all services option" as seen below.




    6. php.ini
    There are actually 2 php.ini files that will need to be configured here. The first one affects browser based settings, and the second one affects the php command-line program, which we will be using with task scheduler/CRON jobs, etc. in order to get our program to actually "do stuff". My recommendation would be to edit the browser based php.ini file first and then copy it and replace the command-line php.ini file second.

    The main php.ini file can be easily accessed through the wampserver tray interface, as such:



    Once inside the php.ini file, we need to find our mail function settings. Best bet would be to do a search (CTRL+F in notepad) for "smtp". This is the section we're looking to get to and modify:



    Clear any previous values from the following fields:
    -SMTP
    -smtp_port
    -sendmail_from

    Add the path to our sendmail.exe where it says sendmail_path = . The default will be "C:\wamp\bin\sendmail\sendmail.exe -t" Be sure to include the -t parameter and MAKE SURE you remove any semicolon from the beginning of this line. Semicolon denotes a non-executed comment in our .ini file and will cause our mail() function not to work. IE. Pretty big deal .

    Save your php.ini file.

    Now we need to change the second one. Under default installation, the first php.ini file can be found under \wamp\bin\apache\Apache2.*.*\bin\. Copy the php.ini file located there. The default location for our second php.ini file is \wamp\bin\php\php5.*.*\ - overwrite this php.ini file with the first (edited) one. Feel free to make a .bak version first, if you like.

    Click "restart all services" again to activate our changes.

    7. Download the application
    link: ctgs_v1_0_0.zip

    Unzip and copy the "cgts" folder to the \wamp\www\ folder. You can easily access this folder by clicking the www directory option from within the wampserver tray options.



    Now, click on the "Localhost" option, again under our system tray options. This should open up your default web browser and take you to a page that looks something like this:



    The "ctgs" folder show show up under "Your Projects". If not, your folder didn't get copied to the right place.

    NOTE: Much of the application's interface uses jQuery from this point on. As such, you will need to ensure that javascript is enabled in your browser for this to work. Making a the application fully-web accessible is on my list of things to tackle, but it admittedly is nowhere near the top, so if this is a deal-breaker for you, send me a PM and we can talk about your options.

    8. Installing the Application
    Click on the "ctgs" folder. Then click on "install.php".

    Welcome to the Consortium Total Goblin System!

    The system is built to support up to 5 realms from either US or EU region, however, the install script only accepts 1 to begin with. So, choose the region and realm for the primary realm you'd like to use for your data. Also, enter the email address where you'd like to RECEIVE notifications.

    Mailserver Settings: our wampserver/sendmail setup requires a mailserver in order to actually send email. As I mention in the install, I highly recommend using gmail, but many mailservers will work. FYI - I tried to get this setup under my godaddy-hosted account, but was unsuccessful. Gmail works fine . If you decide to use gmail (which you should), you will need to enable POP access from within your email account settings. See below.



    The SMTP details that gmail uses are explained here: https://mail.google.com/support/bin/...y?answer=13287, but I've also noted them in the install script. If your mailserver requires authentication (as gmail does) be sure to toggle that option and then submit your username and password as well.

    Upon clicking install, you will be redirected to a confirmation page and all of your initial databases and tables will be created and your configuration settings will be saved. You should also note that the install script automatically updates your sendmail.ini file with your SMTP settings. If you need to check/change those settings, the default path would be C:\wamp\bin\sendmail\sendmail.ini. The script should also send you an email to the address you specified. If you do not receive this email (allow an appropriate delay before panicking), it pretty much also means that you won't receive any of your notification emails, so we should really get to the bottom of that before continuing on. Finally, click the "Launch" button to get to the CTGS app.

    9. Running CTGS - Home
    I've envisioned this page as a launching point for other elements of the application. Currently, it doesn't do much.

    9b. Running CTGS - Notifications
    Should be pretty self explanatory. Click "create notification" to get started. NOTE: There might be a slight delay when checking item names, since it fetches it live via the API, but app will return "Item not found" if you enter an invalid item ID, so just be patient.

    There is currently no confirmation when deleting a notification, so use with caution.

    9c. Running CTGS - Database
    Currently not functional. I believe DB options will end up being a focal point of the CTGS, so I've given it it's own page. Expected features will likely include import/export, StatsDB settings, such as time period, fields, etc. Stay tuned, as this will likely be one of the areas that I update first.

    9d. Running CTGS - Settings
    ATM - all it does is display your settings, and none of the update buttons actually do anything. I'll fix this soon enough, but I thought it would be better to get an initial release ready to go, since I can easily walk someone through all of these settings if you run into any installation/configuration issues.

    10 - Getting it to do something.
    Most versions of Windows, especially modern ones, include some form of task scheduling application. This is what we'll be using to get the CTGS to actually do stuff. Included in the download is an executable batch file called ctgs.bat, which calls the php.exe command line and passes it the parameter of our fetch_data.php script. If your installation has used non-default file paths, this will need to be changed as well. PM me for help.

    Anyway - setup a scheduled task for the ctgs.bat. You can specify any frequency you like, since the script will automatically check and compare last modified values. My recommendation would be to at least run the script every 30 mins so that you can ensure you never miss an API dump, but obviously the more frequently you run it, the more recent your notifications are guaranteed to be.

    Final Notes
    Please see the next 2 posts in this thread for Known Issues and Planned Features. If you have any technical issues with the application or come across any bugs, feel free to either reply to this thread, PM me or send me an email at kathroman@breakthebankwow.com.

    I'm quite tired, and am off to sleep .

    Current Version:
    version 1.0.0
    link: ctgs_v1_0_0.zip
    This article was originally published in forum thread: Consortium Total Goblin System started by Kathroman View original post
    Comments 110 Comments
    1. Gadjou's Avatar
      Gadjou -
      Wow! Thanks a lot! Great work
      I installed a couple of notifications. I can't wait to see if it works ^^
      ++
      Gadjou
    1. Gimp's Avatar
      Gimp -
      During step 8 i got Warning: mail() [function.mail]: Failed to connect to mailserver at "localhost" port 25, verify your "SMTP" and "smtp_port" setting in the php.ini or use ini_set() in C:\wamp\www\ctgs\inc\save_settings.php on line 203im no good with database stuff :S
    1. Xiya's Avatar
      Xiya -
      Hey, this is something I've been working on in the past, but never finished. My notification was based on XML output from the armory, but Blizzard disabled it when adding BattleNet API, and I needed to rewrite most of the code. I tought I would give your system a try, so I installed it on a virtual machine. Running Ubuntu, no LAMP installed so I started by installing the regular Apache2+PHP5+MySQL5.

      First thing I ran into was installing other necessary packages. Was solved pretty fast once I understood why I was not getting the realms list : PHP curl was module missing (probably installed by default with WAMP).

      Then I got an error in the save of settings. The last modification DB is created as "lastMod" but then it's used as "lastmod". Looks like my MySQL installation is case sensitive .

      I'm checking why the DB is not populated now. I guess I'm missing something again, but the php script doesn't spit any error... I hope to find out why easily.
    1. Xiya's Avatar
      Xiya -
      I see. The DB was not populated because... it's never populated . Anyway, I got errors when trying to fetch AH data. Fixed by adding a region parameter to ParseAH - the function was complaining about undefined region. I guess you have enabled global variables, and I don't.

      It's now triggering the notifications and sending mails (after installing a mailer).

      Now adding notifications .

      Solution:

      Edit inc/fetch_data.php, change line 42 :
      PHP Code:
      $horde ParseAH"Horde"$realm1$region1 ); 
      And also change line 128 to :
      PHP Code:
      $alliance ParseAH"Alliance"$realm1$region1 ); 
      Edit inc/functions.php, change line 127 to :
      PHP Code:
      function ParseAH$faction$realm$region 
      This should fix it.

      If you want to test without waiting for BattleNet update, you can edit the lastmod table, and change the timestamp from whatever it is to 0. I think you should have phpmyadmin installed with WAMP.
    1. Keldara's Avatar
      Keldara -
      Wow ... This seems awesome.

      Must ... test ... now!
    1. Wanamaker's Avatar
      Wanamaker -
      Can you add a real overview at the beginning please? Currently you say you wanted an alternative to TUJ messages, then you dive into issues you had. A nice summary of what it is would be great.
    1. Kathroman's Avatar
      Kathroman -
      Hey guys. I'll be in IRC in a bit to field questions. I'll also fix/update the files from work, here and upload a new version of the affected files.

      @Xiya - thanks for the help/support so far. TBH, a lot of code was copy/pasted form another project of mine...hence the "re-coding" under planned features.

      @Gimp - you're problem actually looks like it has to do with your php.ini settings - can you confirm that you did everything as outlined in step 6? Specifically, check to ensure you removed the semicolon from the sendmail_path line and removed those other settings mentioned.

      I got that exact same error while trying to setup the .bat file, that's how I realized there were 2 separate php.ini's in the first place.
    1. Kathroman's Avatar
      Kathroman -
      Quote Originally Posted by Wanamaker View Post
      Can you add a real overview at the beginning please? Currently you say you wanted an alternative to TUJ messages, then you dive into issues you had. A nice summary of what it is would be great.
      Unfortunately, that's all it is right now - an alternative to TUJ notifications. It will become more than that over time, but ATM - it's nothing fancy
    1. Kathroman's Avatar
      Kathroman -
      Version 1.0.1 uploaded. Fixed a number of minor bugs people have mentioned. You will need to replace the following files if you have a previously installed version:

      inc/fetch_data.php
      inc/functions.php
      inc/save_settings.php

      If you continue to run into problems, let me know. There might have been some permanent DB changes we'll need to look into.

      EDIT: 1.0.2 uploaded. Changed the ctgs.bat file to not include the "pause". I threw that in there for testing, but forgot to take it out again
    1. frankinabox's Avatar
      frankinabox -
      Awesome, I had started working on something like this for myself. Ill report back once I have it set up, thanks amigo!
    1. Xsinthis's Avatar
      Xsinthis -
      /me skims through the thread

      If I'm correct, you should be able to plop this into an actual webserver without much fuss yes?
    1. Kathroman's Avatar
      Kathroman -
      Quote Originally Posted by Xsinthis View Post
      /me skims through the thread

      If I'm correct, you should be able to plop this into an actual webserver without much fuss yes?
      Actually, I'm working on something like that too. I'm afraid this probably will need some tweaking to be viable on an actual server
    1. Xsinthis's Avatar
      Xsinthis -
      Quote Originally Posted by Kathroman View Post
      Actually, I'm working on something like that too. I'm afraid this probably will need some tweaking to be viable on an actual server
      Well hop to it man! :P

      In all seriousness this looks seriously cool, I'm to lazy to actually set it up though lol. Good work
    1. Nearlydead's Avatar
      Nearlydead -
      Couple of quick comments and questions:

      First one is it says current version is 1.0.0 but in the comments you say you have updated it to 1.0.2. Just wondering if the link has been updated but not the title for the link or if it is actually still version 1.0.0.

      Also I assume under the notifications section price is based on per unit and quantity is how many are available? It would be nice if possible to have it so we have the option for stack sizes i.e When a stack (20) elementium ore is under 70g/stack. Other than that I love it thanks for all the hard work you've put into this, I really do appreciate it!
    1. Gimp's Avatar
      Gimp -
      Dunno if im doign somethign wrong, but i have the notifications all setup and the confirmation email came through etc.

      Ive scheduled the bat to run every 5 mins (which it does) but no notifications are getting sent (I created some test ones I knew wud flag a response)

      I dunno where to start to bug test it
    1. calianna's Avatar
      calianna -
      Looks like good stuff, @Kathroman! This is going to have to be my Thanksgiving project.
    1. Kathroman's Avatar
      Kathroman -
      Quote Originally Posted by Nearlydead View Post
      Couple of quick comments and questions:

      First one is it says current version is 1.0.0 but in the comments you say you have updated it to 1.0.2. Just wondering if the link has been updated but not the title for the link or if it is actually still version 1.0.0.

      Also I assume under the notifications section price is based on per unit and quantity is how many are available? It would be nice if possible to have it so we have the option for stack sizes i.e When a stack (20) elementium ore is under 70g/stack. Other than that I love it thanks for all the hard work you've put into this, I really do appreciate it!
      The download SHOULD be 1.0.2 - to double check, note the version of your functions.php, fetch_data.php, and save_settings.php files. There should be a commented out section near the top with author, date, desc, version, etc. The ones for those 3 files should read 1.0.1.

      Re: stack sizes - I'll add it to the list

      @Gimp - just to clarify, you shouldn't actually be processing data every time you run the script. Blizzard adds AH dumps APPROXIMATELY every hour, but they've made no guarantees about that. As such, I recommend scanning for data in shorter intervals to ensure you don't miss any data. If you run it every 5 mins, 11/12 times, it likely will give you nothing, but it DOES means that your data will never be more than 5-10 minutes old.

      The only thing you might want to be careful of with that is the API resource limits. I doubt you'll have any trouble with just a single realm, but you also need to consider the item name lookups - which also count against your resource limit. I'd love to make the process more efficient, but I haven't come across an elegant way of fetching the entire item API from Blizzard yet. Also on the list, though
    1. Itzal's Avatar
      Itzal -
      This looks great! I can't wait to get home to try this, though, without the RAH, I can't really take the best advantage of notifications.
    1. neow1ng's Avatar
      neow1ng -
      Maybe make an .ini file for mysql db settings, sendmail location.

      also setup instructions for XAMPP would be great
    1. Gimp's Avatar
      Gimp -
      i had it scheudled to every 5 mins for 4 hours while i was at work, and set up a notification for more than 1 linen cloth, more than 1 Elementium ore.

      Both of these should have sent off a notification, but they ddnt in the whole 4 hours period