Details on building a Flex-based dashboard with a PHP backend

Leave a comment

Yesterday (June 1st) I wrote a blog post on how to set up an example application that I did for a webinar with Adobe and building a Flash based dashboard application (I would suggest watching it) connecting into to PHP.  Today I would like to provide some of the details of how I did all that.  I was a bit pressed for time while building the example app, and I did my standard thing, taking the Mythbusters motto to it’s logical conclusion.  But in the end I think it worked out well.  The example is complicated but a dashboard is intended to aggregate data from multiple different places and so while it was complex I also think that the attendees were able to see how and why data was processed as it was.

The dashboard application was a Flex-based app that could get summary information and batched information from a Magento based e-commerce site.  It would display a traffic summary, slightly delayed sales information (which products were selling best) and realtime notification of sales that occurred.

The way it worked was that all of the information that was going to be displayed in the dashboard was sent into one of three message queues.  That’s right.  The PHP application did not use a database. Well, it did, later on, but the actual registering of data was not done in the standard PHP way of doing a database insert.

Heresy!  No, actually I think there are a lot of problems that PHP developers solve with SQL that should be solved with message queues.  In this case, there were three things that I wanted.  I wanted traffic information, notification of a sale and the details of that sale.  But I really didn’t need to keep that information once it was processed and I didn’t care about ACID compliance or anything like that.  I basically wanted to throw data somewhere where I could get it later.  That’s where the message queue came in.

I chose ActiveMQ as my message queue.  I tried RabbitMQ but I ran into authentication issues that I didn’t have time to solve.  It was quicker to install ActiveMQ.  ActiveMQ has a feature that it shares with RabbitMQ and that is the Stomp interface.  Most message queues have some kind of binary protocol that they use to initiate communication.  But Stomp is a text based protocol.  What that means is that it is a) easy to build adapters for, and b) very easy to hack if you need one built for you.

Zend Framework has a Stomp interface, which requires some messing around with, but it also has an ActiveMQ interface that uses Stomp.  What that means is that it uses the Stomp interface but makes sure that it’s working with ActiveMQ with any “special” things that needs to be done.  ActionScript, the Flash-based programming language, also has a Stomp based interface available via a third party.  What this meant is that I could connect to a message queue and both the front end and the back end were able to interact with it.

What I wanted to do was send data into the queue that I was either going to a) retrieve directly using the dashboard, or b) pass to a PHP job for batch processing.  Messages are simple unstructured text.  A) was simple because I was just passing a notification.  B) was more problematic because I wanted to pass structured data, i.e. a log object.  But wait!  A serialized object is unstructured text!  So passing the object was no problem.  To do this I created a logger class that understood a few different classes.  We’ll start with those.

First the LogItem class

1
2
3
4
5
6
7
8
namespace log;
 
class LogItem
{
public $title;
public $url;
public $timestamp;
}

Now let’s take a look at the Logger class. (I am omitting the sales portion)

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
namespace log;
 
class Logger
{
 
private static $queues = array();
 
public function getQueue($queue)
{
    if (!isset(self::$queues[$queue])) {
        $config = new \Zend_Config_Ini(__DIR__ . '/../../etc/config.ini');
        if (!isset($config->mq->$queue)) {
            throw new Exception('Undefined queue configuration');
        }
 
        $stomp = new \Zend_Queue_Adapter_Activemq(array());
        $stomp->setQueue(
            new \Zend_Queue(
                array(
                   'name'=> $config->mq->$queue->endpoint
                )
            )
        );
 
        self::$queues[$queue] = $stomp;
 
    }
    return self::$queues[$queue];
}
 
public function log($queue, LogItem $log)
{
    $this->getQueue($queue)->send(serialize($log));
}
}

 

This class works in conjunction with a config file like this

1
2
3
4
mq.traffic.endpoint = /queue/traffic
mq.traffic.scheme = tcp
mq.traffic.host = localhost
mq.traffic.port = 61613

The Logger object receives a request to log a LogItem object on a specific queue.  It then calls getQueue() which looks to see if a configuration exists for that queue.  If it does it creates a new ActiveMQ adapter object, sets its queue name and returns it.  Then the log() method serializes the $log object and passes it to the send() method of the ActiveMQ.

At this point the data is in the message queue and you’re done, right?

Nope.  Now you need to pull it out of the queue.  On the PHP side it is pretty much the same as queuing it.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$stomp = new \Zend_Queue_Adapter_Activemq(array());
$stomp->setQueue(
  new \Zend_Queue(
    array(
      'name'=> $config->mq->$queue->endpoint
    )
  )
);
 
while (true) {
  $msgs = $stomp->receive();
 
  foreach ($msgs as $msg) {
    /* @var $msg Zend_Queue_Message */
    $logItem = unserialize($msg->body);
    // Do your thing
  }
}

 

In here we create our connection to the queue daemon and state the queue name.  Then, in this loop, we state that we want to receive data from the queue by calling the receive() method.  Generally a message queue will allow a connection to sit idle indefinitely until there is data on the queue.  But for PHP, since most are not long running programs, you will want to have a timeout.  Leaving the first argument null on the receive() call will set the timeout to be the default, but you probably should set it relatively low.  The receive() method will return an object of Zend_Queue_Message_Iterator which implements Iterator which means it can be used in a foreach loop.  Each item in the object returned will be an instance of Zend_Queue_Message which contains your data along with some meta data.  I didn’t care about the meta data so all I did was unserialized the body of the message which then gives the instance of the object that I had passed in to the queue earlier.

That covers about 1/8th of what I still have left to talk about concerning the webinar I did with Adobe.  But one of the more interesting things I did was use a message queue to communicate between a front end PHP application, a back end PHP application and a front end Flex application (which I didn’t get to).  I personally think that PHP developers should spend a little time looking at asynchronous communication and examining whether their applications could use it.  Perhaps this will help you get on your way.


Building a Flex-based dashboard with a PHP backend

Leave a comment

Today (June 1st) I got to give an online talk with Adobe on how to create a stunning analytics dashboard with Flex and PHP (recording here).  I’m a horrible graphic designer and so I don’t know how stunning it was, but from the technical perspective we covered a lot and it was a lot of fun to write.  Several of the attendees asked for code and so here is what I’ve written.  Like the webinar, it will take a little bit of time to get it working.  But the end result is that you’ll get a whole lot of stuff to think through.

What I did was take a Magento installation and create an observer that would send data to a message queue where it that data could be processed for either summary information or real-time notifications.  The dashboard contained graphs for traffic and product sales summaries and a notification list that would show new purchases as they were made.

To do this I used a lot of different programs, which you will need if you want to try it out.  You will need

  1. Magento
  2. ActiveMQ
  3. Zend Server (30 day trail available)
  4. MySQL
  5. Flash Builder for PHP (60 day trial available)

AS3Stomp is also used but I have included it in the workspace download.

Additionally, you will need to have two host names available on your machine.  I named mine magento.local and charting.local.  I set them in my hosts file (/etc/hosts or c:\windows\blah\blah\blah\hosts) but you could also use DNS as well if you needed to.  If you chose to run off of different host names you will need to change the gateway URL for the Charting project by editing the .flexProperties file and changing the serverRootURL property.

The first thing you need to do is download and install Zend Server.  You will need a license key to activate the Job Queue service.  Additionally you will need to add the directory in the project ChartingAPI, /library to your PHP include_path setting so that the logging and library classes can be included in your scripts.

You will then need to download and install Magento and then create a list of products.  Two or three should be enough.  If you want to easily integrate it with my code I would suggest that you create a blank PHP project and import the downloaded contents into the new project so that the project root is the document root for magento.local.  Do this before importing any of my project code.

Next, download and install ActiveMQ.  You will need to create three queues; saledata, sales and traffic.  You will also need to turn on Stomp support which you do by editing the activemq.xml file and adding the transport connector.

<transportConnectors>
<transportConnector name=”openwire” uri=”tcp://0.0.0.0:61616″/>
<transportConnector name=”stomp” uri=”stomp://localhost:61613″/>
</transportConnectors>

You may need to specify the JAVA_HOME directory in activemq.bat in the bin directory.

From there you can import my Magento extension into the Magento project.  Make sure that you import it into the root so that the code ends up in /app/code/local/Zend.  Additionally there should be a file called /app/etc/modules/Zend_Dashboard.xml.  This is there to inform Magento that you have a new extension that you want to use.  Once you have the code in place you will need to 1) Flush the Magento cache and 2) Enable the extension in the Magento settings (hint, it’s in one of the last sections).  With Magento, if you make a change and nothing seems to happen, it’s probably because the cache was not flushed.

To test the messaging configuration go to the ActiveMQ management page at http://localhost:8161.  Browse around the Magento site a little and check back to the queues page.  If you see messages waiting then your configuration is working.

Next up is to set up the  job queue.  In the ChartingAPI project there is a directory called public.  That should be the document root for the charting.local virtual host.  In the public directory is another folder called jobs.  In there are two scripts that will need to be run at least once.  Each of them requeues itself and so if you run it once via the URL (http://charting.local/jobs/process.traffic.php and process.sales.php).  Run them once and only once.  If you want to see what they do prior to injecting them into the job queue comment out the three lines of code starting with the object ZendJobQueue.

Now you should be ready to go with the Charting project.  If it is not able to compile due to a missing Stomp class you will need to reference the src folder in the project AS3Stomp.  After that is working right click in the Charting project on the Charting.mxml file in the src/(default package) directory and select Run As -> Mobile PHP Application.  That will kick off the charting application.  If there are problems you can select Debug As -> Mobile PHP Application and debug both the Charting application’s ActionScript or the PHP application’s service calls.

The link for the workspace zip is adobe.webinar.zip.

You can also see my slides on SlideShare.

I will leave you with that for the time being.  I intend to do another blog post on some of the reasons for doing things the way I did them along with some discussions about the architecture for the different integration points.  Additionally, I found that this webinar has highlighted a couple of pieces of needed documentation both for ActionScript and asynchronous community in PHP.  So, I have my work cut out for me for the next little while.  🙂


Selling 12k tickets in 2 minutes

Leave a comment

One of my little projects that I’ve been working on is an example where I wanted to build a system that would scale to be able selling 100k tickets in 5 minutes.  I’m happy to say I’ve succeeded, but I am not able to prove it.  It turns out that SOMEONE put a quota on how many EC2 instances I could spin up at a given point in time.  So, I had to opt for a cluster of 1 admin node and 19 worker nodes.  It wasn’t able to scale to the 100k tickets in 5 minutes, but with a few more instances I could have done it.

I intend to milk this one for content for a while, but at the same time I wanted to get a video up and running showing some of the things that I did to get it running.

I had originally decided to try and run it with simple Zend Server instances (without Cluster Manager) but after getting a cluster of 20 servers up and running I went through a test run and there was an error… and I didn’t know where.   So I relented and installed Cluster Manager.  Holy crap, you might say.  While I knew the selling points for Cluster Manager, and stood behind each one, this was the first time I had personally experienced how great it is to have your cluster managed by it.  This application depends very much on throwing data somewhere and expecting it to get processed.  But I had a front end load balancer to handle browser requests and then two other load balancers to handle logic execution and, quite honestly, there was no easy way for me to track the problem from end to end.  I installed Cluster Manager and had the issue identified in 5 minutes.  Stupid config.  Fixed it and was on my way.  My conclusion is that if you do anything with a load balancer in PHP and you want to catch bugs, Cluster Manager is a virtual necessity.


Added (PHP 5.3) job queuing to my WordPress instance

4 Comments

One of the things I liked on my old blog was the ability to have a Popular Posts section that was based off of Google Analytics. I had also like that I was using pre-caching, i.e., a cache that does not expire but is, rather, overwritten.  There are two benefits to this.  1) There is no lag time when the cache expires and I need to contact Google to get new data. 2) If my connection goes down for whatever reason (bad data, time expired, modified password), the data stays in the cache until you can get the problem fixed.

So I had missed that, but it was not overly important so I left it.  But yesterday was a day where I needed something that was both engaging and brainless to do.  So I decided to implement my Job Queue API code for WordPress so that I could write a WordPress widget that would put the popular posts in the sidebar.

It was actually relatively easy to do.  But the cool part was that I was able to extend WordPress, which still contains code that was written around when the Martini was invented, and PHP 5.3 code, which is what my Job Queue code was based off of.  This is part of some “ESchrade enablement” which I am building as a WordPress plugin and so it contains some other stuff than just the JQ part of the plugin.  You can download it from here and put it in your wp-content/plugins directory.  There are some things that aren’t working quite right yet, and probably won’t.  I don’t really have the desire to become a WordPress developer.

What have I learned or re-enforced from this?

  1. PHP backwards compatibility is pretty darn good
  2. If you build a software application, make building plugins easy, even if it’s your own app.  (WordPress is kind of win/lose on this)  I may end up looking at Event_Dispatcher.