Tag Archives: Zend-server

Magento-based asynchronous execution

Working with an off-the-shelf shopping cart usually requires a little bit of patience. Scaling an e-commerce site does have its share of problems. There is a LOT of interactivity that needs to be implemented. This can be things along the lines of generating targeted ads, sending email or charging a credit card.

To charge a credit card, the ecommerce software will usually take the credit card information from the end user, put it into some form of web service request and the request is submitted to a remote system.  While that web service request is taking place the process handling PHP is unable to take additional require requests to serve regular pages.

One option that you have is to complain that PHP doesn’t have threading.  That’s not the best thing to do.  As Marco Tabini said recently on Twitter “Every time someone mentions threading in PHP, an angel’s wings enter a race condition”.  Threading solves some problems.  However, chances are that while you may want threading you probably don’t need it.

However, while you probably don’t need threading, there are plenty of times when being able to do things asynchronously would be beneficial.  The example that I started looking at was a credit card request.  While waiting for the credit card transaction to occur you have one of two options.  1) Let the screen be blank while you’re waiting for the transaction or, 2) use some kind of output buffering and progressive rendering to let the end user know that the transaction is, in fact, being processed.

However, there is another, better, option. Rather than either spending loads of CPU time to process loads of logic, such as personalized ads, or have long wait times, such as processing a credit card, you can have this processed “behind the scenes” so you can immediately respond to your customer.

A simple example of what a Job Queue architecture can look like is almost like a hub and spoke architecture except that instead of the hub being the center it is actually the outside.  Ok, so a simple Job Queue architecture is exactly the opposite of a hub and spoke architecture.  Sue me.

The way it works is that there is a backend server, or cluster of servers, that handle servicing Job Queue requests. The requests are made from your front end web servers which is sent to a URL on the backend.  The URL is where the logic is that needs to be run.

Using a simple architecture you can just have that URL be a simple script that is run.  However, I prefer a more structured solution if I am going to integrate asynchronous processing in my application.

This is where the Magento connection starts.  I have already written about how to implement a structured asynchronous mechanism.  This is the same implementation that I use on this blog site.  What I’ve done is take that implementation and re-implement it so that it works within the context of a Magento application.  I have placed this implementation on Github.  It is not yet part of Magento Connect, though I intend to do it and I intend for it to be provide free of charge.  However, what I also wanted to do was give others the chance to look at it and improve it prior to putting it on Magento Connect.

Implementing your own task, be it pre-processing advertisements or processing a credit card is very easy.  Processing a credit card, however, should be done with the addition of encrypting the data is that data is stored “as-is” in the Job Queue database.

Defined in this library is a class called ZendServer_JobQueue_Job_Abstract. This is the base class for defining a task.  There is only one method that you need to implement, though you can implement as many of your own method as you want, such as getters and setters.  The method is called _execute() and this is where you would implement the logic that you want to implement.  However, it is important to note that because this is run on a completely different machine once the task has been set to execute no changes that you make will be reflected in the job if it has started running already.

In the code download there is an example of how to implement this class.  It is called ZendServer_JobQueue_Model_Mock.  All it does is write to the PHP error log, but does so asynchronously from the Job Queue URL.  The code looks like this

class ZendServer_JobQueue_Model_Mock extends ZendServer_JobQueue_Job_Abstract
{
            protected function _execute()
            {
                        error_log('Mock Model run');
            }         
}

One thing to note.  It’s freaking easy to implement this!  If you want to run this, here is your code.

$task = new ZendServer_JobQueue_Model_Mock();
$task->execute();

Wham.  Bam.  Done.  It is now running on your Job Queue server.  I won’t get into all of the details on how it’s done, though.   You can take a look at the abstract class and understand the details yourself.  It is open source after all.

But if you were to run this code right now you would probably get an exception thrown.  That is because you have not configured your Job Queue yet.  In order to do that you need to look etc/config.xml file.  You need to edit the element config/modules/ZendServer_JobQueue/jobqueue/url and specify the URL of the job queue entry point.  Since there is an index controller for the ZendServer_JobQueue extension and I just used the standard router, the URL would be $HOST/jobqueue.  It is recommended (highly recommended) that you make this URL available over the localhost or private.  It is not by default, so I recommend that you set this up using either a virtual host that only listens on 127.0.0.1 or on a machine that is behind a firewall.

So, that’s pretty much it.  Though I suppose you’ll need Zend Server as well. 

To install Zend Server you can go to zend.com and set up your system to install or download (for Windows) Zend Server.  It comes with a 30 day free trial.  Give it a shot.  If you have trouble feel free to post on the forums at forums.zend.com or you can post a comment here and I can try to answer it.

Happy coding!

Unit Testing Zend Server PHP Job Queue actions

One of the problems with doing asynchronous execution is that it can be actually quite difficult to test those asynchronous jobs.  In this video I describe how to implement a testable queuing system.

Pre-caching PHP content with Zend_Cache_Manager and the Zend Server Job Queue

With the web being what it is today there can be a lot of times when you want to aggregate data from many different sources and bring them together in a single page.  I have not done much of that on my site simply because that means that I then need to learn a bunch of different API's.  However, since Youtube is the #2 search engine I figured that it might not be a bad idea to aggregate some of my YouTube content on my page automatically.  I don't necessarily want to do a blog post about each individual video I post, but I wanted there to be some place where I could just list them out.

I have two places where I post content.  Youtube and Facebook.  However, polling each site individually for each request is not conducive to having a page that renders quickly.  The thing you do NOT want to do is poll YouTube each time someone comes to an individual page.  The way around this is to cache the contents of the YouTube or Facebook query so you don't have to do that.  Then people are able to re-use the previously defined data when they view that page.  What this does is make most of the new requests to that page much faster since they don't have to re-load that data from YouTube or Facebook.  However, there's a bit of a problem there as well.  Every X number of minutes, the cache will expire and someone will take the hit of connecting to Youtube.  With a moderately low traffic site such as mine, that hit is something I didn't want to make my users endure when they came to the site since there is a decent probability that the cache will expire in between individual page requests.  And, working for Zend, I can't have a page that renders slowly, can I.

So what I did was create a new Zend Server Job Queue task, which I have detailed several times (and there should be a link to several on the side) that would connect to both YouTube and Facebook.  This task would insert the results into a cache (you could use a database if you liked) so that when someone came to a page that they would be seeing the cached data rather than polling YouTube.  From a settings perpective, the cache is set to never expire the content there.  But because I set the task to run once an hour the content is going to be refreshed.  Using this pre-population method I am able to keep requests snappy which at the same time providing mostly up to date content.

The task to do this is relatively simple.  First I edit my application.ini file to set up the cache manager.

resources.cachemanager.video.frontend.name = Core
resources.cachemanager.video.frontend.options.automatic_serialization = true
resources.cachemanager.video.frontend.options.lifetime = null
resources.cachemanager.video.backend.name = File

By defining these ini settings, Zend_Application will automatically instantiate an instance of Zend_Cache_Manager and set up a cache that is named "video" with the individual options as specified.  What this means is that I could create another cache interface by taking these configuration lines and giving it its own configuration settings.  It could be different settings or even a completely different backend, or a different front end.

Then I create my task class.

class Admin_Task_VideoPreCache extends Esc_Queue_TaskAbstract

    protected function _execute(Zend_Application $app)
    {
        $yt = new Zend_Gdata_YouTube();
        $options = $app->getOption('video');
        $uploads = $yt->getUserUploads($options['youtube']['id']);
        $manager = $app->getBootstrap()->getResource('cachemanager');
        /* @var $manager Zend_Cache_Manager */
        $manager->getCache('video')->save($uploads, 'youtube');
       
        $query = 'SELECT title, description, embed_html FROM video WHERE owner=' . $options['facebook']['id'];
        $url = 'https://api.facebook.com/method/fql.query?query='.urlencode($query);
        $data = simplexml_load_string(file_get_contents($url));
        $videos = array();
        foreach ($data->video as $video) {
            $videos[] = array(
                'title'    => (string)$video->title,    
                'description'    => (string)$video->description,
                'embed_html'    => (string)$video->embed_html
            );
        }
        $manager->getCache('video')->save($videos, 'facebook');
    }
}

Because the Zend_Application instance is always passed in I can easily get access to the predefined cache manager object in here for when I need to store the data at the end of the task.  Then in the task I use Zend_GData_Youtube to query YouTube and I do a simple FQL query to Facebook to get the Facebook videos (which stopped working between test, staging and production.  Go figure).

The next thing I have to do is make that data available to a view.  To do that I need to create a new controller action that queries the cache manager.

    public function myvideosAction()
    {
        $app = $this->getInvokeArg('bootstrap')->getApplication();
        /* @var $app Zend_Application */
        $cm = $app->getBootstrap()->getResource('cachemanager');
        /* @var $cm Zend_Cache_Manager */
        $this->view->youtube = $cm->getCache('video')->load('youtube');
        $this->view->facebook = $cm->getCache('video')->load('facebook');
    }

Then all I need to do in my view is iterate over the data and I'm pretty much good to go.  Because the cache data has been prepopulated my visitors should never have to take the hit of populating the cache and by using the Zend Server Job Queue the task of populating the cache is extremely easy to do.

Debug Tunneling

Debugging is one of the things that every single PHP developer in the world needs to know how to do.  Unless you are simply learning the language, learning how to debug will save you countless hours and save you loads of money in anti-anxiety medications.  var_dump() or print_r() is not debugging. At least, it's not debugging in a way that is very useful, or safe. 

There are two primary debuggers in the PHP world.  XDebug and the Zend Debugger.  I am not an expert in XDebug (though I really need to learn it better) so I will leave those discussions to someone else. 

Debugging on a local workstation is really quite simple.  I use the Zend Debugger Toolbar in my browser when I'm not debugging a Unit Test (how I usually test functional components, as opposed to visual components).  The reason I do this is because it's much easier to debug a certain context, such as a logged in admin user, than using the internal browser in Eclipse.  The toolbar is installed when you install Zend Studio or you can download it seperately and install it in Firefox manually.

The debugger toolbar

This works very easily when you are working in an environment with very few restrictions, such as a debug environment.  However, say you run into a problem in your testing or staging environments?  (I could also add your production environment, but you should actually never have the debugger installed on a production environment for security reasons.)  The way the debugger works is that Zend Studio opens up a port on the local machine so that when a debug session is kicked off, the debugger in PHP will open up a TCP connection back to Zend Studio and start the debug session.  The problem is that testing and staging environments often have outbound port restrictions, for good reason.  But that makes debugging an issue in one of those environments a little tricky.

The solution is the debug tunnel.  What Zend Studio does is instead of waiting passively on the local machine for a connection to come in, it will open up an HTTP connection to the remote server.  Then when a debug session occurs the debugger extension will communicate over a local port, which then forwards the debug information back to Zend Studio over HTTP.  The data is the same, it's just being tunneled over HTTP.  Because there is no outbound connection initiated and because the communication is occuring over port 80, if you can connect to the web server you can debug on it.  It should, however, be noted, that tunneling does not work on Windows systems.  I don't know the exact reason, but I would venture to say that it is due to PHP on Windows running as a FastCGI instance.  This would likely conflict with the long running request required to facilitate a debug tunnel.

The way this is done is by Zend Studio making a connection to a debug file.  That file is usually called dummy.php and contains the following code.

@ini_set('zend_monitor.enable', 0);
if(@function_exists('output_cache_disable')) {
        @output_cache_disable();
}
if(isset($_GET['debugger_connect']) && $_GET['debugger_connect'] == 1) {
        if(function_exists('debugger_connect'))  {
                debugger_connect();
                exit();
        } else {
                echo "No connector is installed.";
        }
}
?>

This code turns off all of the timers in monitoring, disables the cache and then checks to see if a debug tunnel is being requested.  If so, it basically sites on the debugger_connect() function.  This function will open up a port on the local interface and wait for connections to come in.  This file does not need to be on the domain name of the web site you are trying to debug.  In fact, just so you are sure that you don't accidentally deploy the dummy.php file it would be a good idea to have a seperate virtual host on the Apache server where it would reside.

On the Zend Studio side what you now need to do is set up the tunneling.  That is done by clicking on the arrow next to the tunneling button and clicking on "Servers".

Step 1

Clicking the Server button

Step 2

Selecting the server

Step 3

Enabling the tunnel

Step 4

Enabling the tunnel.

Step 5.

All done

With that, you should be golden.  Or in this case, green.

If you are using the Zend Studio toolbar you can test to actually see this working.  Simply go to Extra Stuff -> Settings and click on "Test".  You should see something similar to this.

Testing the connection

Go to your Linux server and then type in "lsof -i | grep |PORT|" replacing |PORT| with the port in the window.  In my case, 57192.

httpd     3016     apache    8u  IPv4 4100270       TCP *:57192 (LISTEN)

As you can see we have a connection setting there.  If we now do the same thing, but grep for the PID (3016) you can see our connection via our tunnel.

httpd     3016     apache    7u  IPv6 4100269       TCP li114-69.members.linode.com:http->xxxxxxxxxx:news (ESTABLISHED)
httpd     3016     apache    8u  IPv4 4100270       TCP *:57192 (LISTEN)

All done.

Sharing feedback with Twitter using Bit.ly – Part 2

In our previous installment we looked at setting our backend up so it could automatically retrieve the bit.ly URL for a given URL and store it as part of the data for a given instance of a Content model.  What we're going to do this time is take a look at the front end components.

Sometimes I find that doing things backwards can actually make things a little more clear.  That way you can see the end result and then, as you work backwards, see how all the pieces work together.

With that in mind, let's start with our view code, since that's the most important part of the whole thing.  The first thing we are going to do is define our HTML.

<div id="sliderMessage">Message is previewed before it is sent</div>
<div id="sliderContainer">
<div id="slider" style="width: 80%; margin: auto;"></div>
<div style="width: 191px; margin-right: 40px;">
<div id="customTwitterMessage">
<textarea id="twitterMessage" name="twitterMessage"></textarea>
<font size="1">(Maximum characters: 140)
You have characters left.</font>
 </div>
<div id="kudoTweetButton">
<a href="" target="_blank"><span style="color: white;">Tweet!</span></a></div>
<div class="kudos" id="slider-5" style="display: block;">Great Post!</div>
<div class="kudos" id="slider-4">Good Post</div>
<div class="kudos" id="slider-3">Decent Post</div>
<div class="kudos" id="slider-2">Didn't Like</div>
<div class="kudos" id="slider-1">Not Good</div>
</div>
</div>
<div id="sliderThanks">Thanks!</div>

There are a few elements in here.  The first is the slider with the ID of "slider".  The slider allows you to choose how high you want to rate the individual posting.  After that we have some code for writing custom Twitter messages if the review is really low.  It has the requisite 140 character limitation on it.  That is relatively simple to do, so I won't go into counting the characters.

Below that is the Tweet button. It floats to the right, so it is printed before our ratings.  After that are DIV tags that contain the individual messages.  They all have an ID that corresponds to the value of the slider and are all hidden, to start out with, except for "slider-5".  As the slider moves, each box will be displayed.

Rating the post

We have a couple of page-specific JavaScript variables that we need to have.  None of them are "required" to do this, but they are what makes it a little more automated.  All of the view script values are set in a controller.

var currentSlider = 5;
var twitterUser = "<php echo $this->twitterUser ?>";
var bitLy = "<php echo $this->content->getBitly() ?>";
$tags = array();
foreach ($this->content->getTags() as $t) {
    $t = (string) $t;
    $tags[] = '#' . preg_replace('/[W_]/', '', $t);
}
?>
var contentTags = ;
var twitterText = "";

currentSlider is the default value for the rating.  twitterUser is for if you rate a posting badly you can mention the Twitter user instead of just saying it sucked.  In other wordsd, it gives them a chance to redeem themselves.  bitLy is the variable that contains the bit.ly URL that we had before.  After that we echo out all of the tags that we have, but making them a little friendlier to Twitter but removing an non-white space and the underscore, since tags on Twitter generally don't have underscores.  It also adds the hash on the front of each tag.  They are then rendered as JSON because that's the easiest way to pass the information to the JavaScript in the view.  twitterText contains the full message that will be sent.

Speaking of twitterText we need to be able to set it.  That is done via the writeNormalTwitterMessage() function.  Is there an "abnormal" Twitter message?  Yep, but we'll look at that later.

function writeNormalTwitterMessage()
{
    $("#customTwitterMessage").hide();
    count = 0;
    twitterText = $("#slider-" + currentSlider).text() + " " + bitLy + " ";
    while (twitterText.length < 140 && count < contentTags.length

           && twitterText.length + contentTags[count].length < 140) {
        twitterText += " " + contentTags[count++];
    }
    twitterText = escape(twitterText);
}

Because this function is only called when the slider is moved, the custom message box is first hidden.  It is only used for non-kudos.  Then it takes the value of the currently selected DIV element and starts the string with that value, appending the bit.ly value to the end of it.  Then it iterates over a loop, adding the tags that we had created previously until we reach the 140 character limit or run out of tags.  Then we escape that value and store it on the twitterText varialbe.

Now we have to implement the functionality in the slider so that when we slide it, it is able to actually set the message in the function we had just defined.

if (twitterUser && bitLy) {
    $("#slider").slider({
        min: 1,
        max: 5,
        value:5,
        slide: function(event, ui) {
            $("#slider-" + currentSlider).hide();
            $("#slider-" + ui.value).show();
            currentSlider = ui.value;
            if (ui.value >= 3) {
                writeNormalTwitterMessage();    
            } else {
                if (ui.value == 1 ) {
                    $("#twitterMessage").val("@" + twitterUser + " " + bitLy + " wasn't good because ");
                } else {
                    $("#twitterMessage").val("@" + twitterUser + " I didn't like " + bitLy + " because ");
                }
                setTwitterMessageLength();
                $("#customTwitterMessage").show();
            }
        }
    });
    writeNormalTwitterMessage();
}

It looks like a bunch of code, but it's not.  What we do is bind to the slider and use some JSON to configure it.  We set the min as 1, the max as 5 and the default value as 5, or fully awesome kudos.  Then, for the slide event we define our functionality.  We first hide the previous slider caption DIV and then show the new one, resetting the previous value for the new one so we can hide it when we slide it again.  Then we check the value of the slider that was passed.  If it is greater or equal to 3 then the author did a good job and all we want to do is post the kudos.  If the value is 2 or 1, we want to give the author the chance to redeem him, or herself.  So we set it to give you  a text box.

The last thing to do from this side is to actually submit the text.  However, Twitter, for very good reasons, does not allow a web page to kick off some JavaScript and post a status update.  Otherwise you'd be seeing Twitter accounts being used as a spambot the likes of which you have never seen.  You could do it via an API, but your blog post isn't so important that someone will grant your website permission to do anything for them. So, to post this to Twitter, rather than using a form, we simply present a URL to be clicked on.  And the way we present that is via this code.

$("#kudoTweetButton").mouseover(function(){
    url = "http:///home?status="+twitterText;
    $("#kudoTweetButton a").attr("href", url);
});

What this does is set the href attribute to our twitterText value so that when the user clicks on it they will be brought to the Twitter page with kudos pre-populated.  It will look something like this.

Kudos post

The Twitter user then clicks "update" and the kudo is delivered.

Try it yourself a little bit and see what you think.

Sharing feedback with Twitter using Bit.ly – Part 1

Over here there is a good article on sharing page feedback on Twitter.  I'll end up doing something similar but in a different manner.  A little while I posted an article (Do you queue?) on how you could use the Zend Server Job Queue to run individual tasks.  Well you can do the same thing here.  I've made a few changes since that article, namely that Zend_Application is passed in both to execute() and to run() so I could easily retrieve application settings.  Don't know why I didn't think of that earlier.  Oh well.

The way this is different is that I want to pre-populate the bit.ly URL for an individual page so I only have to make one call to bit.ly.  This is because I don't want to slow the performance of my web page down when someone is viewing the page.  Additionally, I don't want to have a time out when I submit a page.  Therefore I need to create a task.

class Admin_Task_Bitly extends Esc_Queue_TaskAbstract
{

    private $_contentKey;
    
    public function __construct($contentKey)
    {
        $this->_contentKey = $contentKey;
    }
    
    protected function _execute (Zend_Application $app)
    {
        $ct = new Model_DbTable_Content();
        $c = $ct->find($this->_contentKey)->current();
        if (!$c) return;
        // Don't need to do it again
        if ($c->getBitly()) return;
        $options = $app->getOption('bitly');
        $url = 'http://api.bit.ly/shorten?version=2.0.1&longUrl='
               . urlencode(
                       $options['url']
                    . $c->getPageId())
               . '&login='
               . $options['login']
               . '&apiKey='
               . $options['key']
               . '&format=json';
        $results = json_decode(
            file_get_contents($url),
            true
        );
        if (!$results['errorCode']) {
            $res = array_shift($results['results']);
            $c->setBitly($res['shortUrl']);
            $c->save();
        }
    }
}

What this code does is take the primary key of the content being shortened and stores it for serialization when being passed to the Job Queue.  Then when the Job Queue executes it will get a new instance of the object, check to see if it already has a bit.ly URL and then build the URL, sending it to the bit.ly API servers.  When it gets the response back it checks to see if it's an error and if not, saves the result to the database for later usage.

But we're not quite there yet.  We still have some application.ini settings to set.

bitly.login = "xxxxxxxxxx"
bitly.url = http://www.eschrade.com/page/
bitly.key = "xxxxxxxxxxxxxx"

… not that some of those are hard to find out.  And then we need to add the task to the code that saves the article after I have finished working on it.

$mod->save();
$task = new Admin_Task_Bitly($mod->getId());
$task->execute($this->getInvokeArg('bootstrap')->getApplication());

That's it, from the functional perspective.  All that's left to do now is test it.

class Admin_Task_BitlyTest extends PHPUnit_Framework_TestCase
{
 
    public function testGetBitly()
    {
        $ct  = new Model_DbTable_Content();
        $c = $ct->fetchAll()->current();
        /* @var $c Model_Content */
        $id = $c->getId();
        $c->setBitly('');
        $c->save();
        
        $task = new Admin_Task_Bitly($c->getId());
        $task->run(Zend_Registry::get('Zend_Application'));
        
        $c = $ct->find($id)->current();
        $this->assertNotEquals(
            '',
            $c->getBitly()
        );
        
    }
}

And we're done.  Except for the actual part about posting to Twitter.  But we'll do that in a bit.

Do you queue? Introduction to the Zend Server PHP Job Queue

There has been a lot of talk over the past several years about the difference between performance and scalability.  Never mind that the difference between the two will probably not really affect most developers.  Never mind that the “difference between performance and scalability” argument is often used when someone’s code performs poorly and their best argument is “Yeah, but my code scales”.  Yeah, sure it does.

But when talking about building a scalable application there is a big concept out there that many PHP developers are not overly familiar with.  That concept is queuing.  It is becoming much more prevalent in PHP-land but the concept of a queue is still relatively unused among PHP developers.

So, what is a queue?  Basically it means “take this and do something later”.  “This” could be anything, from a certain point of view (requisite Star Wars reference).  What that means is that “something” can be offloaded somewhere else (a queue) for further processing.  A queue is generally not an endpoint, but a conduit.  A pipe (requisite political reference).  But it is a pipe with a flow-control valve on it (requisite plumbing reference).  In other words the “something” will stay in the pipe until a) someone gets it, or b) it expires.  Hopefully, a.

This “something” is sometimes data and sometimes it is functionality.  There are a lot of data queues out there and the nice thing about data queues is that they are pretty much language independent.  In other words you can connect to a Java-based data queue from a PHP-based application and as long as you agree upon the format, like Stomp or JMS (if using the Java Bridge) then you can pass data back and forth without much problem.

However, there can be a problem when it comes to queueing functionality.  You clearly are not language independent.  Not that it’s a problem, but you’re not.  What this means is that now you have to have a specific method for implementing the queueing functionality.  There are a couple of open source options available, Gearman for one, but not many.  What I’d like to do is provide an example using the Job Queue in Zend Server 5.

Queueing a job is actually very easy to do.  A job is run by calling a URL where that job resides.  The Job Queue daemon will receive the request from your application and will then call the URL that you specified in the API call.  Once you call that URL your application can continue going on its merry way to finish serving up the request.

Serving the request

On your front end machine, the code to call the queue is pretty simple.  It consists of creating a ZendJobQueue object and calling the createHttpJob() method.  If you have any parameters that you need to pass to that job you can specify them in the second parameter of the call

 

1
2
3
4
5
6
$q->createHttpJob(
    'http://localhost/sendemail',
    array(
        'email' => $_POST['email']
    )
);

Then on the “sendemail” side your code would be

 

1
2
3
4
5
$params = ZendJobQueue::getCurrentJobParams();
if (isset($params['email'])) {
    mail($params['email'], 'Welcome', 'Welcome to my nightmare');
    ZendJobQueue::setCurrentJobStatus(ZendJobQueue::OK); 
}

That’s really all there is to it.

Or is there…

Serving the request… cool-y

My problem with this method is that it really is not as structured as I would like.  Modern applications are not really “scripts” even if they are written in a scripting language.  So, what I like doing is taking this existing functionality and providing some structure.  What I did for this website is take the existing Job Queue functionality and added something kinda similar to Java’s RMI.  It’s not quite, but kinda.  Or kinda like threading.  Not really, but kinda.

What I start out with is a generic abstract task class.  It looks like this.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
abstract class Esc_Queue_TaskAbstract
{
    const OPT_NAME = 'name';
    const OPT_SCHEDULE = 'schedule';
    const OPT_SCHEDULE_TIME = 'schedule_time';
 
    private $_options = array();
 
    protected abstract function _execute();
 
    public final function execute(Zend_Application $app, $qOptions = array())
    {
        $q = new ZendJobQueue();
        $jqOpts = $app->getOption('jobqueue');
        $qOptions = array_merge(
            array('name' => get_class($this)),
            $qOptions
        );
 
        $ret = $q->createHttpJob(
            $jqOpts['url'],
            array(
                'obj' => base64_encode(serialize($this))
            ),
            $qOptions
        );
        return $ret;
    }
 
    public final function run()
    {
        $this->_execute();
    }   
}

There are two defined methods and one abstract method.  The two defined methods are final because they need not and should not be overridden for the sake of predicability (final is under-used IMHO).  The execute() function doesn’t really execute anything.  It just takes the current class, serializes it and base64 encodes it, because the params don’t like binary data and sets it as a parameter called “obj”.  From there it inserts it into the Job Queue which is specified by a Zend_Application configuration setting.  That setting is

 

1
jobqueue.url = http://localhost/jq

Since queues generally contain privileged information it is a good idea to hide it from the outside world either on another machine/VM or web server directive.

The second method is called run().  It is not called on the front end machine.  The back end Job Queue will call that to execute the functionality that is defined in this class in the abstract method, called _execute().

So that’s the abstract class that our tasks are based off of, but how about an individual task?  What does that look like.  Well, to take our code that we had previously written…

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class Admin_Task_Mail extends Esc_Queue_TaskAbstract
{
 
    private $_email;
    private $_message;
    private $_subject;
 
    public function __construct($email, $subject, $message)
    {
        $this->_email     = $email;
        $this->_subject   = $subject;
        $this->_message   = $message;
    }
 
    public function _execute()
    {
        mail(
            $this->_email,
            $this->_subject,
            $this->_message
        );
    }
}

I put this code into my /application/modules/admin/tasks directory and added the following line to my bootstrap.

 

1
$al->addResourceType('task', 'tasks', 'Task');

That way the Zend_Application autoloader can easily autoload any tasks I have defined.

To execute this task, in my controllers, I simply type.

 

1
2
3
4
5
6
7
8
9
$mail = new Admin_Task_Mail(
    $_POST['to'],
    $_POST['subject'],
    $_POST['message']
);
 
$mail->execute(
    $this->getInvokeArg('bootstrap')->getApplication()
);

This will then send the job to the Job Queue daemon.

Speaking of.  We need to now execute our job.  That is done by defining a controller with code similar to the following.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
$params = ZendJobQueue::getCurrentJobParams();
if (isset($params['obj'])) {
    $obj = unserialize(base64_decode($params['obj']));
    if ($obj instanceof Esc_Queue_TaskAbstract) {
        try {
            $obj->run();
            ZendJobQueue::setCurrentJobStatus(ZendJobQueue::OK);
            exit;
        } catch (Exception $e) {}
    }
}
ZendJobQueue::setCurrentJobStatus(ZendJobQueue::FAILED);
exit;

It retrieves the parameters and checks for one called “obj”.  It then unserializes the base64 decoded data, which should recreate the object that you created on the front end server.  After testing to make sure that it is an instance of Esc_Queue_TaskAbstract we call the run() method, which in turn calls the actual functionality we defined in _execute().

Sweet.

Summary

Key points on building super-cool job queue applications

  1. Create an abstract class to wrap around your tasks
  2. Use that abstract class to add itself to the Job Queue
  3. Write a controller script that is the queue endpoint
  4. Have that script recreate the object and execute the method you had defined in the code

Server Monitoring, Studio Tasks and Gmail?

I deployed this blog about a week ago.  Since then there has been little traffic, mostly  because I have not promoted it and I have also been working on a bunch of other content and such.  Since I also just needed to just get it out the door and running I have not spent a lot of time testing for performance.  This blogging app is based off of Zend_Application and as such there is a bit of overhead.  Since I was not expecting traffic to be that heavy I was not (and am not) overly concerned about snappy performance at this point.  As long as the page loads and displays in the browser I’m fine with it.

One of my goals with this blog is to show real life examples of how to use Zend Server.  While I am not overly concerned about the performance of this blog for the time being I am still monitoring it to make sure that it performs well.  That means Monitoring.

However, Monitoring is really not all that beneficial if you don’t know what is happening on your server.  To get around that issue, Monitoring has a feature where you can send an email to one or more email addresses when an event occurs, such as slow page load times or a fatal error.  This is great… except that people ignore emails.  We’re conditioned to.  As great as Spam in a can is (I’m not Hawaiian, but Spam is a delicacy IMHO), Spam in the Inbox has conditioned us to ignore the things we’re not interested in.  And, to be honest, we’re not interested in fixing problems.  We’d rather be doing new and cool things.

Task driven development is part of the Agile process.  Test driven development is too, but I’m going to talk about Task driven development.  What that basically means is that if there is something to be done, a task is created and it is worked on by the next available developer.  The benefit of this is that with short, well defined tasks, it is easy to track the progress that is being made on a project.  That is why task driven development is part of Zend Studio.

“What?  I thought you were talking about Monitoring?”  Yep.  Still am.  What I am going to show you is how you can integrate Monitoring with Zend Studio using Gmail to make you a better developer.

First, we need to set up our Gmail account.  Log in with your email address and go to Gmail’s “Settings” -> “Labels” section and create a new label called “Task”.

Create Label

Next, click on “Filters”.  Create a new filter and enter a “From” address; [email protected] in my case.  The actual email address is relatively unimportant, and doesn’t really need to exist, as mine doesn't.  What is important is that you can match it up with the “From” address that Monitoring will use, which we will set up later on.  Click on “Next Step” and tell it to apply the label “Task” to any message coming in from that email address.

Filter Step 1

Filter Step 2

Zend Studio has the ability to connect in with Mylyn task connectors.  To access this, go to the Planning perspective in Zend Studio.  On the right hand side you should see a View window that says “Task Repository”.  From there you can right click on the screen and select “Add Task Repository”.  You will be asked to choose a specific type of task repository.  I will use the Gmail type.  If you don’t have the Gmail connector installed click on “Install More Connectors” and install it from there.  You may need to restart Studio (“may” meaning you will need to).  You may (same thing) need to create a TaskTop account.  It seems to be free.

Add Repository

Once you’re back up, go back to the same view and get to the “Add Task Repository” window again.  This time select Gmail as the repository type.  Enter in your email address and your password, but set “Task” as the label.  Click Finish.

Add Label

Then you will probably get a window popping up asking if you want to configure a query.  Of course you do.  Set the title of the query as "Task" and then choose the label that you want to query by.  This should be the new label that you had entered earlier on.

Set Query

Next we will go into the Zend Server admin GUI.  Select “Server Setup” and then “Monitor”.  From there you should see some mail server settings.  The sender email address is the one you set up for the “from” address in the Gmail filter; [email protected] in my case.  Click Save.

Monitor Mail Settings

From there go to the “Rule Management” tab and select the rule that you want to receive alerts for.  Scroll down to “Step 2: Event Action”. Enter in your Gmail email address here, save it and restart PHP.  I chose Slow Request Execution.  And since slow requests are difficult to debug I also chose to enable Code Tracing when a slow request occurs.

Monitor Action

Now you wait for an event to occur.  Me?  I just refreshed the home page.  (I may have some optimizations to do).  Go back to the Planning perspective and right click on the task list and click “Synchronize”.  You should see a new task listed.  Double click on it and you should see something like the following.

Task Received

Now, rather than using your email program and ignoring messages that come in, you can have them directly integrated into a simple task management system.

Summary

  1. Create Label in Gmail called "Task" with a filter for the "From" address, set up in Zend Server
  2. Add Mylyn connector for Gmail and set up to use your Gmail account with the label "Task"
  3. Set up Zend Server Monitoring to send monitoring events from the "From" address
  4. For a specific event set it to send an email to your Gmail account
  5. Wait for events to come in.
  6. Fix them

Introduction to Caffeinated PHP – Using Java to extend PHP

If you are reading that title and thinking "what is that guy smoking?" then you are probably in good company.  Developers often make development decisions based off of philosophy.  Some developers go the route of "we will decide on the best tool (language) for this individual job" and end up having hundreds of individual tools that they need to end up supporting.  Others desire to take a more puritanical approach in saying that "we will only use this one tool (language) for this individual job."  The problem is that the decision is often not as cut and dry as many development philosophies allow.

In my experience, having a homogeneous environment is most desireable.  Limiting the number of languages and tools that you need to support is often a good thing.  The more tools you need to support, the less likely you are to be an expert in any of them.  Jobs whose posting states that the developer needs to know Java, PHP, ASP, Perl and C++ will likely end up getting someone who is not only decent in all of them, but lacking a depth of knowledge in all of them.  It's kind of like a job posting I saw for a C# developer that required 2 years of C# experience after C# had been out for 6 months.  I am mostly convinced that "using the best tool for the job" is the wrong approach.  What the better approach is "what is the best tool for solving the organization's problems".  Is PHP the best choice for a daemon?  Probably not.  But does your organization run PHP for your web properties and have minimal needs for some daemonized services?  Running PHP as a daemon may be a viable choice.  It's not "the best" choice for the job, but it might be the best choice for your organization.

But there are times when you have two disparate ecosystems that need to work together.  PHP is often the top contender for building out a web-based interface, but many organizations have internal systems that are running Java.  So do you have to squash your dreams of building your website in PHP because you need to have Java connectivity?  By no means!!

As part of Zend Server ( and CE) there is a feature called the Java Bridge.  Zend Server CE is our free version of Zend Server.  While Zend Server does have several useful "for-pay" features there are also several additional features that are shared between Zend Server and CE.  The Java Bridge is one of those features.  This means that you get the Java Bridge for free.

Using the Java Bridge is really quite simple.  First you need to log in to your instance of Zend Server and make sure that you have the Java Bridge installed.  You can see that we have it installed on the image on the right.  This means that both the Java Bridge extension and the Java Bridge daemon are up and running.  The daemon is basically a JVM that is running in the background and servicing requests made by the Java Bridge extension running in PHP.

If the Java Bridge has not been installed you can use the native installer.  In this case we are using yum.

[root@localhost ~]# yum search java-bridge
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.san.fastserv.com
 * extras: mirror.skiplink.com
 * updates: mirrors.igsobe.com
============================= Matched: java-bridge =============================
php-5.2-java-bridge-zend-server.i386 : Zend Java bridge
php-5.3-java-bridge-zend-server.i386 : Zend Java bridge

We have the name, now we need to install it.

[root@localhost ~]# yum install php-5.3-java-bridge-zend-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.san.fastserv.com
 * extras: mirror.skiplink.com
 * updates: mirrors.igsobe.com
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package php-5.3-java-bridge-zend-server.i386 0:3.1.26-2 set to be updated
--> Processing Dependency: java-daemon-zend-server for package: php-5.3-java-bridge-zend-server
--> Running transaction check
---> Package java-daemon-zend-server.noarch 0:3.1.25-15 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                          Arch    Version      Repository          Size
================================================================================
Installing:
 php-5.3-java-bridge-zend-server  i386    3.1.26-2     ZendServer          21 k
Installing for dependencies:
 java-daemon-zend-server          noarch  3.1.25-15    ZendServer-noarch   30 k

Transaction Summary
================================================================================
Install      2 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 51 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): php-5.3-java-bridge-zend-server-3.1.26-2.i386.rpm |  21 kB     00:00     
(2/2): java-daemon-zend-server-3.1.25-15.noarch.rpm      |  30 kB     00:00     
--------------------------------------------------------------------------------
Total                                            96 kB/s |  51 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : java-daemon-zend-server                                  1/2
To enable Java bridge, please run /usr/local/zend/bin/setup_jb.sh
  Installing     : php-5.3-java-bridge-zend-server                          2/2

Installed:
  php-5.3-java-bridge-zend-server.i386 0:3.1.26-2                               

Dependency Installed:
  java-daemon-zend-server.noarch 0:3.1.25-15                                    

Complete!

From there we need to set up the Java Bridge daemon.  Because there is an external dependency, namely Java, we need to do some basic setup on the Java Bridge.  Java may not be installed so you will need to make sure that you have that available.  If you have Java already installed you can run the zendctl.sh command, which will set up the Java Bridge daemon.  I have bolded my type.

[root@localhost bin]# /usr/local/zend/bin/zendctl.sh setup-jb
Welcome to Java bridge setup!
Please specify path the Java executable :
NOTE: please use Java by Sun or IBM version 1.5 and above.
/usr/bin/java
Starting Java bridge [OK]
[ 01.03.2010 16:23:58 SYSTEM] watchdog for java_daemon is not running.
[ 01.03.2010 16:23:58 SYSTEM] java_daemon is not running.
Stopping httpd:                                                      [  OK  ]
Starting httpd:                                                      [  OK  ]

At that point we're pretty much ready to go.  The Java Bridge works by initiating a TCP connection between the Apache process and the JVM.  The Java Bridge extension then sends command such as 'create object "System"' or 'run method "getProperties" on object "42fse2@System"'.  The Java Bridge then executes the request and returns the return value of that individual method call.  That return value can either be a regular Java object, a primitive or one of a few types that can be converted into native PHP types.  If you want to test your installation to make sure that it is running you can do so with the following code:

echo '

';

	$java = new java('java.lang.System');

	var_dump($java->getProperties());

If all is well and good you should get output similar to this:


array(54) {
  ["java.runtime.name"]=>
  string(28) "OpenJDK  Runtime Environment"
  ["sun.boot.library.path"]=>
  string(52) "/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386"
  ["java.vm.version"]=>
  string(9) "1.6.0-b09"
  ["java.vm.vendor"]=>
  string(21) "Sun Microsystems Inc."
  ["java.vendor.url"]=>
  string(20) "http://java.sun.com/"
  ["path.separator"]=>
  string(1) ":"
  ["java.vm.name"]=>
  string(17) "OpenJDK Client VM"
  ["file.encoding.pkg"]=>
  string(6) "sun.io"
...

That's all for today.  We will look at some more practical examples in our next look at the Java Bridge.

Installing Zend Server 5 on Linux in 4 minutes and 37 seconds

Web Analytics