Zend Server 5.6 is out

6 Comments

Zend Server 5.6 is out.  I’m actually pretty stoked about this release.  Here’s a few reasons why.

  1. Zend Server CE now contains the full Zend Server stack.  Why do I find this exciting?  Because the upgrade path from CE to the full version is stupid easy.  That means you can try out all of the features in Zend Server and if you don’t want them, just let the license expire.  I call it a gateway drug.  Seriously.  Get Studio or PDT and try the deployment feature.  If you’re not using it you’ll ask yourself “why am I not doing this?”
  2. All of you Mac users can finally stop bitching and complaining about the lack of Mac support.  We don’t (and probably won’t) support Mac for production (does anyone besides Apple actually use their server products in production?) but for dev work you now get the full stack on your Mac.  This is not GA yet but will be later on.  If you want a preview release you can go to http://forums.zend.com/viewtopic.php?f=8&t=26873.
  3. Fault tolerance in the Job Queue through MySQL clustering.  In clustered environments job queue information has moved from local storage to remote MySQL storage.  So you can have a single MySQL server sitting there, a clustered setup or RDS or some other cloud-based DB, it doesn’t matter.  If it can be accessed using the MySQL drivers then it can be used by the new Job Queue.

Creating deployment packages with Zend Studio 9

Leave a comment

Deploying your work in Zend Studio is quite easy.  You have the ability to deploy directly from your IDE, which is, in general, more for local environments, the Zend Developer Cloud or testing.  The reason for this is because generally the developer should not have access to production.  Therefore, the drag and drop deployment that Zend Studio supports is usually not going to be used, though in smaller development shops it may still be quite useful.

The deployment mechanism in Zend Server uses a ZPK file, which contains the source code, assets and the deployment descriptor.  The deployment mechanism on the server side has already been discussed on this blog.

When creating a deployment package the first you need to do is either create a new project with deployment support

or add it by right clicking on your project and selecting “Add Application Deployment Support”

This will allow you to then either drag the project onto a target to deploy it or, as we’re talking about here, allow you to export it as we’re going to show here.

The deployment.xml file is where all of the configuration information resides.  Double click on this file and you will get a display similar to this

Most of the fields are self explanatory but a few might require a few words.

  • Document Root – This is the document root (duh) which is relative to the base directory of the deployment.  Your project, in other words.
  • License – This is the relative path to a text file in the project directory structure that contains the EULA for the project.  During the deployment workflow in the UI the end user will be presented with this if the file is available and will be required to agree to it before proceeding.
  • Persistent Resources – These are items that you don’t want to have overwritten during an upgrade.  For example, cache directories.

Dependencies

There are several different types of dependencies you can specify for your application.
These dependencies will be checked prior to deploying the application.  If they are not satisfied then the application will not deploy.

Triggers

There are several triggers that can be hooked into during the deployment process, each of which has a Pre and Post stage
  • Activate
  • Deactivate
  • Stage
  • Unstage
To set up a trigger simply double click on the stage that you would like to edit and a new file will be created for you.  In that file will be documentation on information on how to retrieve variables and parameters for your deployment scripts.
Speaking of variables and parameters, what is the difference?  There are two differences.
  1. Variables you cannot change during the deployment process.  What the value is in the deployment file is the value that you will get in the deployment script.  Parameters need to be specified during the deployment workflow and also have some validation that you can do on the entered values whereas with variables you do not.
  2. Both are accessible via getenv() during deployment but variables are retrieved with their names “as is” but parameters are upper cased and prepended with “ZS_”.  So if you have a parameter named “ugly_Duckling” it would be accessed via getenv(‘ZS_UGLY_DUCKLING’)

Package

There may be files in your application that you want to include (really!?) or exclude.  You can specify those in the Package panel.

Exporting

The last step is to export your project.  Right click on the project and select Export and choose “Deployment Package”.  This will output the project into a ZPK file that you can then upload to your Zend Server instance or Zend Application Fabric installation where it will be deployed to your website.  Lickety Split.

Conclusion

There are other panels there and other information that I have not included.  The reason for that is that it is either relatively self explanatory on not necessary from an initial “getting started” point of view.  Or because it’s Friday and I’m running out of time.  🙂
The deployment functionality in Zend Server was something that I have personally been waiting for for a long time.  This addition in Zend Studio now makes the circle complete.  Granted, there will always be more that we can do, but this was a feature that has long been needed and I’m quite glad that it’s here.

Enter Deployment: Zend Server 5.5

1 Comment

Today Zend Server 5.5 was released.  This is actually a pretty good sized deal.  The big addition to the Zend Server feature list is deployment.  Now, you might be saying “come on, Kevin, there are many PHP deployment solutions already”.  To which I would say “yes, there are.  But there are NO solutions that work out of the box“.  Ladies and gentlemen, welcome to the box.  If you have Zend Server 5.5, you have deployment.

So what I’m going to do with this post is give you a quick run-through on getting a simple application up and running.  There are a lot of features I am going to skip over (primarily deployment scripts) and show you a simple workflow so you can see how quickly and easily an application can be deployed.

Step 1: Install The Zend SDK

First of all, you need the Zend SDK (and, of course, Zend Server) and set up the tools directory to be in your path.  Actually, you don’t need the SDK since there is a GUI interface.  But I’m showing you a simple workflow that will get your application running very quickly.  We’ll get into more details in a different blog post that demonstrates the full GUI interface.

Step 2: Create an API Key

In the Zend Server UI you need to name and create a new API key so that the SDK can communicate with your instance of Zend Server.

If you click on the “Show full key” link you will get a popup of the full key so you can copy it into your terminal.  The name of the key is also important.

Step 3: Connect the SDK to Zend Server

Once you have your API key you can then associate it with your local SDK instance.  You will need the API key that you generated earlier along with the name of that key.  Provide the hostname and you’re good to go.

D:\workspace>zend add target -s 8a29cd9333ca366af986df69c32b44ed84aff6af0e741209a82cf4a444e64187
   -k deployment -h http://192.168.0.248
 
Target http://192.168.0.248 was added successfully, with id 0

The ID number is important if you are going to have multiple Zend Server clusters to talk to, such as staging and production.  Then you will need to add the -t argument with the target ID.  However, if you have only one installation then the SDK will automatically use that target.

Step 4: Create a deployment scenario for your PHP project

Once your target is set up you need to provide a deployment descriptor file for your project.  For my example project I’m just going to use a blank Hello World application with a /public application/document root.

D:\workspace>zend create project -t simple -n HelloWorld
 
Project resources were created successfully for HelloWorld under D:\workspace\HelloWorld

The -t option is the template of the application.  You have simple, quickstart and zend.  Because my application already existed I specified that it was a simple application so no additional files would be created.  If you are creating a new project from scratch you can quite easily use one of those templates.  If you have an existing application, using the simple template will give you a good starting point.

The deployment.xml file for my application looks something like this (thanks WordPress for not allowing me to post XML, even the code formatters barf on it)

package
 ---> name: HelloWorld
 ---> version
    ---> release: 1.0.0
 ---> appdir: public

Step 5: Deploy!

The next step is to deploy the application.  You can, again use the SDK to do this from the command line.

D:\workspace\HelloWorld>zend deploy application -b /
 
Application HelloWorld (id 9) is deployed to http://<default-server>/

The -b tells the SDK the base URL that we want to install the application at.  If we wanted it at /test, it would be -b /test.

If we want to remove the application we simply tell the SDK to do it.

D:\workspace\HelloWorld>zend remove application -a 9
 
Application HelloWorld (id 9) was removed from http://<default-server>/

The -a tells the SDK the application ID of the application that we want to remove.  We got the application ID from the return message when we originally deployed it.

Step 6 (optional): Rollback

Every once in a while you deploy an application that wasn’t quite as baked as you thought it was.  With the new deployment feature you can log in to your Zend Server GUI and do a one-click rollback of your application (well, two clicks once you click the “are you sure button”).

Conclusion

That is how easy it is to deploy a (very) simple application using the new deployment techniques.  This works both for Zend Server and Zend Server Cluster Manager installations, so you can deploy to a single machine or let ZSCM push your application to your entire cluster.

Later on I will go through using the Zend Server GUI to set up more complex applications requiring configuration parameters or deployment scripts and also go over the zdpack command which is the package builder that you use if you are not using command line based deployments.

Where to go from here?

It’s simple.  Go to our download site, or set up your test machine to access our yum or deb repositories.  In other words, try it out.


Deployment beta for Zend Server 5.5 – Getting Started

22 Comments

Deployment for PHP applications is one of those things where it’s tough to get a general reading on where things are.  On one hand you have people who have deployment mechanisms that seemingly automate the rotation of the earth and on the other hand you have people who jump up and down on their code trying to force it into a small hole (visions of Wile E. Coyote come to mind).  Others set up complex build procedures that has more processing on it than Cheez-Whiz (now I’m hungry for a grilled cheese sandwich).  My own thought on the matter is that for most PHP implementations a lot of the build tools out there are probably overkill for a lot of PHP applications.

Given all that, we have announced that we are making the beta for our new deployment feature in Zend Server 5.5 beta, available for download.  It’s not feature complete (there are some new features being worked on), but it provides the functionality needed for the 90% of us who do not have the need to have crazy complex deployment scenarios.

This begs the question, under what circumstances would the new deployment feature in Zend Server benefit you?

  • You simply copy files into production as part of your existing deployment mechanism using FTP (gasp), SFTP or rsync.
  • You do NOT want to maintain the infrastructure or tooling needed to deploy your application to production boxes (singly or in a cluster)

Zend Server deployment will be available for both Zend Server and Zend Server Cluster Manager meaning that you will be able to deploy applications to either a single server or to a cluster of servers with just a few clicks of the mouse.  We would like your help in looking at what works and what doesn’t.  Not just in the “what breaks” sense, but also in terms of the workflow and also the integration points, both in the Zend Server UI and also using the deployment API.

We would like to see innovative ways that you could use it, but it would probably help for you to see a quick example of how to do it to get you up and running quickly.  For that I have a short video that how you can utilize the deployment feature .

With that, let’s take a look at some of the things that you need to work with the deployment beta.

The first thing you need to be aware of is a program called zdpack.  This program helps to create the package to deploy to your Zend Server instance or cluster.  You can easily script it into an existing packaging routine or your can run it manually.  How you get it integrated into your own environment is up to you, but here are the steps that I used for my own testing.  These are not “best practices” necessarily.  There are variations in how you can do this, but if you want to start with a clean project (probably the easiest way to get familiar with it) this is a good way of doing it.

Step 0: Download the Zend Server 5.5 beta

Windows or Linux

Step 1: Create a Zend Framework project

The reason I chose Zend Framework instead of a PHP project was because a Zend Framework project (in Zend Studio) has an internal document root folder called “public” where the public files are stored and the source code of the application, configuration files and such are stored outside of the document root.  Those are some of the reasons why I prefer using a ZF application for this.  The other reason is that when we build out the deployment skeleton, having the document root one directory lower allows you to write your deployment hooks easily in the same project as well as restricting access to the deployment config file.

Step 2: Create a deployment skeleton

Open up a CLI and go to the main workspace and type

1
zdpack create ProjectName

replacing “ProjectName” with the name of your project.  This will create a few files and directories in your project.  You can do this outside of a project, or anywhere on your computer for that matter.  But this just makes it easy to get started.  Your project should look something like the image on the right.  The red highlighted areas are files that the zdpack application created.

In the root directory of your application it will have created the directories “data” and “scripts”.  Inside of scripts you will see templates for individual files that allow you to run deployment commands at various points along the deployment process.  One that I created was a hook that would send an email each time the application was deployed.  I put the following code into the post_activate.php

1
2
3
4
5
6
7
8
9
10
11
12
require_once 'Zend/Mail.php';
$mail = new Zend_Mail();
$mail->addTo([email protected]');
$mail->setSubject('New deployment');
$currentAppVersion = getenv('ZS_CURRENT_APP_VERSION');
$mail->setBodyText(
"
New application has been deployed.
Version: {$currentAppVersion}
"
);
$mail->send();

The most important line is the $currentAppVersion.  Any data or meta data that you want will want will be made available via environment variables.

Step 3: Modify the deployment.xml file

The deployment.xml file in the main directory contains the meta information that the deployer needs to get your application up and running.  Here is a screenshot of what that might look like.  Most the elements are self descriptive and so I won’t spend much time talking about them, but there are a few things I would like to highlight.

First of all, the highlighted elements on the bottom-right side of the page.  This is the XML editor in Zend Studio and as you can see you are limited to the names of the nodes that you are able to enter into the deployment file at various node positions due to the validation that occurs.  This actually makes it easy for you to create a deployment file in a simple XML editor.

The second (more important) thing I want to direct your attention to are the three nodes on the left.  The first is the “eula” node.  If you have an end user license agreement you put the location of it there.  The second one is “appdir”.  When you first create the deployment.xml file the directory “data” will be in there, which is where the deployer is expecting the base of the application to be.  But if you are following this example then the base of the project is technically the base of your application.  That is why I put the value “public” in the “docroot” node.  These are all relative paths to the base of the application directory.

Step 4: Create the deployment package

Going back to your CLI in the main workspace directory type in

zdpack pack <ProjectName>

This will create what is basically a zip file with an expected file structure. The file is called a zpk, but it’s really just a zip file.

Step 5: Deploy

Log in to the Zend Server GUI and click on the new “Applications” tab.  On the right hand side of the screen you will see a button that says “Deploy Application”.  Click on that button and follow the instructions.  When you get to the end of the wizard click “Done” and your application will start to deploy.

That’s it.

Where to go from here

Try it.  Go to the download page and install it.

Check to see if there are better ways of integrating it with your application than what I have noted here.

Go to the forum.  Report bugs.  Download sample deployment applications.

As I said earlier there are more features coming in the near future but there is definitely enough for you to get cracking on it.  I’m actually quite happy to see this new feature being added.  Yes there are definitely alternate ways of deploying your applications.  But in the experimentation I have done I have found that this is a nice, simple, easy to use method for deploying your PHP based applications.


Details on building a Flex-based dashboard with a PHP backend

Leave a comment

Yesterday (June 1st) I wrote a blog post on how to set up an example application that I did for a webinar with Adobe and building a Flash based dashboard application (I would suggest watching it) connecting into to PHP.  Today I would like to provide some of the details of how I did all that.  I was a bit pressed for time while building the example app, and I did my standard thing, taking the Mythbusters motto to it’s logical conclusion.  But in the end I think it worked out well.  The example is complicated but a dashboard is intended to aggregate data from multiple different places and so while it was complex I also think that the attendees were able to see how and why data was processed as it was.

The dashboard application was a Flex-based app that could get summary information and batched information from a Magento based e-commerce site.  It would display a traffic summary, slightly delayed sales information (which products were selling best) and realtime notification of sales that occurred.

The way it worked was that all of the information that was going to be displayed in the dashboard was sent into one of three message queues.  That’s right.  The PHP application did not use a database. Well, it did, later on, but the actual registering of data was not done in the standard PHP way of doing a database insert.

Heresy!  No, actually I think there are a lot of problems that PHP developers solve with SQL that should be solved with message queues.  In this case, there were three things that I wanted.  I wanted traffic information, notification of a sale and the details of that sale.  But I really didn’t need to keep that information once it was processed and I didn’t care about ACID compliance or anything like that.  I basically wanted to throw data somewhere where I could get it later.  That’s where the message queue came in.

I chose ActiveMQ as my message queue.  I tried RabbitMQ but I ran into authentication issues that I didn’t have time to solve.  It was quicker to install ActiveMQ.  ActiveMQ has a feature that it shares with RabbitMQ and that is the Stomp interface.  Most message queues have some kind of binary protocol that they use to initiate communication.  But Stomp is a text based protocol.  What that means is that it is a) easy to build adapters for, and b) very easy to hack if you need one built for you.

Zend Framework has a Stomp interface, which requires some messing around with, but it also has an ActiveMQ interface that uses Stomp.  What that means is that it uses the Stomp interface but makes sure that it’s working with ActiveMQ with any “special” things that needs to be done.  ActionScript, the Flash-based programming language, also has a Stomp based interface available via a third party.  What this meant is that I could connect to a message queue and both the front end and the back end were able to interact with it.

What I wanted to do was send data into the queue that I was either going to a) retrieve directly using the dashboard, or b) pass to a PHP job for batch processing.  Messages are simple unstructured text.  A) was simple because I was just passing a notification.  B) was more problematic because I wanted to pass structured data, i.e. a log object.  But wait!  A serialized object is unstructured text!  So passing the object was no problem.  To do this I created a logger class that understood a few different classes.  We’ll start with those.

First the LogItem class

1
2
3
4
5
6
7
8
namespace log;
 
class LogItem
{
public $title;
public $url;
public $timestamp;
}

Now let’s take a look at the Logger class. (I am omitting the sales portion)

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
namespace log;
 
class Logger
{
 
private static $queues = array();
 
public function getQueue($queue)
{
    if (!isset(self::$queues[$queue])) {
        $config = new \Zend_Config_Ini(__DIR__ . '/../../etc/config.ini');
        if (!isset($config->mq->$queue)) {
            throw new Exception('Undefined queue configuration');
        }
 
        $stomp = new \Zend_Queue_Adapter_Activemq(array());
        $stomp->setQueue(
            new \Zend_Queue(
                array(
                   'name'=> $config->mq->$queue->endpoint
                )
            )
        );
 
        self::$queues[$queue] = $stomp;
 
    }
    return self::$queues[$queue];
}
 
public function log($queue, LogItem $log)
{
    $this->getQueue($queue)->send(serialize($log));
}
}

 

This class works in conjunction with a config file like this

1
2
3
4
mq.traffic.endpoint = /queue/traffic
mq.traffic.scheme = tcp
mq.traffic.host = localhost
mq.traffic.port = 61613

The Logger object receives a request to log a LogItem object on a specific queue.  It then calls getQueue() which looks to see if a configuration exists for that queue.  If it does it creates a new ActiveMQ adapter object, sets its queue name and returns it.  Then the log() method serializes the $log object and passes it to the send() method of the ActiveMQ.

At this point the data is in the message queue and you’re done, right?

Nope.  Now you need to pull it out of the queue.  On the PHP side it is pretty much the same as queuing it.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$stomp = new \Zend_Queue_Adapter_Activemq(array());
$stomp->setQueue(
  new \Zend_Queue(
    array(
      'name'=> $config->mq->$queue->endpoint
    )
  )
);
 
while (true) {
  $msgs = $stomp->receive();
 
  foreach ($msgs as $msg) {
    /* @var $msg Zend_Queue_Message */
    $logItem = unserialize($msg->body);
    // Do your thing
  }
}

 

In here we create our connection to the queue daemon and state the queue name.  Then, in this loop, we state that we want to receive data from the queue by calling the receive() method.  Generally a message queue will allow a connection to sit idle indefinitely until there is data on the queue.  But for PHP, since most are not long running programs, you will want to have a timeout.  Leaving the first argument null on the receive() call will set the timeout to be the default, but you probably should set it relatively low.  The receive() method will return an object of Zend_Queue_Message_Iterator which implements Iterator which means it can be used in a foreach loop.  Each item in the object returned will be an instance of Zend_Queue_Message which contains your data along with some meta data.  I didn’t care about the meta data so all I did was unserialized the body of the message which then gives the instance of the object that I had passed in to the queue earlier.

That covers about 1/8th of what I still have left to talk about concerning the webinar I did with Adobe.  But one of the more interesting things I did was use a message queue to communicate between a front end PHP application, a back end PHP application and a front end Flex application (which I didn’t get to).  I personally think that PHP developers should spend a little time looking at asynchronous communication and examining whether their applications could use it.  Perhaps this will help you get on your way.


Building a Flex-based dashboard with a PHP backend

Leave a comment

Today (June 1st) I got to give an online talk with Adobe on how to create a stunning analytics dashboard with Flex and PHP (recording here).  I’m a horrible graphic designer and so I don’t know how stunning it was, but from the technical perspective we covered a lot and it was a lot of fun to write.  Several of the attendees asked for code and so here is what I’ve written.  Like the webinar, it will take a little bit of time to get it working.  But the end result is that you’ll get a whole lot of stuff to think through.

What I did was take a Magento installation and create an observer that would send data to a message queue where it that data could be processed for either summary information or real-time notifications.  The dashboard contained graphs for traffic and product sales summaries and a notification list that would show new purchases as they were made.

To do this I used a lot of different programs, which you will need if you want to try it out.  You will need

  1. Magento
  2. ActiveMQ
  3. Zend Server (30 day trail available)
  4. MySQL
  5. Flash Builder for PHP (60 day trial available)

AS3Stomp is also used but I have included it in the workspace download.

Additionally, you will need to have two host names available on your machine.  I named mine magento.local and charting.local.  I set them in my hosts file (/etc/hosts or c:\windows\blah\blah\blah\hosts) but you could also use DNS as well if you needed to.  If you chose to run off of different host names you will need to change the gateway URL for the Charting project by editing the .flexProperties file and changing the serverRootURL property.

The first thing you need to do is download and install Zend Server.  You will need a license key to activate the Job Queue service.  Additionally you will need to add the directory in the project ChartingAPI, /library to your PHP include_path setting so that the logging and library classes can be included in your scripts.

You will then need to download and install Magento and then create a list of products.  Two or three should be enough.  If you want to easily integrate it with my code I would suggest that you create a blank PHP project and import the downloaded contents into the new project so that the project root is the document root for magento.local.  Do this before importing any of my project code.

Next, download and install ActiveMQ.  You will need to create three queues; saledata, sales and traffic.  You will also need to turn on Stomp support which you do by editing the activemq.xml file and adding the transport connector.

<transportConnectors>
<transportConnector name=”openwire” uri=”tcp://0.0.0.0:61616″/>
<transportConnector name=”stomp” uri=”stomp://localhost:61613″/>
</transportConnectors>

You may need to specify the JAVA_HOME directory in activemq.bat in the bin directory.

From there you can import my Magento extension into the Magento project.  Make sure that you import it into the root so that the code ends up in /app/code/local/Zend.  Additionally there should be a file called /app/etc/modules/Zend_Dashboard.xml.  This is there to inform Magento that you have a new extension that you want to use.  Once you have the code in place you will need to 1) Flush the Magento cache and 2) Enable the extension in the Magento settings (hint, it’s in one of the last sections).  With Magento, if you make a change and nothing seems to happen, it’s probably because the cache was not flushed.

To test the messaging configuration go to the ActiveMQ management page at http://localhost:8161.  Browse around the Magento site a little and check back to the queues page.  If you see messages waiting then your configuration is working.

Next up is to set up the  job queue.  In the ChartingAPI project there is a directory called public.  That should be the document root for the charting.local virtual host.  In the public directory is another folder called jobs.  In there are two scripts that will need to be run at least once.  Each of them requeues itself and so if you run it once via the URL (http://charting.local/jobs/process.traffic.php and process.sales.php).  Run them once and only once.  If you want to see what they do prior to injecting them into the job queue comment out the three lines of code starting with the object ZendJobQueue.

Now you should be ready to go with the Charting project.  If it is not able to compile due to a missing Stomp class you will need to reference the src folder in the project AS3Stomp.  After that is working right click in the Charting project on the Charting.mxml file in the src/(default package) directory and select Run As -> Mobile PHP Application.  That will kick off the charting application.  If there are problems you can select Debug As -> Mobile PHP Application and debug both the Charting application’s ActionScript or the PHP application’s service calls.

The link for the workspace zip is adobe.webinar.zip.

You can also see my slides on SlideShare.

I will leave you with that for the time being.  I intend to do another blog post on some of the reasons for doing things the way I did them along with some discussions about the architecture for the different integration points.  Additionally, I found that this webinar has highlighted a couple of pieces of needed documentation both for ActionScript and asynchronous community in PHP.  So, I have my work cut out for me for the next little while.  🙂


Selling 12k tickets in 2 minutes

Leave a comment

One of my little projects that I’ve been working on is an example where I wanted to build a system that would scale to be able selling 100k tickets in 5 minutes.  I’m happy to say I’ve succeeded, but I am not able to prove it.  It turns out that SOMEONE put a quota on how many EC2 instances I could spin up at a given point in time.  So, I had to opt for a cluster of 1 admin node and 19 worker nodes.  It wasn’t able to scale to the 100k tickets in 5 minutes, but with a few more instances I could have done it.

I intend to milk this one for content for a while, but at the same time I wanted to get a video up and running showing some of the things that I did to get it running.

I had originally decided to try and run it with simple Zend Server instances (without Cluster Manager) but after getting a cluster of 20 servers up and running I went through a test run and there was an error… and I didn’t know where.   So I relented and installed Cluster Manager.  Holy crap, you might say.  While I knew the selling points for Cluster Manager, and stood behind each one, this was the first time I had personally experienced how great it is to have your cluster managed by it.  This application depends very much on throwing data somewhere and expecting it to get processed.  But I had a front end load balancer to handle browser requests and then two other load balancers to handle logic execution and, quite honestly, there was no easy way for me to track the problem from end to end.  I installed Cluster Manager and had the issue identified in 5 minutes.  Stupid config.  Fixed it and was on my way.  My conclusion is that if you do anything with a load balancer in PHP and you want to catch bugs, Cluster Manager is a virtual necessity.


Added (PHP 5.3) job queuing to my WordPress instance

4 Comments

One of the things I liked on my old blog was the ability to have a Popular Posts section that was based off of Google Analytics. I had also like that I was using pre-caching, i.e., a cache that does not expire but is, rather, overwritten.  There are two benefits to this.  1) There is no lag time when the cache expires and I need to contact Google to get new data. 2) If my connection goes down for whatever reason (bad data, time expired, modified password), the data stays in the cache until you can get the problem fixed.

So I had missed that, but it was not overly important so I left it.  But yesterday was a day where I needed something that was both engaging and brainless to do.  So I decided to implement my Job Queue API code for WordPress so that I could write a WordPress widget that would put the popular posts in the sidebar.

It was actually relatively easy to do.  But the cool part was that I was able to extend WordPress, which still contains code that was written around when the Martini was invented, and PHP 5.3 code, which is what my Job Queue code was based off of.  This is part of some “ESchrade enablement” which I am building as a WordPress plugin and so it contains some other stuff than just the JQ part of the plugin.  You can download it from here and put it in your wp-content/plugins directory.  There are some things that aren’t working quite right yet, and probably won’t.  I don’t really have the desire to become a WordPress developer.

What have I learned or re-enforced from this?

  1. PHP backwards compatibility is pretty darn good
  2. If you build a software application, make building plugins easy, even if it’s your own app.  (WordPress is kind of win/lose on this)  I may end up looking at Event_Dispatcher.

Zend Server 5.1 Web API

Leave a comment

This weekend we soft launched the Zend Server 5.1 update.  So soft, in fact, that I didn’t even know it was launched.  But I did know it was coming.  Most of the improvements were either bug fixes or performance improvements.  Some of the performance improvements have actually been pretty significant, particularly under load.

But we did add a new feature that works both with Zend Server and Zend Server Cluster Manager.  This release for Cluster Manager is actually really nice.  I have a video demo that I recorded last week on scaling web applications that I could not have pulled off without the cluster manager.  One of the things that a scalable app does is “find” resources.  This can be application level, or in this case, infrastructure level, as I was using load balancing.  I had an error that I could not find because I did not know which server the logic ran on.  People often add loads of logging to store all of that information so they can go and retrace the steps that caused it.  I just said “screw it” and installed the Cluster Manager.  It took one request to reproduce it and I had all of the information I needed.

Seriously, if you have more than 2 or 3 servers and have enough budget to cover some basic employee benefits you NEED to look at Cluster Manager.  I knew the selling points of Cluster Manager before this, but I was sold on it when I needed it to do its thing when I recorded that video.

But I have digressed a little.

The new feature we added in both Zend Server and Zend Server Cluster Manager 5.1 was a new web API.  It allows you to programmatically access a subset of Zend Server’s provisioning features.  Some of them are available only to the Cluster Manager, but others are available for the regular version of Zend Server, i.e. the “single instance” version.

The feature list is as follows.

  • Get System Info
  • Get Server Status
  • Add Server
  • Remove Server
  • Disable Server
  • Enable Server
  • Restart PHP
  • Configuration Export
  • Configuration Import

The web API is RESTful and has an authentication method that calculates a request signature based off of a secret key that you can generate with either full access or read-only rights in the Zend Server GUI.

But I personally found that while the API was quite useful I could see that people would end up building their own adapters or scripting it in a way that would make it look like spaghetti.

So I wrote my own API that plugs in to the web API and makes your life easier.  I have included it as part of my Job Queue API up on GitHub and it requires PHP 5.3

There are a few simple steps to using it.

  1. Instantiate an API key object
  2. Instantiate a method object that represents the method you wish to call
  3. Instantiate the manager which controls the communication
  4. Call the method by passing it into the controller
  5. Read the response (which will either be raw data, a response object of a defined type (com\zend\api\response\*) or an array of response objects).

It’s really easy to use.  Take, for example, this code that will restart PHP on an individual server.

1
2
3
4
5
6
7
8
9
10
$mgr = new Manager();
$apiKey = new ApiKey(
'kevin',
'b562c7a6a46a9495df03e0b2c083f3ccfdb0de606b1abab05d056898fd6f410b'
);
$mgr->setApiKey($apiKey);
$mgr->setHost('192.168.0.246');
 
$restart = new RestartPhp();
$mgr->call($restart);

That’s it.  Or this code to export the current configuration.

1
2
3
4
5
6
7
8
9
10
11
$mgr = new Manager();
$apiKey = new ApiKey(
'kevin',
'b562c7a6a46a9495df03e0b2c083f3ccfdb0de606b1abab05d056898fd6f410b'
);
$mgr->setApiKey($apiKey);
$mgr->setHost('192.168.0.246');
 
$export = new ConfigurationExport();
$data = $mgr->call($export)->getResult();
// Store $data somewhere

Those are examples of API calls that work on the regular version of Zend Server.  Here are a few that work with Cluster Manager.

Say you want to add a server to your Zend Server cluster.  All you need to do is put this code in a provisioning script that runs on the node.

1
2
3
4
5
6
7
8
9
10
11
$mgr = new Manager();
$apiKey = new ApiKey(
'kevin',
'59455fb2e117e00bdcf04b37f9942da105e6bfd6a6cccf330fe79c79e89ddd4c'
);
$mgr->setApiKey($apiKey);
$mgr->setHost('192.168.0.242');
$addServer = new ClusterAddServer(
new ServerInfo('local', 'http://192.168.0.254:10081', 'password')
);
$mgr->call($addServer);

Done with the server and you want to remove it from the cluster (including graceful handling of session data)?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$mgr = new Manager();
$apiKey = new ApiKey(
'kevin',
'59455fb2e117e00bdcf04b37f9942da105e6bfd6a6cccf330fe79c79e89ddd4c'
);
$mgr->setApiKey($apiKey);
$mgr->setHost('192.168.0.242');
 
$serverInfo = new ServerInfo();
$serverInfo->setId(get_cfg_var('zend.node_id'));
 
$addServer = new ClusterRemoveServer(
$serverInfo
);
$mgr->call($addServer);

As you can see with the new web API you now have the ability to automate a bunch of the more brainless tasks that you’d rather let a machine take care of.

Get Zend Server (trial available)


New Zend Server Job Queue Library

9 Comments

This will probably be one of the last posts I do on the Zend Server Job Queue functionality.  From this point on they will probably be less frequent, though I’m sure several posts will allude to it.  The reason I’m going to put it on the back burner is because I have written a library which is now available on GitHub.  What it does is encapsulate all of the functionality that I think needs to be there to be able to implement asynchronous functionality.  I’ve been working on it off and on for at least a year, trying out different things.

As I was working through it there were a few things that I wanted to accomplish.

  1. It needed to be easy to use
  2. It needed to be installed in a few minutes
  3. It needed to hide a lot of the implementation details.

That said, I think that I’ve been able to accomplish a lot of that.  With that, let’s take a quick run through.

Creating the job class

There is only one class that you need to know.  It is called JobAbstract and it is in the com\zend\jobqueue namespace.  Yes, the library requires PHP 5.3, though it works with PHP 5.2 code as I’ve already built a Magento extension for it.  In order to implement the job queue, on the code side, you simply create objects that represent your tasks, extending the JobAbstract class and implementing the job() method.  So if you had a job that you wanted to have send out an email it would look something like this.

namespace org\eschrade\job;
use com\zend\jobqueue\JobAbstract;
 
class SendEmail extends JobAbstract
{
    private $to;
    private $message;
 
    public function __construct($to, $message)
    {
        $this->to = $to;
        $this->message = $message;
    }
 
    public function job()
    {
        $mail = new \Zend_Mail();
        $mail->addTo($this->to);
        $mail->setBodyHtml($this->message);
        $mail->send();
    }
}

Then when you want to execute this job you simply call the job

$job = new org\eschrade\job\SendEmail([email protected]', 'Welcome here!');
$response = $job->execute();

This will invoke the Manager object (which we’ll look at in a bit) which will then send the request to the specified job queue server, or to a load balancer.  The response is important if you want to get the results of the job.  If the request is serviced through a load balancer you don’t necessarily have any knowledge about where your job is going to execute.  Thus, if you want to get the result of the job you need to know how to get the job.  That information is stored in the result.  It’s a very simple, serializable object that is intended for persistent storage, such as a session or database.

Setting up the Manager

The manager is mostly self-contained.  The only thing it requires is that you provide a URL for the backend job queue end point.  The end point needs to only contain the following code.

1
2
3
4
use com\zend\jobqueue\Manager;
 
$mgr = new Manager();
$mgr->invoke();

Now, your application may require more at the end point to bootstrap it, but that’s all it needs to interact with your front end servers.

On the front end you need to tell the Manager what the URL is to connect to a job queue server.  As I said before, it can be an individual server or a load balancer in front of a hundred job queue servers, it really doesn’t matter, except that you can scale quite easily without any configuration changes.  To set the URL simply tell the Manager what the URL is.

1
Manager::setDefaultUrl('http://lb/queue.php');

And that’s pretty much it.

Communicating with the backend

Once you’ve sent the job off to the backend to execute you might want to interact with the results of the job.  Once the job object has finished executing, the Manager will serialize the job so you can get the results.  This would typically be done via getters and setters.  I have a good example of this in the library code on GitHub.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
class GetRemoteLinks extends JobAbstract
{
 
    private $url;
    private $links = array();
 
    public function __construct($url)
    {
        $this->url = $url;
    }
 
    public function getLinks()
    {
        return $this->links;
    }
 
    public function getUrl()
    {
        return $this->url;
    }
 
    public function job()
    {
 
        $http = new \Zend_Http_Client($this->url);
        $response = $http->request();
        $selector = new \Zend_Dom_Query();
        $selector->setDocumentHtml(
                $response->getBody()
         );
        foreach ($selector->query('a') as $result) {
            if ($result->getAttribute('href')) {
                $this->links[] = $result->getAttribute('href');
            }
        }
    }
}

To execute this job you would call

1
2
$job = new GetRemoteLinks('http://www.eschrade.com/');
$result = $job->execute();

Then later on

1
2
3
4
5
6
$mgr = new Manager();
if (($job = $mgr->getCompletedJob($result)) !== null) {
    foreach ($job->getLinks() as $link) {
        echo $link . '<br />';
    }
}

 

That’s pretty much it.