Using PHP WITH Jetty, Camel, ActiveMQ, WebSockets and other things that might score high on SEO

Leave a comment

Over the past several days (more than I would like to admit) I’ve been working on getting a system based off of Jetty and ActiveMQ to work out of the box for PHP developers.  While “out of the box” isn’t quite true I have been able to get everything to work in a single downloadable container.  It uses ActiveMQ as the messaging system, which also has a WebSocket listener.  ActiveMQ also has a queue called “worker” which is used to send job queue requests to the queue.  That queue has an HTTP camel component which listens on the queue and forwards the requests to an HTTP pool.  Running the whole thing is a pre-configured Jetty server.  Getting that set up takes a smidgen of work, but it’s not that hard.  grep -R is your friend.

The example application is a chat application (of course), but it has some added functionality.  You can create a pure WebSocket chat server quite easily, no PHP needed.  But I wanted to include PHP functionality so what I did was make the (ugly) frontend push the chat message to the queue and the worker would do three things.

  1. Parse it for Markdown-based content
  2. Regex the output for URLs and send those URLs back to the queue
  3. When dequeued the URLs are parsed for a content snippet, title and an image URL.  If those are found they are sent back to the frontend which then attaches the snip to the original message container.

It’s kind of a neat little setup.  And, what’s cool about it, is if you change the document root for the website you can run a Magento instance out of it.

You can download the whole thing here.

Configuration

It is configured to run on port 8161.

There are a few things you are going to need to do to make it work.

First, make sure you have PHP-FPM running on port 9000.

Second, edit the etc/vhosts/localhost.xml file to change the document root for the web application in the download’s html directory.

Third, edit the html/WEB-INF/web.xml to change the document root in that file as well.

To run PHP in Jetty you need to have the WEB-INF/web.xml file in there.  There is a configuration setting where the only path to change it is in that file.  I spent, literally, days trying to get this to work without building my own into Jetty.  I could have done that, but I wanted this to run as much out-of-the-box as possible.  The setting is async-supported, in case you are wondering.  There was a change in one of the Jetty versions where FastCGI went from announcing it supported async by default to saying it didn’t.  And the only way to change that setting is in the WEB-INF setting.

The Application

The application, as noted before, is a simple chat application.  You can have multiple people sharing the same chat window.  Everyone can chat with everyone else.  There is no concept of a room, this is a demo after all.

To make it work, go to either http://localhost:8161/ or http://localhost/ if you set up iptables to route to port 8161.  From there the chat window will open and you can start typing.  Note that I have not built anything resembling security into this application.

Some notes on the packages installed.  The first is the eschrade/async_pack composer package.  I have used only one component, the Stomp client.  That client extends the Fusesource Stomp package.  I needed to extend it because I was getting connection errors when creating the WebSocket connection and it’s easier to extend a package than to get a fix done… especially a fix that could break other adapters.

One note on how I do queuing.  It is wrong.  Horribly wrong.  Don’t use it as an example.  In the eschrade/async_pack I created a queue client/server model that can be used.  But for a simple demo what I did was sufficient.  Maybe when I get some free time I’ll make it better.

Moving On

I hope you try it out and like it.  I’ve been a little bit surprised by how much I’ve liked it despite the fact that I’m using it largely as a web server.  Perhaps it’s because this approach opens up a world of opportunities with PHP developers that, beforehand, required a lot of infrastructure chicanery.  This package has most of what I’ve wanted in an all-in-one PHP environment for almost a decade.


Configuring Jetty to run Magento

Leave a comment

Yesterday I wrote a blog post on my surprising finding that Jetty was able to out-perform, or at minimum keep up (depending on how you looked at the numbers), Nginx when it comes to static files.  Today I wanted to get it up and running and able to serve Magento requests.

It turns out that it is stupid easy.  That said, it was stupid hard figuring out how stupid easy it was.

To get Magento running with Jetty

  1. Set up PHP-FPM as you would with Nginx, making sure that you use TCP and not Unix sockets
  2. Download Jetty.  Do not use the RPM, if you are using CentOS.  Yum does not have all the components you need.  Use the full download from the Jetty page on the Eclipse site.  Install into /usr/share/java/jetty
  3. Configure Jetty.

So let’s look at number 3.

There are two things you need to know about this.

First of all is that FastCGI is not enabled by default.  You need to add it to its start.d directory.

FastCGI is now installed.  Now you need to configure a default context.  Create a file called $JETTY_HOME/etc/webapps/fcgi.xml (or whatever.xml) and add the following.

THAT  is a lot of stuff.  Well, actually it’s not, but it’s very verbose.  There are a few things you need to know

  1. //New[@id=”root”] – this is document root for this web application
  2. //Set[@name=”contextPath”] – This is the base URI where your application lives.  http://example.com/ would be /, whereas http://example.com/blog/ would be /blog/
  3. Find org.eclipse.jetty.fcgi.server.proxy.TryFilesFilter.  That is the equivalent of Nginx’s try_files setting.  Here, if it doesn’t find the file, route it to index.php.
  4. The last addServlet item is the FastCGI component.  I trust you can see the two important settings.

And there you have it.  A Jetty instance configured to run Magento.

Oh, and how does this compare to Nginx in terms of Magento throughput?

2016-06-03_1250


Browser Communication with Nginx and WebSockets…

Leave a comment

… was what I was going to write about.  I’ve been meaning to come up with a solution where I could do a combination of regular HTTP requests combined with Web Sockets combined with Redis to send messages easily back and forth between the backend and the frontend.  My thought was that I might be able to take Nginx, which has WebSockets proxy support and combine it with Redis to create a means of facilitating communication between a frontend browser user and any backend asynchronous tasks that might have pertinent messages to send to the frontend.

There are definitely ways of doing it.  But, basically, all of them involve having a web frontend, Redis, and then a bunch of Node code that you have to write and maintain.  The Node block was one that has held my research on this back for several months.  I just don’t want to have yet another service running, that requires yet more code for me to write, in yet another programming language.

Fast forward to this week.  I started working on a load testing mechanism that allows Magium tests to be used as a means of load testing.  Obviously you cannot generate significant loads from a browser test on one server and so the solution REQUIRED some form of messaging and synchronization.  In my searching I found that ActiveMQ could be used as an embedded messaging service.  Well, I know Java, and I know that ActiveMQ has WebSockets support (so I could control the tests from a browser), so “why not?” I figured.

It turns out that it actually all worked really, really well.

So it turns out that Nginx + this + that + plus the kitchen sink, wasn’t actually necessary.  Everything could be handled within two distributables from Apache: Jetty and ActiveMQ (with Camel which acts as a bridge between the two).  With that I get WebSockets, HTTP/1 & 2 (HTTP push maybe?), and an integrated async messaging system (I LOVE async!).  Add an ESB like MuleSoft and you’re set for almost any kind of communication.

But I’m getting ahead of myself.

While I intend to answer deeper questions of how Jetty fits into a larger pattern of advanced web applications, including running PHP, I don’t have the time to do that today.  I owe some folks some deliverables and so I will continue my investigation into using Jetty in another post.

What I want to discuss at this point is static throughput.  Java has the reputation of being slow and, in particular, bloated.  I believe that this reputation is somewhat deserved.  But like many who assert that the filesystem is slow sometimes the bigger picture is a little more complicated.  But if Jetty turned out to be “enterprisey” (which means big, bloated and slow) then this wouldn’t work as a solution.

So like a good little boy, I tested to see if Jetty would be a blocker on the performance/bloat front.

The first graph is the test results.  The test was retrieving the favicon.ico file on a 4xIntel(R) Xeon(R) CPU X3430 @ 2.40GHz. with 16GB RAM.  All webservers were configured to run on their defaults.  That means that these numbers could be improved.  In fact, Nginx could possibly double its throughput (but, then again, so could Jetty, for reasons we’ll see in a bit).  Apache is pretty much at its limit.

throughput

What?  Jetty was faster than Nginx?  Yep.  In this test it was.  I was working under the assumption that, given the ratio of static content to PHP requests in a Magento site, I would consider Jetty a contender if it was simply better than Apache.  I was most definitely NOT expecting it to be faster than Nginx.

So lets get down into some of the details.

Let’s dispense with Apache first, since we all know why it performed like this (though, 10k requests per second is definitely not bad).

apache.static.500

Apache hit the throughput limit pretty quickly and never got above 10k requests per second

apache-vmstat.500

In terms of system resource usage, the usage pretty much maps to throughput.  The throughput was limited by CPU.

No surprises there.

Next up is Nginx.  Nginx was configured with worker_processes set to auto, which configured itself to 4.  The result of the throughput test is this:

nginx-static.500

Throughput went up very quickly and was serving upwards of 47,000 requests per second.

nginx-vmstat.500

This is the system usage during the Nginx test.  Clearly system time was the driver for throughput.  It matches the throughput almost exactly.  Though, I wonder if this is a function of network saturation because of the downward slope of the throughput for system time.  If the throughput decline was due to increased system time then the system time should increase while throughput decreases.  Instead we see them in lock step.  For that reason I believe that the overhead of additional HTTP payload (headers, and such), would explain the decline.  In other words, Nginx could probably do better.

But that’s not really the point of all this.  The point of all this, in my mind, was that Jetty was able to keep up with (and surpass) Nginx in the static throughput department.

jetty-static.500

The peak throughput was around 53,000 requests per second, compared to Nginx’s 47k.

jetty-vmstat.500

System usage was similar to Nginx, perhaps even a little better.  Nginx hit 70% and stayed there whereas Jetty peaked at 70%, dropped to 37% and then hit 60%.

In short, I was dumbfounded when my first test graphs were being rendered.  I had figured that there was an error; it was not what I was expecting at all.

Does this mean that you should switch to Jetty?

No.

Jetty has a lot of complexity and most Magento (my focus) system admins do not have the experience to administer Jetty.  If you have standard needs, Nginx + PHP-FPM is most likely the best option still.

However, do you have an application that needs WebSockets or some form of messaging (such as JMS)?  These test results make for some interesting thoughts.

That highlights one of the things I like about Java.  I like the language syntax, but what I like about Java more is that a lot of the stuff that “feels” cobbled together in the PHP world already exists in the Java world.  Instead of writing yet another abstraction layer in PHP that does about 20% of what you need, perhaps the Java infrastructure running on Jetty gets you 80% of the way there instead of 20%.

But, like I said, I have deliverables I need to deliver and so I need to wrap up this blog post.  There are three questions I still want to answer.

  1. How does this Jetty work wit PHP-FPM?  I expect this to be a slam dunk since PHP-FPM works fine with Nginx.
  2. How can a PHP developer put this all together and give themselves more features than they could ever know what to do with?
  3. I still need to directly answer the question of easy integration of WebSockets and messaging.  I expected that either Jetty + ActiveMQ or Jetty + Redis will provide an out-of-the-box(ish) solution.

 


Metaphysics and Software Design

Leave a comment

When I say that this is an experimental blog post I mean it.  I’m thinking out loud here.

Those who know me know that I am somewhat critical of some modern advancements in software design.  It’s not that I’m a Luddite or anything but the root of my discontent is that software and real life often do not match well.  Inbox by Google is a good example of this.  They refuse to put an “unread” feature into their software.  It messes with their blessed workflow.  Never mind that sometimes real life breaks the workflow, like if you accidentally click on an item. In the case of Inbox that accident means that you either have to open up other software (Gmail) or snooze the item you accidentally opened.

That’s an example of a mismatch of UX and reality.  But it also deeper than that.  Essentially any place where there is an interface between the human and the machine, from the user interface all the way to an API, there is going to be a mismatch between what the machine needs and how the human needs to interact with the machine.  And our response is often to make the human match the machine instead of the machine matching the human.  And given that the machine is often the interface between two humans, this seems backwards to me.  But I digress.

Consider the old interface with Magento 1.

How in the world does this break with metaphysics?  Well, one of the principles of Aristotelian metaphysics is that an object cannot actuate a potential that it does not possess.  In other words a ball cannot make water because it does not contain the potential for water.  But Oxygen and Hydrogen do contain the potential for water.  (Remember we’re talking about METAphysics, not physics.  It is very easy to get hung up on that.)

Going back to the example it could be argued that the code is bad code insofar as it breaks from metaphysics.  ArbitraryClass does not have the potential for a request.  How do we know that?  Because it’s Formal Cause (or Form) is not request-like, or request-ish.  And because there is a dissonance between its formality and efficiency, it is broken insofar as that dissonance occurs.  That’s not to say that it won’t work, but it’s workability will be a function of its alignment with its form.

Consider a triangle.  Have you ever seen a triangle?  No.  Never.  “You’re insane, Kevin,”  You might be thinking.  “I have seen triangles all over the place!”  Yes, you have seen examples of triangles, but they have never been perfect triangles.  A perfect triangle exists only in the immaterial sense.  What you have seen are objects that represent triangularity, but due to slightly curved lines, shifts on the paper, variations in the molecular density of the lead, among other things, you have never, ever in your life seen a triangle.

But you can still recognize a triangle based on its resemblance of triangularity.  In other words, it will still work as a triangle just like our code will still work like our code should.  But the degree of its success will be based on how well it confirms to its reality.

You might be thinking that this is all crap.  But let’s look at a more current implementation.

This example follows the principle that an object cannot give what it does not have.  As a result it is also “better” code.

But it also does a lot more than that.  It actually demonstrates some of the fundamental parts of Aristotle’s philosophy.

  1. It has an Essence or Form.  The DoSomethingInterface is its form, like triangularity.
  2. It does not have “request-ability” and so it must be given by something that does (the Dependency Injection Container).
  3. It has its material cause in its instantiation.
  4. It has efficient causality.  It “does something”.
  5. It has final causality.  It has a purpose.  (nobody intentionally creates un-purposed code)

And so forth.

This does not mean that you are required to code according to philosophy.  What it does seem to show is that the quality of code can be measured against how well the code fits against a logical metaphysical framework.

Bear in mind that this idea is new to me and there could very well be some disqualifying concept that renders this moot.  I’m, in a sense, debating myself in public.

But the flip side of this is that if this is correct, and good software does seem to follow good metaphysical flows, that may be able to be a framework for people to write code that is not only better, but works better in the real world.

But then again, this would turn programmers into Philosophers and God help us if that happens.  🙂


Validating Individual Shipping Methods

Leave a comment

Out of the box Magium will select the first shipping method on the screen. But there might be some times when you need to validate a particular shipping method, or force an error to be thrown if a shipping method does not exist. ’tis now easy to do. use MagiumMagentoAbstractMagentoTestCase; class ShippingTest extends AbstractMagentoTestCase { … Continue reading “Validating Individual Shipping Methods”

Powered by WPeMatico


Asserting page load times in Magento with Magium

Leave a comment

Validating SLAs (Service Level Agreements) is often part of executing browser tests. With Magium you can now include page timings as part of your test. Say, for example, that part of your SLA is that a page must be rendered in less than 5 seconds. This is done by using the TTFBLoadedLessThan, DOMPageLoadedLessThan or DOMContentLoadedLessThan … Continue reading “Asserting page load times in Magento with Magium”

Powered by WPeMatico


Validating that page assets are cached

Leave a comment

It’s not just HTML elements that need to be tested, and sometimes not even testing that images or CSS files are being tested. Sometimes you need to test things like server configurations. Magium’s position is that there are a wealth of different areas where browser testing can benefit your organization. Validating browser testing configurations is … Continue reading “Validating that page assets are cached”

Powered by WPeMatico