What’s wrong with the sound in movies

Leave a comment

My wife and I saw the new Star Trek movie in Imax 3D last night.  It wasn’t too bad.  She complained about it being too loud (fuddy duddy) whereas I was impressed with the acoustic treatments in the theater and the clarity of the low end frequencies.  Imax has always been cool with the large screens, but I have always been more impressed by the sound.  Incidentally, I was really disappointed in the screen size.  I was expecting the huge 4:3 screen that I used to watch documentaries on.  This was just a slightly larger 16:9.  When I go to Imax I expect to feel like barfing.

But the thing that got me was the nature of the sound effects.  Here you have these massive, epic sequences and they just didn’t sound right.  Why it didn’t sound right hit me during one seen where the camera shot was on the ground and the Enterprise is careening through a series of sharp-peaked mountains.  Every time the Enterprise hit a peak (It’s brave new mission to be destroyed during the filming of each movie) the sound was immediate.  You heard the hit, and felt it, thanks to the amazing Imax subs, exactly as it happened.

Come back in your imagination with me to somewhere around 1995.  I was at my first air show, the Winnipeg International Air Show.  I was with a friend and we spent an hour or two shopping the grounds for a new F-14.  I remember being amazed as I walked through the belly of a C-5 Galaxy and around a B-1 Lancer.  The show was about to start so we mozied to the show grounds and waited for it to start.

We waited for a bit, with some of the requisite ’80’s music playing over the loudspeakers.  I had seen a CF-18 take off and was looking around the sky for it.  I suddenly saw a small speck out in the west that was not moving like a bird.  It flew in a straight line and came closer and closer.  I could soon make out the tail fins.  And it was coming in fast.

I was a little apprehensive.  I’m a farm boy and I’d seen a CF-18 once flying over a field, but this was the first time I’d seen one flying like this.  It was flying like it wanted to show what it could do.

It entered the airfield with the mach indicator reading all 9’s with a dot in front of it.  It was flying like an arrow.  A 0.999 mach arrow.  With increasing velocity it slid through the air faster than I had ever seen before.  And what did I hear?

Nothing.

I didn’t hear anything.  And I became nervous.  I like loud things but I didn’t know what to expect because it was acting in a way that I had not expected.  I expected it to be awesome.  I did not expect it to be silent.

Then I heard a ripple making its way across the crowd.  It was almost like the pressure wave was surfing the heads of the people to my left.  It didn’t get louder, but I could hear the wave coming to greet my ears with utter disregard.

And then it hit.  It was a blast of sound like I had never heard before.  I had heard jets, but I had never heard the sound racing towards me before.  It was amazing.

That is what is missing in many movies nowadays.  There is sound, and the guys who do the sound are, literally, amazing.  But there is one thing that is missing.  The experience of being in the movie.  Sound is the conduit for that.  When everything is experienced as if you were in the middle of the action, even when the camera isn’t, lessens the immersion.

There are a few movies that do it well.  I think that I remember Saving Private Ryan having sound like this.  The end of Aliens, when the processing station blows up on LV-426, is another one.  Ripley sees the light, covers Newt’s eyes, and, then, you hear the explosion.

When making a movie, if you really want to bring the audience in and immerse them in the experience you need to do this.  It is the anticipation of the noise, not the noise itself, that makes it memorable.  In Mythbusters when have you ever heard the shock wave exactly when the explosion happened?  Nope.  And what is Adam and Jamie’s level of anticipation each time?  Epic.


Shutting down Magium (and (eventually) looking for a new awesome company to work for)

2 Comments

If you don’t know what Magium is, it is a Selenium based testing framework for Magento and other platforms that makes it much, MUCH, easier to test those applications.  I’ve been working on it since December and it is now 7 months out.  7 months of very little income (thank you to MagentoU for having me do some training work for you).  And after 7 months of working on Magium with little by way of progress (as a business.  The technology progressed quite nicely) I’ve decided that it’s time to throw in the towel.

It is, perhaps, fitting that I post this the week of the July 4th holiday as it was exactly a year ago where I was working through the holiday on building browser tests for a customer module for the Expert Consulting Group.  It was then that I discovered the horrid nature browser testing for those who are not QA professionals.  And while it was several months before Magium started to take form in my mind, the July 4th weekend of 2015 is when I was convinced that there had to be a better way.

Magium still is the better way, but it will not be a business for me.

Something like this of course requires some level of reflection.  This is now the second venture that I’ve folded in the past decade.  The first was trying my hand at mobile apps after I burned myself out at a previous job.  This time it was actually something that had, and still does IMHO, a lot of potential.  80% of Magento developers (and developers in general) do not execute automated browser tests with difficulty being the primary objection.  The approach I took for Magium significantly reduced the investment required to automate your browser testing, especially if you were not a dedicated QA person who knows what you’re doing.

But for whatever reason I was not able to make it stick.  I’ve been watching the installs on Packagist stay relatively stagnant and after switching to a services based method over a month ago I have not been able to drum up any paying significant  gigs.  It’s hard to try and build an open source project that few people show up for.  There have been several people who have expressed interest, and I’ve even had a few contributions.  But this is prima facie evidence that the mantra of “if you build it, they will come” is utterly false.

Not that I ever believed that anyway.

So, what to do?

Right now, nothing.  I burned myself out.  And it’s the middle of summer.  And I’m pretty sure I need some time to rest.  Shutting it down after I was so utterly convinced that there was business potential here is like a giant kick in the gut… or lower.  At least I can still tell inappropriate jokes.  Run for the hills if I ever lose that ability.

So right now I’m going to be staying put.  Maybe do some renovations.  Do some writing.  Do some work on Magium.  Hopefully do some recording.  Really hopefully do some recording.  It’s one of the only things I’ve ever done that has brought me no money but gobs of joy.  I still listen to my last two albums and love them, even though they aren’t as good as a lot of stuff out there.

Another thing I intend to work on is building my wife’s eCommerce empire.  By “empire” I mean the hundred small decorative accessories that she has accumulated over the past few months and are sitting idly in my dining room.  She has an excellent combination of eye for quality, eye for design, and eye for value.  She is quite talented in that regard.  I, on the other hand, can architect any website except the really big ones.  So, empire it is.

We’ll see how that goes.  Hopefully it goes well.  At worst my wife gets to exercise her gifts and you’ll get a wonderful little accessory for your house.

You’re going to buy something, right? 🙂

If you want to get on her New Product Newsletter, feel free to do so here.  We’re still a little ways from getting it up and running but hopefully it won’t be too long.

Here are the kinds of things you might see.


In the meantime, if you are a company who would like someone who has been a (in no particular order)

  1. Conference speaker (several times over)
  2. Conference MC (twice, I think)
  3. PHP Architect
  4. Magento Consultant/Developer/System Administrator
  5. PHP Consultant
  6. Sun System Administrator
  7. Linux System Administrator
  8. Book Author (3 books on PHP)
  9. Trainer
  10. Course Developer
  11. Product Evangelist
  12. Technical Salesman
  13. Sales Engineer
  14. Blogger
  15. Business Owner (failed, but who’s counting?)
  16. Member of the Zend Certification Board
  17. MCDPlus Certified Developer

… don’t hire me quite yet.

I need some rest.  Though it won’t be too long; I’m running low on Cognac and my Cointreau is gone.  But in the next month or two I will be looking for something to fill my time in exchange for gargantuan sums of money.  Product Marketing and Architecture sound interesting.  Having adopted my three children I have some unique family circumstances which causes me to lean towards a position where I can help at home here and there.  That’s not to say I would say no to an office position, but if given two equal opportunities and the only difference were working remotely I would take the remote position not because of the typical benefits of working at home (which I think are often exaggerated), but because of those unique familial issues I have.

I would be most interested in working for a (funded, or on its way to be funded) startup.  Creative established companies interest me as well.

I would love to do work with a music-based company.  Not required, but it would be nice to have both software and music as part of my career so the two could stop fighting.

If you are looking for a cog for your wheel I am not your guy.  A lot of companies need a lot of those, and many people dig that kind of work.  I have no quarrels with them.  But that’s not me and you wouldn’t be getting your money’s worth if that’s what you needed.

If you want someone who occasionally takes stupid risks that looks like genius in hindsight, and thinks about technical problems in a market-oriented way and writes gluten-free code, then I’m your guy.  Many companies claim to want people who “think outside the box” but don’t really mean it.  Not only do I think outside the box, but my box is a 10 dimensional chiliagon.  It’s scary in there but oh, so interesting.

If you would like to talk to me, drop me a line at [email protected].  I’d love to hear from you.

I will still be accepting consulting work for Magium, or anything PHP related, really, while I try to figure out the best place for me to land.


Using PHP WITH Jetty, Camel, ActiveMQ, WebSockets and other things that might score high on SEO

Leave a comment

Over the past several days (more than I would like to admit) I’ve been working on getting a system based off of Jetty and ActiveMQ to work out of the box for PHP developers.  While “out of the box” isn’t quite true I have been able to get everything to work in a single downloadable container.  It uses ActiveMQ as the messaging system, which also has a WebSocket listener.  ActiveMQ also has a queue called “worker” which is used to send job queue requests to the queue.  That queue has an HTTP camel component which listens on the queue and forwards the requests to an HTTP pool.  Running the whole thing is a pre-configured Jetty server.  Getting that set up takes a smidgen of work, but it’s not that hard.  grep -R is your friend.

The example application is a chat application (of course), but it has some added functionality.  You can create a pure WebSocket chat server quite easily, no PHP needed.  But I wanted to include PHP functionality so what I did was make the (ugly) frontend push the chat message to the queue and the worker would do three things.

  1. Parse it for Markdown-based content
  2. Regex the output for URLs and send those URLs back to the queue
  3. When dequeued the URLs are parsed for a content snippet, title and an image URL.  If those are found they are sent back to the frontend which then attaches the snip to the original message container.

It’s kind of a neat little setup.  And, what’s cool about it, is if you change the document root for the website you can run a Magento instance out of it.

You can download the whole thing here.

Configuration

It is configured to run on port 8161.

There are a few things you are going to need to do to make it work.

First, make sure you have PHP-FPM running on port 9000.

Second, edit the etc/vhosts/localhost.xml file to change the document root for the web application in the download’s html directory.

Third, edit the html/WEB-INF/web.xml to change the document root in that file as well.

To run PHP in Jetty you need to have the WEB-INF/web.xml file in there.  There is a configuration setting where the only path to change it is in that file.  I spent, literally, days trying to get this to work without building my own into Jetty.  I could have done that, but I wanted this to run as much out-of-the-box as possible.  The setting is async-supported, in case you are wondering.  There was a change in one of the Jetty versions where FastCGI went from announcing it supported async by default to saying it didn’t.  And the only way to change that setting is in the WEB-INF setting.

The Application

The application, as noted before, is a simple chat application.  You can have multiple people sharing the same chat window.  Everyone can chat with everyone else.  There is no concept of a room, this is a demo after all.

To make it work, go to either http://localhost:8161/ or http://localhost/ if you set up iptables to route to port 8161.  From there the chat window will open and you can start typing.  Note that I have not built anything resembling security into this application.

Some notes on the packages installed.  The first is the eschrade/async_pack composer package.  I have used only one component, the Stomp client.  That client extends the Fusesource Stomp package.  I needed to extend it because I was getting connection errors when creating the WebSocket connection and it’s easier to extend a package than to get a fix done… especially a fix that could break other adapters.

One note on how I do queuing.  It is wrong.  Horribly wrong.  Don’t use it as an example.  In the eschrade/async_pack I created a queue client/server model that can be used.  But for a simple demo what I did was sufficient.  Maybe when I get some free time I’ll make it better.

Moving On

I hope you try it out and like it.  I’ve been a little bit surprised by how much I’ve liked it despite the fact that I’m using it largely as a web server.  Perhaps it’s because this approach opens up a world of opportunities with PHP developers that, beforehand, required a lot of infrastructure chicanery.  This package has most of what I’ve wanted in an all-in-one PHP environment for almost a decade.


Configuring Jetty to run Magento

Leave a comment

Yesterday I wrote a blog post on my surprising finding that Jetty was able to out-perform, or at minimum keep up (depending on how you looked at the numbers), Nginx when it comes to static files.  Today I wanted to get it up and running and able to serve Magento requests.

It turns out that it is stupid easy.  That said, it was stupid hard figuring out how stupid easy it was.

To get Magento running with Jetty

  1. Set up PHP-FPM as you would with Nginx, making sure that you use TCP and not Unix sockets
  2. Download Jetty.  Do not use the RPM, if you are using CentOS.  Yum does not have all the components you need.  Use the full download from the Jetty page on the Eclipse site.  Install into /usr/share/java/jetty
  3. Configure Jetty.

So let’s look at number 3.

There are two things you need to know about this.

First of all is that FastCGI is not enabled by default.  You need to add it to its start.d directory.

# export JETTY_HOME=/usr/share/java/jetty
# cd /usr/share/java/jetty/etc 
# java -jar $JETTY_HOME/start.jar --add-to-start=fcgi
INFO: fcgi initialised in ${jetty.base}/start.ini
INFO: Base directory was modified

FastCGI is now installed.  Now you need to configure a default context.  Create a file called $JETTY_HOME/etc/webapps/fcgi.xml (or whatever.xml) and add the following.

<?xml version="1.0" encoding="UTF-8"?>



 
    
        /var/www/magento19/magento/
    
 
    /
    
    
        index.php
    
 
    
        org.eclipse.jetty.fcgi.server.proxy.TryFilesFilter
        /*
        
            
                
            
        
        
            files
            $path /index.php
        
    
 
    
        
            
                default
                
                    
                        org.eclipse.jetty.servlet.DefaultServlet
                    
                
                
                    dirAllowed
                    false
                
            
        
        /
    
 
    
        org.eclipse.jetty.fcgi.server.proxy.FastCGIProxyServlet
        *.php
        
            proxyTo
            http://localhost:9000
        
        
            prefix
            /
        
        
            scriptRoot
            
        
        
            scriptPattern
            (.+?\\.php)
        
    
 


THAT  is a lot of stuff.  Well, actually it’s not, but it’s very verbose.  There are a few things you need to know

  1. //New[@id=”root”] – this is document root for this web application
  2. //Set[@name=”contextPath”] – This is the base URI where your application lives.  http://example.com/ would be /, whereas http://example.com/blog/ would be /blog/
  3. Find org.eclipse.jetty.fcgi.server.proxy.TryFilesFilter.  That is the equivalent of Nginx’s try_files setting.  Here, if it doesn’t find the file, route it to index.php.
  4. The last addServlet item is the FastCGI component.  I trust you can see the two important settings.

And there you have it.  A Jetty instance configured to run Magento.

Oh, and how does this compare to Nginx in terms of Magento throughput?

2016-06-03_1250


Browser Communication with Nginx and WebSockets…

Leave a comment

… was what I was going to write about.  I’ve been meaning to come up with a solution where I could do a combination of regular HTTP requests combined with Web Sockets combined with Redis to send messages easily back and forth between the backend and the frontend.  My thought was that I might be able to take Nginx, which has WebSockets proxy support and combine it with Redis to create a means of facilitating communication between a frontend browser user and any backend asynchronous tasks that might have pertinent messages to send to the frontend.

There are definitely ways of doing it.  But, basically, all of them involve having a web frontend, Redis, and then a bunch of Node code that you have to write and maintain.  The Node block was one that has held my research on this back for several months.  I just don’t want to have yet another service running, that requires yet more code for me to write, in yet another programming language.

Fast forward to this week.  I started working on a load testing mechanism that allows Magium tests to be used as a means of load testing.  Obviously you cannot generate significant loads from a browser test on one server and so the solution REQUIRED some form of messaging and synchronization.  In my searching I found that ActiveMQ could be used as an embedded messaging service.  Well, I know Java, and I know that ActiveMQ has WebSockets support (so I could control the tests from a browser), so “why not?” I figured.

It turns out that it actually all worked really, really well.

So it turns out that Nginx + this + that + plus the kitchen sink, wasn’t actually necessary.  Everything could be handled within two distributables from Apache: Jetty and ActiveMQ (with Camel which acts as a bridge between the two).  With that I get WebSockets, HTTP/1 & 2 (HTTP push maybe?), and an integrated async messaging system (I LOVE async!).  Add an ESB like MuleSoft and you’re set for almost any kind of communication.

But I’m getting ahead of myself.

While I intend to answer deeper questions of how Jetty fits into a larger pattern of advanced web applications, including running PHP, I don’t have the time to do that today.  I owe some folks some deliverables and so I will continue my investigation into using Jetty in another post.

What I want to discuss at this point is static throughput.  Java has the reputation of being slow and, in particular, bloated.  I believe that this reputation is somewhat deserved.  But like many who assert that the filesystem is slow sometimes the bigger picture is a little more complicated.  But if Jetty turned out to be “enterprisey” (which means big, bloated and slow) then this wouldn’t work as a solution.

So like a good little boy, I tested to see if Jetty would be a blocker on the performance/bloat front.

The first graph is the test results.  The test was retrieving the favicon.ico file on a 4xIntel(R) Xeon(R) CPU X3430 @ 2.40GHz. with 16GB RAM.  All webservers were configured to run on their defaults.  That means that these numbers could be improved.  In fact, Nginx could possibly double its throughput (but, then again, so could Jetty, for reasons we’ll see in a bit).  Apache is pretty much at its limit.

throughput

What?  Jetty was faster than Nginx?  Yep.  In this test it was.  I was working under the assumption that, given the ratio of static content to PHP requests in a Magento site, I would consider Jetty a contender if it was simply better than Apache.  I was most definitely NOT expecting it to be faster than Nginx.

So lets get down into some of the details.

Let’s dispense with Apache first, since we all know why it performed like this (though, 10k requests per second is definitely not bad).

apache.static.500

Apache hit the throughput limit pretty quickly and never got above 10k requests per second

apache-vmstat.500

In terms of system resource usage, the usage pretty much maps to throughput.  The throughput was limited by CPU.

No surprises there.

Next up is Nginx.  Nginx was configured with worker_processes set to auto, which configured itself to 4.  The result of the throughput test is this:

nginx-static.500

Throughput went up very quickly and was serving upwards of 47,000 requests per second.

nginx-vmstat.500

This is the system usage during the Nginx test.  Clearly system time was the driver for throughput.  It matches the throughput almost exactly.  Though, I wonder if this is a function of network saturation because of the downward slope of the throughput for system time.  If the throughput decline was due to increased system time then the system time should increase while throughput decreases.  Instead we see them in lock step.  For that reason I believe that the overhead of additional HTTP payload (headers, and such), would explain the decline.  In other words, Nginx could probably do better.

But that’s not really the point of all this.  The point of all this, in my mind, was that Jetty was able to keep up with (and surpass) Nginx in the static throughput department.

jetty-static.500

The peak throughput was around 53,000 requests per second, compared to Nginx’s 47k.

jetty-vmstat.500

System usage was similar to Nginx, perhaps even a little better.  Nginx hit 70% and stayed there whereas Jetty peaked at 70%, dropped to 37% and then hit 60%.

In short, I was dumbfounded when my first test graphs were being rendered.  I had figured that there was an error; it was not what I was expecting at all.

Does this mean that you should switch to Jetty?

No.

Jetty has a lot of complexity and most Magento (my focus) system admins do not have the experience to administer Jetty.  If you have standard needs, Nginx + PHP-FPM is most likely the best option still.

However, do you have an application that needs WebSockets or some form of messaging (such as JMS)?  These test results make for some interesting thoughts.

That highlights one of the things I like about Java.  I like the language syntax, but what I like about Java more is that a lot of the stuff that “feels” cobbled together in the PHP world already exists in the Java world.  Instead of writing yet another abstraction layer in PHP that does about 20% of what you need, perhaps the Java infrastructure running on Jetty gets you 80% of the way there instead of 20%.

But, like I said, I have deliverables I need to deliver and so I need to wrap up this blog post.  There are three questions I still want to answer.

  1. How does this Jetty work wit PHP-FPM?  I expect this to be a slam dunk since PHP-FPM works fine with Nginx.
  2. How can a PHP developer put this all together and give themselves more features than they could ever know what to do with?
  3. I still need to directly answer the question of easy integration of WebSockets and messaging.  I expected that either Jetty + ActiveMQ or Jetty + Redis will provide an out-of-the-box(ish) solution.

 


Metaphysics and Software Design

Leave a comment

When I say that this is an experimental blog post I mean it.  I’m thinking out loud here.

Those who know me know that I am somewhat critical of some modern advancements in software design.  It’s not that I’m a Luddite or anything but the root of my discontent is that software and real life often do not match well.  Inbox by Google is a good example of this.  They refuse to put an “unread” feature into their software.  It messes with their blessed workflow.  Never mind that sometimes real life breaks the workflow, like if you accidentally click on an item. In the case of Inbox that accident means that you either have to open up other software (Gmail) or snooze the item you accidentally opened.

That’s an example of a mismatch of UX and reality.  But it also deeper than that.  Essentially any place where there is an interface between the human and the machine, from the user interface all the way to an API, there is going to be a mismatch between what the machine needs and how the human needs to interact with the machine.  And our response is often to make the human match the machine instead of the machine matching the human.  And given that the machine is often the interface between two humans, this seems backwards to me.  But I digress.

Consider the old interface with Magento 1.

class ArbitraryClass
{
    public function doSomething()
    {
         $request = Mage:app()->getRequest()->getQuery();
    }
}

How in the world does this break with metaphysics?  Well, one of the principles of Aristotelian metaphysics is that an object cannot actuate a potential that it does not possess.  In other words a ball cannot make water because it does not contain the potential for water.  But Oxygen and Hydrogen do contain the potential for water.  (Remember we’re talking about METAphysics, not physics.  It is very easy to get hung up on that.)

Going back to the example it could be argued that the code is bad code insofar as it breaks from metaphysics.  ArbitraryClass does not have the potential for a request.  How do we know that?  Because it’s Formal Cause (or Form) is not request-like, or request-ish.  And because there is a dissonance between its formality and efficiency, it is broken insofar as that dissonance occurs.  That’s not to say that it won’t work, but it’s workability will be a function of its alignment with its form.

Consider a triangle.  Have you ever seen a triangle?  No.  Never.  “You’re insane, Kevin,”  You might be thinking.  “I have seen triangles all over the place!”  Yes, you have seen examples of triangles, but they have never been perfect triangles.  A perfect triangle exists only in the immaterial sense.  What you have seen are objects that represent triangularity, but due to slightly curved lines, shifts on the paper, variations in the molecular density of the lead, among other things, you have never, ever in your life seen a triangle.

But you can still recognize a triangle based on its resemblance of triangularity.  In other words, it will still work as a triangle just like our code will still work like our code should.  But the degree of its success will be based on how well it confirms to its reality.

You might be thinking that this is all crap.  But let’s look at a more current implementation.

class ArbitraryClass implements DoSomethingInterface
{
   protected $request;

    public function __construct(RequestInterface $request)
    {
        $this->request = $request;
    }

    public function doSomething()
    {
        $this->request->getQuery();
    }
}

$object = $dependencyInjectionContainer->get('ArbitraryClass');
$object->doSomething();

This example follows the principle that an object cannot give what it does not have.  As a result it is also “better” code.

But it also does a lot more than that.  It actually demonstrates some of the fundamental parts of Aristotle’s philosophy.

  1. It has an Essence or Form.  The DoSomethingInterface is its form, like triangularity.
  2. It does not have “request-ability” and so it must be given by something that does (the Dependency Injection Container).
  3. It has its material cause in its instantiation.
  4. It has efficient causality.  It “does something”.
  5. It has final causality.  It has a purpose.  (nobody intentionally creates un-purposed code)

And so forth.

This does not mean that you are required to code according to philosophy.  What it does seem to show is that the quality of code can be measured against how well the code fits against a logical metaphysical framework.

Bear in mind that this idea is new to me and there could very well be some disqualifying concept that renders this moot.  I’m, in a sense, debating myself in public.

But the flip side of this is that if this is correct, and good software does seem to follow good metaphysical flows, that may be able to be a framework for people to write code that is not only better, but works better in the real world.

But then again, this would turn programmers into Philosophers and God help us if that happens.  🙂


Validating Individual Shipping Methods

Leave a comment

Out of the box Magium will select the first shipping method on the screen. But there might be some times when you need to validate a particular shipping method, or force an error to be thrown if a shipping method does not exist. ’tis now easy to do. use MagiumMagentoAbstractMagentoTestCase; class ShippingTest extends AbstractMagentoTestCase { … Continue reading “Validating Individual Shipping Methods”

Powered by WPeMatico