Stop me if this sounds familiar.

1. Hey look, a Magento Connect extension, let’s try it out

2. Oh, right, I need to be logged in to get the extension key

3. Where’s that password again? [knocks over pills]

4. Waiting for Magento site to log me in

5. [Still waiting]

There are, of course (I assume), reasons for Magento to force a sign-in before letting you grab a free extension key. That said, four plus years (!) in it’s still one of the things that annoys me about the platform. So here’s a bookmarklet that’ll extract the Connect 2.0 key from an extension page, even if you’re not logged in.

Visit the bookmarklet landing page, drag the Connect Key link to your bookmarks bar, click on it while you’re looking at a Magento Connect page, and a text field with a link will be appended at the top of the page. Tested in Chrome/OS X, should work elsewhere, bug reports welcome Your millage mary very, PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, etc.

Maybe if I write this down I’ll remember it.

You probably know all Magento admin console URLs need a nonce/key in the URL

http://magento.example.com/admin/catalog_product/edit/id/174/key/c4df66cd2118cb5422c9fb5eff7eq4f0/

That’s why we use the Mage::getModel('adminhtml/url') model object to generate URLs.

What I always forget is any POST to Magento’s backend also needs a form_key variable. Double the CSRF protection. Without this a POST will be redirected to the dashboard. You can generate a form_key with

Mage::getSingleton('core/session')->getFormKey()

This article is part of a longer series covering the n98-magerun power tool

Now that we’ve got a build environment up and running, we can get to work creating our first n98-magerun command. Our end goal for today is to add a helloworld command to n98-magrun

$ ./n98-magerun.phar list

//...

Available commands:
  helloworld                       Displays a Hello World message. (pedagogical

Development Stub

So far, all our examples have used the n98-magerun.phar bundled application. Therefore, our first step today is to run the application from source code. Every phar archive can have an optional “stub” file. This file is meant to encourage clean coding practices in your phar archive, with the stub acting as a main entry-point into your phar based application. Think of it like you would your index.php bootstrap file in a web application, or a main function in a c program.

The n98-magerun.phar bootstrap file is _cli_stub.php. You might think we could run this ourselves. However, running

$ php _cli_stub.php

will produce the error

PHP Fatal error:  Uncaught exception 'PharException' with message 'internal corruption of phar

Let’s look at the contents of the phar

#!/usr/bin/env php
<?php

Phar::mapPhar('n98-magerun.phar');

$application = require_once 'phar://n98-magerun.phar/src/bootstrap.php';
$application->setPharMode(true);
$application->run();

__HALT_COMPILER();

It turns out that phar stub files are meant to be used from phar archives only. The call to mapPhar, the require using a phar:// url scheme, and the __HALT_COMPILER() call all produce errors if used outside of a phar archive.

We’re going to need to create our own stub file. In the root of your repository, create the following file with the following contents

<?php
$application = require_once realpath(dirname(__FILE__)) . '/src/bootstrap.php';
$application->setPharMode(false);
$application->run();    

This is almost the same code that’s the the phar stub, but written with generic PHP.

Let’s try running this from the command line.

$ php dev-stub.php

     ___ ___
 _ _/ _ ( _ )___ _ __  __ _ __ _ ___ _ _ _  _ _ _
| ' \_, / _ \___| '  \/ _` / _` / -_) '_| || | ' \
|_||_/_/\___/   |_|_|_\__,_\__, \___|_|  \_,_|_||_|
                           |___/
n98-magerun version 1.63.0 by netz98 new media GmbH

Usage:
  [options] command [arguments]

...

Eureka! We’re now running n98-magerun strictly from code. You can make sure everything’s working correctly by navigating to your Magento folder and running a simple command like sys:info.

$ cd /path/to/magento
$ php /path/to/n98-magerun/dev-stub.php sys:info

  Magento System Information  

Version                  : 1.7.0.1
Edition                  : Community
Cache Backend            : Zend_Cache_Backend_File
Cache Directory          : /Users/alanstorm/Sites2012/magento1point7pointzeropoint1.dev/var/cache
Session                  : files
Crypt Key                : 18cea96f92f35a540d83a2fe7f33c005
Install Date             : Sun, 01 Jul 2012 17:06:09 +0000

We’re now ready to start with our helloworld command.

Hello World

To create a new n98-magerun command there’s five basic steps

1. Register the command

2. Create the command class

3. Configure the command name

4. Implement the command

5. Build our new command into the phar

By the end of the article, you’ll know how to do all five.

Registering the Command Class and Namespaces

Our first task is to register the command class. Open up the main application file and look at the registerCommands method

#File: src/N98/Magento/Application.php
//...    
class Application extends BaseApplication
{    
    //...
    protected function registerCommands()
    {
        $this->add(new GenerateLocalXmlConfigCommand());
        $this->add(new DatabaseDumpCommand());
        $this->add(new DatabaseDropCommand());        
        //...
    }
    //...    
}    

The n98-magerun team made use of Symfony’s console package as the basis for their application. A Symfony console application’s add method allows client programmers to tell their application about a command, which in turn exposes it to end users of the console application.

A few of you may be wondering why this code uses such generic class names. The name Application, and BaseApplication seem like things that could be used in other code bases — so why did n98-magerun choose Application and why did Symfony choose BaseApplication?

The short answer is namespaces. If you look at the top of this file

#File: src/N98/Magento/Application.php
namespace N98\Magento;
//...

you’ll see we’re working in the N98\Magerun namespace. That means this application class’s full name is actually

N98\Magento\Application

Similarly, further down in the file, you can see (nestled in among many similar calls)

use Symfony\Component\Console\Application as BaseApplication;

This aliases the class with the full name of

Symfony\Component\Console\Application

as the class BaseApplication in this file.

PHP namespace programming can be confusing if you’re not used to it. Even if you are used to it, the lack of a gold standard in how to approach namespaces can leave you feeling as though you’re flailing about. While this isn’t necessarily a namespace tutorial, we’ll try to explain how and why each classes interacts in this brave new namespaced world.

The namespace detour completed, let’s add our command! Add the following line to the registerCommands method.

#File: src/N98/Magento/Application.php
//...    
class Application extends BaseApplication
{    
    //...
    protected function registerCommands()
    {
        $this->add(
            new HelloWorldCommand()
        );
        //...
    }
}

What we’ve done here is pass in an object, instantiated from the class HelloWorldCommand. Let’s run our command with this in place

$ php /path/to/n98-magerun/dev-stub.php    
PHP Fatal error:  Class 'N98\Magento\HelloWorldCommand' not found    

A fatal error! PHP is complaining it can’t find a class named HelloWorldCommand. Actually, it’s complaining it can’t find a class named N98\Magento\HelloWorldCommand since we’re working in the N98\Magento namespace. Regardless, this make sense since we haven’t actually created our class yet. Let’s get to that.

Creating the Class

Each command in n98-magerun corresponds to a single PHP class file. These files are, by convention, created in the src/N98/Magento/Command folder.

If you list out the contents of this folder, you’ll see a number of classes and directories

$ ls -1 src/N98/Magento/Command
AbstractMagentoCommand.php
AbstractMagentoStoreConfigCommand.php
Admin
Cache
Cms
Config
ConfigurationLoader.php
Customer
Database
Design
Developer
Indexer
Installer
LocalConfig
MagentoConnect
OpenBrowserCommand.php
PHPUnit
ScriptCommand.php
SelfUpdateCommand.php
ShellCommand.php
System

We’re going to create a new class file in this folder. The n98-magerun autoloader expects file names and class names to match, so create the following file at the following location

#File: src/N98/Magento/Command/HelloWorldCommand.php
<?php
namespace N98\Magento\Command;    
use N98\Magento\Command\AbstractMagentoCommand;

class HelloWorldCommand extends AbstractMagentoCommand
{
}    

The first line of this file puts us in the

N98\Magento\Command

namespace. That means the full name for the HelloWorldCommand class we’ve declared is actually N98\Magento\Command\HelloWorldCommand. You’ll notice the namespace convention matches the file path convention.

N98/Magento/Command/HelloWorldCommand.php
N98\Magento\Command\HelloWorldCommand

Without getting too deeply into it, the autoloader depends on this, so make sure you stick to this convention.

Our HelloWorldCommand class extends the AbstractMagentoCommand class. This is the abstract class that all n98-magerun commands inherit from. The full name of this class is actually

N98\Magento\Command\AbstractMagentoCommand

We’re able to refer to it as AbstractMagentoCommand since the following code was used at the top of our file

use N98\Magento\Command\AbstractMagentoCommand;

If we run our application with the above class in place

$ php /path/to/dev-stub.php
Class 'N98\Magento\HelloWorldCommand' not found

We’re still getting the same error. That’s because our register function is trying to load a class in the current namespace of the Application file. If we change our add method call so it uses the full class name

#File: src/N98/Magento/Application.php
//...    
class Application extends BaseApplication
{    
    //...
    protected function registerCommands()
    {
        $this->add(
            new \N98\Magento\Command\HelloWorldCommand()
        );
    }
}

we should be good to go. Notice the leading \\ on the class name. This tells PHP to start looking from the global namespace, (vs. looking for a deeper namespace, starting at the current one) .

If we run our stub with the above in place

$ php /path/to/dev-stub.php
PHP Fatal error:  Uncaught exception 'LogicException' with message 'The command name cannot be empty.

Sort of Eureka! We’ve gotten rid of the class not found error, and replaced with with a new “the command name cannot be empty” error. That’s progress of a sort, and brings us to step 3, configure the command name.

Update: Alexander Menk got in touch to point out the project ships with a stub file that works in a PHP (vs. phar) environment. We regret the error — feel free to substitute this file for your own.

Configuring the Command

Every command has a configure method which is called automatically. This is where we’ll want to assign our command a name. Add the following method to your command class

#File: src/N98/Magento/Command/HelloWorldCommand.php
<?php
namespace N98\Magento\Command;    
use N98\Magento\Command\AbstractMagentoCommand;

class HelloWorldCommand extends AbstractMagentoCommand
{
    protected function configure()
    {
        $this->setName('helloworld');

    }    
}    

With the above in place, lets run our command again

$ php /path/to/dev-stub.php
     ___ ___
 _ _/ _ ( _ )___ _ __  __ _ __ _ ___ _ _ _  _ _ _
| ' \_, / _ \___| '  \/ _` / _` / -_) '_| || | ' \
|_||_/_/\___/   |_|_|_\__,_\__, \___|_|  \_,_|_||_|
                           |___/
n98-magerun version 1.63.0 by netz98 new media GmbH

Usage:
  [options] command [arguments]

Options:
  --help           -h Display this help message.
  --quiet          -q Do not output any message.
  --verbose        -v Increase verbosity of messages.
  --version        -V Display this application version.
  --ansi              Force ANSI output.
  --no-ansi           Disable ANSI output.
  --no-interaction -n Do not ask any interactive question.

Available commands:
  helloworld                       
  help                             Displays help for a command

Finally! We’ve got a clean run. Also, if you’re playing close attention, you’ll notice that helloworld is now listed as an available command.

Available commands:
  helloworld                       
  help                             Displays help for a command

If you want to give your command a description, just call the setDescription method in configure

#File: src/N98/Magento/Command/HelloWorldCommand.php
protected function configure()
{
    $this->setName('helloworld');
    $this->setDescription('Displays a Hello World message. (pedagogical)');
}

Give the stub another run, and you should see your command description.

$ php /path/to/dev-stub.php
...
Available commands:
  helloworld                       Displays a Hello World message. (pedagogical)

Notice that the text we included in the “ tag has been automatically made into a parenthetical.

Implementing our Command

We’ve got our stub running clean again — lets press our luck and try calling our helloworld command.

$ php /path/to/dev-stub.php helloworld

  [LogicException]                                                       
  You must override the execute() method in the concrete command class.  

Drats! At least we’ we’ve replaced our gross PHP errors with pretty Symfony errors.

Every command in a Symfony console application needs an execute method. The execute method is where we implement our command logic. To create this method, add the following to your HelloWorldCommand class

#File: src/N98/Magento/Command/HelloWorldCommand.php
<?php
//...
class HelloWorldCommand extends AbstractMagentoCommand
{
    //...        

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $output->writeln("Hello World!");
    }        
}    

As you can see, we’ve added an execute method to our class. The method has two arguments. The first, $input, allows you to access arguments and options passed to your command. The second, $output, allows you to send feedback to the users of your command. You can also see we’ve called the writeln method of the output object to print out our ubiquitous Hello World! message. Let’s give the command a final run and call it a day.

$ php /path/to/dev-stub.php helloworld
PHP Catchable fatal error:  Argument 1 passed to N98\Magento\Command\HelloWorldCommand::execute() must be an instance of N98\Magento\Command\InputInterface, instance of Symfony\Component\Console\Input\ArgvInput given    

Crud. That would have been too easy, wouldn’t it have? PHP is complaining that the first argument to execute failed to match the type safety check. PHP was expecting a N98\Magento\Command\InputInterface, but was provided with a Symfony\Component\Console\Input\ArgvInput. Again, namespaces rear their hydra like heads.

The InputInterface interface we want is actually Symfony\Component\Console\Input\InputInterface. Instead of changing the type check in front of the parameter, let’s import Symfony\Component\Console\Input\InputInterface and Symfony\Component\Console\Output\OutputInterface into the current namespace. Add the following lines just below the namespace declaration.

#File: src/N98/Magento/Command/HelloWorldCommand.php
namespace N98\Magento\Command;

use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;    
//...

With the above in place, we should be able to run our command.

$ php ~/Documents/github_netz98/n98-magerun/dev-stub.php helloworld
Hello World!

There we have it, another Hello World example program added to the world.

Following Conventions

Before we finish things off by compiling our new code into a phar, let’s jump back to our add method call.

#File: src/N98/Magento/Application.php
//...
$this->add(
    new \N98\Magento\Command\HelloWorldCommand()
);

$this->add(new GenerateLocalXmlConfigCommand());    

While this code works, it’s not keeping with the conventions setup by the netz98 team. When they add a command object, they’re not using the full qualified namespace, they’re just using simple class names like GenerateLocalXmlConfigCommand. How do they get away with this?

If you search this file for GenerateLocalXmlConfigCommand, you’ll find your answer. Near the top of the file is the following line

#File: src/N98/Magento/Application.php
use N98\Magento\Command\LocalConfig\GenerateCommand as GenerateLocalXmlConfigCommand;    

Using the use operator, the fully qualified N98\Magento\Command\LocalConfig\GenerateCommand class is being imported into the local namespace as GenerateLocalXmlConfigCommand. Let’s bring our class into the local namespace. Add the following right above the GenerateLocalXmlConfigCommand line

#File: src/N98/Magento/Application.php
//our code
use N98\Magento\Command\HelloWorldCommand;

//their code
use N98\Magento\Command\LocalConfig\GenerateCommand as GenerateLocalXmlConfigCommand;    

and then change your call to add

$this->add(new HelloWorldCommand());

Your command should behave exactly the same, but now you’re matching the current coding conventions used in the project. Some people might write this off as bike shedding, but a consistant codebase helps with team cohesion, and allows new programmers a stable base to learn the fundamentals of a project without having to worry about parsing multiple different coding styles. This is particularly true in the PHP world, since “idiomatic PHP” varies from project to project.

Building a New phar

The only thing left is to build a new phar archive. As we learned last time, our build script is just a single call to

$ phing
...
BUILD FINISHED

Total time: 8.2973 seconds away.  

After running phing, make sure a new phar was created with today’s date and time (May XX XX:XX below)

$ ls -l n98-magerun.phar 
-rwxrwxr-x  1 alanstorm  staff  3617497 May XX XX:XX n98-magerun.phar

Then, try running your command

$ ./n98-magerun.phar helloworld
Hello World!

Congratulations, you’ve just created and built your first n98-magerun command.

I was about to update the n98-magerun article with some positive developments on those phpunit dependencies when, much to my surprise, this happened

All the XML in code blocks is gone. Vanished. Something something other people’s free blogging systems.

Fortunately the post is mostly available via the V1 API (with entity encoded HTML), but that’s still some pretty weak sauce. I for one do not welcome our always available in the cloud closed source applications.

This article is part of a longer series covering the n98-magerun power tool

Today we’re going to run through setting up your own n98-magerun development environment. This will allow you to develop your own features and contribute bug fixes back to the main repository. This one’s a little longer than our normal Magento Quickies fare, so if you’re in a hurry you may want to save this for later.

There’s two (or maybe three) extra software packages you’ll need to install if you want to work with the n98-magerun source code. The first is PHP Composer, the second is the phing build system.

Composer

Composer is a software package for managing project dependencies. The n98-magerun program relies on many third party packages for its functionality. Rather than include these in the main source tree, n98-magerun includes a top level vendor/ folder. This is where third party dependencies are downloaded to.

This is a standard best practice in profesional software development. By separating your vendor packages from the main source tree, you make upgrading the packages easier. You also encourage/force a “no hacking the vendor package” discipline on your team.

There’s instructions for installing Composer on their website. The short version is there’s a composer.phar (similar to n98-magerun.phar) archive that needs to be in your shell path and needs to be executable. The composer team has a “download install script with curl and run it” one liner, or you may download the phar file directly and place it in your path.

You can test if composer is installed correctly by running the following

$ composer.phar --version
Composer version 0209bd31a0ac3aeb2a68fc81e2d03c71072bef33

Phing

Phing is a PHP build system based on apache ant. For The Kids™ in the audience, ant is a system that was created to give developers a “pure java” build system (as opposed to the powerful-but-tortuous options of the day like GNU make). Phing, in turn, is a PHP version of ant.

Phing implements an XML based build language that allows you to perform common build tasks. This XML format is an example of a Domain Specific Language (DSL). Folks using Magento’s layout system may get a sense of Déjà vu.

Installing phing is slightly trickier than installing PHP composer. The preferred way of installing phing is via PHP PEAR. PEAR (the third package we mentioned above) is PHP’s original dependencies and package management system. While powerful, it never became consistently adopted by the disperate PHP communities. A side effect of this is many PHP distributions don’t have PEAR properly installed, or have it only available to users with sudo permission.

If a fully operational PEAR isn’t available to you, phing is also available via a package download, as well as a stand alone phar file.

Regardless how you install it, at the end of the day you’ll have a phing (or phing.phar or phing-latest.phar) command executable from your shell.

$ phing -v
Phing 2.5.0

Downloading Dependencies with Composer

Now that our tools are in place, we’re ready to start. Step 1 for working with n98-magerun is to get the codebase from GitHub.

$ git clone https://github.com/netz98/n98-magerun.git

However, even after you’ve cloned the git source tree, you’re still missing a number of files n98-magerun needs to function. If you look in the vendor folder, you’ll see it’s empty

$ ls -la vendor/
total 16
drwxr-xr-x   3 alanstorm  staff   102 Apr 26 14:40 .
drwxr-xr-x  26 alanstorm  staff   884 Apr 25 23:24 ..

As mentioned earlier, the vendor folder is where third party code libraries are kept. To download these files, run the composer.phar install command from the root folder of your repository.

$ cd n98-magerun
$ composer.phar install
Loading composer repositories with package information
Installing dependencies from lock file
  - Installing symfony/process (v2.2.1)
    Loading from cache

  - Installing symfony/finder (v2.2.1)
    Loading from cache

  - Installing symfony/console (2.2.x-dev a10a2fb)
    Cloning a10a2fb156267b9b8f01d717a06979e6ebb69251

  - Installing seld/jsonlint (1.1.1)
    Loading from cache

  - Installing justinrainbow/json-schema (1.1.0)
    Loading from cache

  - Installing composer/composer (dev-master c655674)
    Cloning c65567447c84eb6a8c0f229c40b0865cd4dcecb5

  - Installing fzaninotto/faker (v1.1.0)
    Loading from cache

  - Installing n98/junit-xml (dev-master 6381433)
    Cloning 6381433a39498dc22bec60cebdb2576a9d2327b0

  - Installing symfony/translation (v2.2.1)
    Loading from cache

  - Installing symfony/validator (v2.2.1)
    Loading from cache

  - Installing symfony/yaml (v2.2.1)
    Loading from cache

symfony/translation suggests installing symfony/config (2.2.*)
symfony/validator suggests installing doctrine/common (~2.2)
symfony/validator suggests installing symfony/config (2.2.*)
symfony/validator suggests installing symfony/http-foundation (2.2.*)
symfony/validator suggests installing symfony/locale (2.2.*)
Generating autoload files

If you take a look at your vendor folder after running the install command, you’ll see the required vendor packages have been installed.

$ ls -1a vendor/
.
..    
autoload.php
bin
composer
fzaninotto
justinrainbow
n98
seld
symfony

Of course, this raises a few questions: How does composer.phar know which packages to download? And where are these packages coming from? While this knowledge isn’t strictly required to procede, we’re going to cover it to put your (and our) minds at ease.

The composer.phar command looks for a file named composer.json. This file contains configuration information for a composer based project (which n98-magerun is). This configuration information will include a list of package dependencies

//File: composer.json
"require": {
    "php": ">=5.3.3",
    "symfony/console": "2.2.x-dev",
    "symfony/finder": "2.2.1",
    "symfony/yaml": "2.2.1",
    "symfony/process": "2.2.1",
    "symfony/validator": "2.2.1",
    "justinrainbow/json-schema": "1.1.*",
    "seld/jsonlint": "1.*",
    "n98/junit-xml": "dev-master",
    "fzaninotto/faker": "1.1.0",
    "composer/composer": "dev-master"
},

As you can see, the require key contains a list of packages and versions. These package names correspond with the downloaded packages from the install command. So this answers the “how does composer know” question. Before we move on the “where” question, it’s worth pointing out that composer is actually reading from the composer.lock file. This composer.lock file is generated with the information in composer.json, and is not meant to be human editable. If you want to get into the nitty gritty of this checkout the composer manual.

The “from where” question is a little trickier. There’s two grand arcs to the composer project. The first is the composer.phar application we downloaded, which contains the logic for dependency management. The second is a composer repository at packagist.org. Unless a different repository is listed in composer.json, packagist.org is where composer.phar will grab package information from.

Does that mean composer is downloading packages from packagist.org? No. Confused? Read on!

A composer repository only contains meta-data about a package. This meta-data includes a source code repository where the package source can be found. The packagist.org repository simply tells composer.phar where it should download the package source from.

Figuring out where composer.phar is grabbing packages from is a two step process. First, you can list out all the packages used in your project with the show command and and --self flag.

$ composer.phar show --self
name     : n98/magerun
descrip. : Tools for managing Magento projects and installations
keywords : magento, installer
versions : * dev-master
type     : library
license  : MIT
source   : []  
dist     : []  
names    : n98/magerun

support
issues : https://github.com/netz98/n98-magerun/issues

autoload
psr-0
N98 => src
Composer => src
Xanido => src

requires
php >=5.3.3
symfony/console 2.2.x-dev
symfony/finder 2.2.1
symfony/yaml 2.2.1
symfony/process 2.2.1
symfony/validator 2.2.1
justinrainbow/json-schema 1.1.*
seld/jsonlint 1.*
n98/junit-xml dev-master
fzaninotto/faker 1.1.0
composer/composer dev-master

requires (dev)
phpunit/phpunit 3.7.19
phpunit/phpunit-story 1.0.2
phpunit/phpunit-selenium 1.2.12
phpunit/dbunit 1.2.3
phpunit/php-invoker 1.1.2

Once you have a package name (such as symfony/yaml), you can see its repository by using the show command again, this time with the package name as an argument.

$ composer.phar show symfony/yaml
name     : symfony/yaml
descrip. : Symfony Yaml Component
keywords : 
versions : dev-master, 2.3.x-dev, 2.2.x-dev, * v2.2.1, v2.2.0, v2.2.0-RC3, v2.2.0-RC2, v2.2.0-RC1, v2.2.0-BETA2, v2.2.0-BETA1, 2.1.x-dev, v2.1.9, v2.1.8, v2.1.7, v2.1.6, v2.1.5, v2.1.4, v2.1.3, v2.1.2, v2.1.1, v2.1.0, v2.1.0-RC2, v2.1.0-RC1, v2.1.0-BETA4, v2.1.0-BETA3, v2.1.0-BETA2, v2.1.0-BETA1, 2.0.x-dev, v2.0.23, v2.0.22, v2.0.21, v2.0.20, v2.0.19, v2.0.18, v2.0.17, v2.0.16, v2.0.15, v2.0.14, v2.0.13, v2.0.12, v2.0.10, v2.0.9, 2.0.7, 2.0.6, 2.0.5, 2.0.4
type     : library
license  : MIT
source   : [git] https://github.com/symfony/Yaml.git v2.2.1
dist     : [zip] https://api.github.com/repos/symfony/Yaml/zipball/v2.2.1 v2.2.1
names    : symfony/yaml

autoload
psr-0
Symfony\Component\Yaml\ => .

requires
php >=5.3.3

The two sections we’re interested in here are source and dist. The source lists the repository where the source code should be downloaded from, (git, svn, and hg repository types are supported). This is how composer.phar will grab the needed vendor packages.

The archive (as opposed to repository) at dist can be used to install a package instead, but only if the --prefer-dist flag is used with the install command.

If you’re paranoid about security, you’ll want to review these package locations — although at some point using a distributed packaging system like CPAN, ruby gems, or packagist requires a certain level of trust. That, however, is another subject better left to another time.

Composer Development Packages

There’s one last thing we should cover before moving on from composer. If you examined the composer.json file carefully, you may have noticed this section

//File: composer.json
"require-dev": {
    "phpunit/phpunit": "3.7.19",
    "phpunit/phpunit-story": "1.0.2",
    "phpunit/phpunit-selenium": "1.2.12",
    "phpunit/dbunit": "1.2.3",
    "phpunit/php-invoker": "1.1.2"
},

This appears to be a second list of packages. It turns out composer projects can have two sets of required packages. The first is the standard distribution you’ll need to use the package. The second, require-dev, is an additional set of packages you’ll need to work on (or develop) the project.

Since this whole article is about building our own version of n98-magerun.phar, we’ll want to install these development packages. Let’s run composer.phar with the --dev flag.

$ composer.phar install --dev
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for phpunit/php-invoker 1.1.2 -> satisfiable by phpunit/php-invoker 1.1.2.
    - phpunit/php-invoker 1.1.2 requires ext-pcntl * -> the requested PHP extension pcntl is missing from your system.

Uh oh. At this point, you may (as we did above) run into an intractable problem. The phpunit/php-invoker package couldn’t be installed because it requires that PHP have the process control (ext-pcntl) extension installed. This extension is omitted from many standard PHP distributions.

We’ve now run into what I like to call the Software is Hard problem. In order to install these dependencies, we’d need to recompile our version of PHP. That’s something many people can’t, or won’t do. Unless we’re willing to make a massive change to our environment, we won’t be able to install the full suite of required phpunit tools.

On the flip side, we could remove phpunit/php-invoker as a requirement, and then successfully install the other development packages. However, this would mean we’re veering off from what the main team is doing. Our system may start to behave differently than theirs, causing subtle and hard to track down compatibility problems in the future.

There’s going to be times in your career where you run into problems where there is no ideal solution. The only option is to raise your concerns, and then press on with the less than ideal conditions. The trick to a long career is finding a way to not let these small defeats sap your motivation for other, more important tasks.

Fortunately, it’s still possible to work on n98-magerun code without the full phpunit suite, so we’ll carry on without it.

Update: From the sounds of this GitHub thread, the phpunit/php-invoker requirement will be dropped in the 1.64 release, so the previous whinging may not apply if you’re reading this in the future.

Phing

Now that we’ve discussed composer, our next step is using phing to build the project. Take a look at the build.xml file in the root of the project

<!-- File: build.xml -->
<project name="n98-magerun" default="dist">

    <fileset dir="." id="root_folder"><!-- ... --></fileset>
    <fileset dir="src"  id="src_folder"><!-- ... --></fileset>
    <fileset dir="res" id="res_folder"><!-- ... --></fileset>
    <fileset dir="vendor" id="vendor_folder"><!-- ... --></fileset>

    <target name="dist">
        <!-- ... -->
    </target>

    <target name="dist_unix">
        <!-- ... -->
    </target>

    <target name="install">
        <!-- ... -->
    </target>

</project>

This is our phing build file. It contains all the build instructions for our package. We’re interested in the <target/> nodes. Each <target/> node defines a single build command. For example, the install command can be used to install the n98-magerun.phar from the repository to your local computer. This is a relatively simple build target.

<target name="install">
    <exec command="sudo cp ${project.basedir}/n98-magerun.phar /usr/local/bin/n98-magerun.phar;" />
    <exec command="sudo chmod a+x /usr/local/bin/n98-magerun.phar;" />
</target>

As you can see, there’s two <exec/> nodes within this target. These are command nodes — think of each one like you would a single line in a normal programming language or shell script. The <exec/> command is one of the simplest build commands — it simply runs a unix shell script.

Let’s try running the install target.

$ phing install
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > install:

Password:
Sorry, try again.
Password:

BUILD FINISHED

Total time: 5.9880 seconds

As you can see, to run a phing command you simply pass it the target name. The phing command will look for a build.xml file in the current directory, and then run the target you’ve passed in.

As for the command output itself, it prompts you for a password (since it’s running the sudo command), and the finishes the build. Unfortunately, there’s no indication of what the command actually did. If you’d like a little more feedback, try the verbose flag.

$ phing -verbose install
Buildfile: /path/to/n98-magerun/build.xml
Override ignored for user property phing.file
Override ignored for user property phing.dir
parsing buildfile build.xml
Project base dir set to: /path/to/n98-magerun
Build sequence for target 'install' is: install 
Complete build sequence is: install dist dist_unix 

n98-magerun > install:

Property ${project.basedir} => /path/to/n98-magerun
     [exec] Executing command: sudo cp /path/to/n98-magerun/n98-magerun.phar /usr/local/bin/n98-magerun.phar; 2>&1
     [exec] Executing command: sudo chmod a+x /usr/local/bin/n98-magerun.phar; 2>&1

BUILD FINISHED

Total time: 0.1772 seconds

With the verbose flag, phing tells you everything it’s doing. From this we can figure out a file has been copied to /usr/local/bin/n98-magerun.phar. This places the phar in our *nix path, and is another way to install the n98-magerun.phar command.

Building the Project

Most phing build files have a default target which builds the project.

If you run phing with the -l option

$ phing -l
Buildfile: /path/to/n98-magerun/build.xml
Default target:
-------------------------------------------------------------------------------
 dist

Subtargets:
-------------------------------------------------------------------------------
 dist
 dist_unix
 install

you’ll get a list of each target in the application, as well as the default target (in this case, dist). You can also see the default target by looking at the root node of the build.xml file.

<!-- build.xml -->
<project name="n98-magerun" default="dist">

We’re going to run through a default build of the n98-magerun tool, including several problems we ran into and how to work around them. You may not run into these specific problems — and you may run into different problems not mentioned here. Setting up a build environment is always a trial and error process, and things will rarely work out of the gate. Part of the reason teams use a tool like phing is to provide a single entry point into their build system and force developers to download and install the same dependencies. If you run into trouble getting this up and running, the Magento Stack Exchange is a great place to ask questions to help get your system up and running.

To run the default build step, just run phing from the project root directory with no arguments

$ phing

The first thing you’ll see is phing going through the project dependencies and downloading the needed files to the vendor folder.

$ phing
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
  - Installing symfony/process (v2.2.1)
    Loading from cache

  - Installing symfony/finder (v2.2.1)
    Loading from cache

... snipped ...

If you’ve already downloaded these files with the composer.phar install command, these downloads will be skipped. The output for that looks like this

$ phing
Buildfile: /private/tmp/building/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files

If you look at the build.xml file, you can see all this happens because the command in the dist target runs the composer install command.

<target name="dist">
    <exec command="composer.phar install" dir="${project.basedir}" passthru="true" />

So, it would be more accurate to say phing still attempts to update your dependencies, but if the latest versions are already downloaded composer will skip re-downloading them.

Writeable PHAR Archives

After phing runs the composer.phar command, it moves on to the next build steps, and (for some of you) will hit another error.

Execution of target “dist_unix” failed for the following reason: /path/to/n98-magerun/build.xml:45:65: Problem creating package: creating archive “/path/to/n98-magerun/n98-magerun.phar” disabled by the php.ini setting phar.readonly

Here we’ve discovered a constraint of PHP’s phar implementation. By default, PHP won’t let you edit an existing phar archive. While annoying, if you consider that a phar archive is meant to “emulate jars and dlls”, this makes sense. Allowing a program to modify its libraries would be a recipe for disaster.

The problem, of course, is that it’s PHP code that creates a phar archive. In order to solve this chicken/egg problem, PHP has a phar.readonly flag for php.ini. If you set this flag to 0 or Off, PHP will allow you to access the methods which modify a phar archive.

This flag is one you’ll need to set in your php.ini file, it can’t be set at runtime with ini_set. If you’re not sure where your php.ini file is located, try running the following

$ php --ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File:         /private/etc/php.ini
Scan for additional .ini files in: (none)
Additional .ini files parsed:      (none)

Remember, it’s not uncommon for the command line version of PHP to have a different ini file than the web server version of PHP.

BZip Compression Problems

With the above in place, try running the build again. If you’re like me, you’re about to run into another problem

Execution of target “dist_unix” failed for the following reason: /path/to/n98-magerun/build.xml:45:65: Problem creating package: unable to create temporary file [wrapped: unable to create temporary file]

This one is a head scratcher, but I finally tracked down the problem. In short: While PHP’s phar format works across a variety of platforms and environments, the methods for creating phar archives are still very beta-ish. The specific problem here is, when compressing phar archives PHP likes to open a temporary file — and from the sound of things it opens a temporary file for each file in the archive, and doesn’t close them until the command is complete. All that library code means n98-magerun has over 1,000 files.

This quickly hits the limits that a non-server operating system has in place for the number of files each process is allowed to have open. To solve this problem, you’ll need to edit build.xml. Find the following XML node

<!-- File: build.xml -->
<pharpackage basedir="./" stub="_cli_stub.php" signature="sha512" compression="bzip2" destfile="./n98-magerun.phar">    

The pharpackage node is a phing build command that allows you to automatically create a phar application from a PHP codebase. We’re interested in the compression="bzip2" attribute. This tells phing to run the phar archive through the compressFiles method, and is done to reduce the file size of the archive that’s distributed. If you change this attribute to

compression="none"

then phing will skip the compression step, and we should be able to get a complete build.

$ phing
Buildfile: /path/to/n98-magerun/build.xml

n98-magerun > dist:

Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
[phingcall] Calling Buildfile '/path/to/n98-magerun/build.xml' with target 'dist_unix'

n98-magerun > dist_unix:

[pharpackage] Building package: /path/to/n98-magerun/n98-magerun.phar
    [chmod] Changed file mode on '/path/to/n98-magerun/n98-magerun.phar' to 775
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file

BUILD FINISHED

Total time: 3.5711 seconds

That looks like a clean build, let’s test out the n98-magerun.phar file that was just created.

$ ./n98-magerun.phar --version
n98-magerun version 1.63.0 by netz98 new media GmbH

Success! However, our n98-magerun.phar file

$ ls -lh n98-magerun.phar
-rwxrwxr-x  1 alanstorm  staff   3.4M Apr 25 15:50 n98-magerun.phar

is over 3 MB in size. Compare that to the 1 MB file distributed in the main github archive. So, while we can’t build exactly what the core n98-magerun team does, we can get a clean build in place with a testable phar, which is good enough for our needs.

While frustrating, you’ll find many points in your career where your development environment won’t match up exactly with that of the team you’re working with. While it’s true (especially among non-distributed teams) that a chaotic build environment can lead to development delays, these delays are subtle and complicated in nature, and not the sort of thing that shows up in ROI metrics. Unfair as it may be, as an individual developer working with a group, it will often fall on you to solve these problems for yourself in a way that doesn’t disrupt team flow.

Wrap Up

Regardless of the subtle incompatibilities, we’ve successfully built the n98-magerun.phar file from source. This means we can start hacking on bugs, as well as adding new features to the tool. Next time we’ll create our very first n98-magerun command.