This article is from the development perspective. For end-user documentation, please read the plugin installation instructions.
The Downloadable Plugins project is a framework for Gallery 2 that enables users to browse a central repository of plugins (both modules and themes) on the project's web server through the Site Administration, check if their modules are up to date, download compatible modules they want and prepare them for installation. It includes tools for the repository maintainers.
Currently, Gallery can be downloaded in several packages, each one containing a set of plugins tailored for a specific user profile. The problem with this approach is that adding plugins not included in the initially downloaded package requires knowledge of archiving utilities and at least a basic understanding of FTP or shell commands. A plugin packaging system which relies on a central repository streamlines this process by allowing users to browse available plugins through a friendly user interface. Users can then simply download and install only the plugins they want to use. It also makes it possible for users to check if there are any newer versions of their favorite modules or themes and upgrade. All plugin installation-related tasks are handled by Gallery, which greatly reduces plugin installation time and prevents version conflicts, because the compatibility of plugins with the installed version of the Gallery core module is automatically determined. It also makes it possible to easily download translations and keep them up-to-date.
Each plugin can be packaged in several ways, to give different profiles of users options that best suit them. For example, developers can download plugins with unit tests included, which are unnecessary for regular users. Some users will want language packs, other will not. This system enables them to download only the required parts of a plugin and nothing more.
The package format does not depend on additional server libraries (e.g. tar, zip, gzip), so its format is relatively simple. It is composed of a descriptor file, which contains metadata about the plugin; and multiple package files, which contain specific parts of a plugin. Since it was decided to create a PHP-based package, the descriptor format is also PHP based. This makes it possible to get the metadata with a single include, skipping the step of interpreting an INI or other type of file.
Every descriptor file contains five arrays: $header, $descriptions, $directories, $files and $languages. By including the descriptor, we automatically get all metadata for a given plugin. This format is extendable -- new data can be easily added, and compatibility is preserved as long as older metadata definition isn't changed. $header specifies the plugin type, ID, version, required API versions, base language (strings.raw) revision and build timestamp. $descriptions contains translated plugin names, descriptions and group names in multiple languages. $directories contains the entire tree of subdirectories beneath a plugin's main directory. $files contains file metadata that is used to verify that files have been properly downloaded. Both $directories and $files include the name of the package each file or directory belongs to. $languages contains the latest available .po revision for each language, for all available strings.raw revisions (see Versioning and Examples for more information).
A descriptor file for a plugin can be created automatically with the repository administration tools. All that is required is that the plugin is properly installed, i.e. recognized by the Gallery API functions. The GalleryModule and GalleryTheme classes already provide almost all of the information that is necessary for creating the descriptor, including the required core and module/theme API versions. This format can be extended to include an archived and/or compressed version of each variant for clients that have appropriate server libraries.
Since we don't want to depend on server libraries, such as tar or zip, a self-extracting PHP-based package format was created. It contains base64-encoded file data. This makes the data grow by ~33%, but since we offer flexibility with variants and the packages are not very big, this should not be an issue. It is also binary-safe and has no Unicode issues. The body of the package is enclosed in an unpack($outputDir) function, which can take a destination directory as an argument. After the unpacking has been completed, the integrity of the unpacked files is checked. It is done by comparing each unpacked file's size and hash with the descriptor metadata. If they match and all files and directories are present, we can be satisfied that the package has been successfully unpacked.
Now that there are separate G2 packages and individual, downloadable plugins, there will be many more people updating individual modules without getting a new version of G2. In the current system, if the version number of a plugin is bumped, it forces the site admin to upgrade that plugin. So the version number is only bumped in the cases where an upgrade is necessary (like an API change, or a new data type). If a small change to a template is made, users should not be forced to upgrade the module. The problem is that if we start automatically making nightlies of the plugins and we don't bump the version number on changes, we could have two nightly packages with the same exact filename and version number, but with different contents. This will cause confusion.
The solution that was chosen for non-language packages is to include build timestamps. In order for the timestamp to work, it is the date of the most recent change to any file inside the plugin. That way even if one byte is changed in a template, a new, unique, file name is generated. Timestamps are drawn from the CVS data because it's the only guaranteed timestamp source. The main reason this approach was chosen over the alternatives is that it doesn't introduce any additional tasks for plugin developers and makes detecting changes easy. This solution also ensures that two different package repository building systems will generate the same exact packages each time, and packages can be generated historically.
The versioning system for language packages is different. Language packages differ from other packages in the way that they don't depend on package version numbers and build timestamps, but on revisions of the strings.raw files. This means that the same translation can work with many different plugin builds. So each packaged language version consists of the strings.raw revision it is intended for and also the revision of the language .po file. These two revision numbers enable G2 to find the latest version of each language package for the strings.raw in the user's installation. Revisions are managed by the CVS. $Id$ tags were added to the .po and strings.raw files that are automatically updated with the latest file revision number on each commit. This solution also works with multiple supported branches.
Below is an overview of the package types that currently exist, and a couple that may be included in the future. It's not noted in the table, but CVS directories are not included in any package.
| Type | Description | Contents |
|---|---|---|
| base | Full plugin functionality with English (US) strings. | Everything not included in the other packages.1 |
| test | Unit tests for developers. | test/* |
| language | Translation in a single language. | locale/xx_XX/* po/xx_XX.po2 |
| developer | Developer add-on packs. | To be discussed. |
| packs | Color/icon/frame packs. | To be discussed. |
1 Includes the GNUmakefile and strings.raw.
2 Most people won't need the po file, but it's easiest to just send that along also so that they can do i18n maintenance if they want/need to.
Earlier, there was a concept of variants, which were a selection of packages. The idea was to only present variants to the user (e.g. bare plugin, plugin for developers, etc.). One of the main reasons this concept was abandoned is that each language should be separately downloadable, which means that there would be many variants with only one language included. This would make most variants equivalent to packages. The main advantage to variants was that they hid the complexity of the underlying package system, but the same can be achieved by creating a good user interface. See the end-user tools for more information on how downloading packages functions from the user's point of view.
One feature that is missing, but should be included initially, is support for digital signatures. I will describe how it will work, so we can discover any flaws or make improvements to the design before the implementation begins.
Currently, hashes of all files in a package are stored in the package descriptor, but the descriptor and packages themselves are not signed. The simplest and most compatible approach would be to simply append an MD5 (or some other one-way hash function) sum to them. This would make sure that they have not been accidentally damaged during transfer, for example. Obviously, someone could change the packages, recalculate the sum and replace the original. But this should not be a problem until there are untrusted mirrors packages can be downloaded from.
A better way, from a security standpoint, is to use a public key cryptography based algorithm to create a "real" digital signature that can be used to verify that the packages have not been tampered with. The downside to this method is that we require an external library to do this work. It may be possible to create a PHP-based replacement, but as these algorithms are computationally costly, I don't think it's a very good idea. I have not researched this option in detail, though. For our purposes, the best way to implement signatures is with OpenSSL, which is widely available, contains all the functionality we need, and is supported by PHP.
If the library is installed and correctly configured, Gallery will be able to let the user know if the package has been tampered with. For users without the encryption library, we should have a fallback mechanism, in the form of a simple unencrypted hash, so at least the integrity of the downloaded files can be verified. We also let the users know that there is a signature, but we can't check if it is valid and explain why.
This feature can be implemented by modifying the package and descriptor generators to append a hash and a signature to each file they create. Another method is to create an additional file that contains the hash and signature for each descriptor/package, but it would double the number of files that have to be downloaded, even if they are relatively small, so I would go with the first option. On the client side, verifying the package integrity/signature would be an additional step after each file has been downloaded.
If the signature turns out to be invalid, we can raise an error and refuse to install the package. We can also provide an override option that forces the installation, but since the file is probably damaged and a forced installation could leave Gallery in an unknown state, I am personally against such an option.
The repository index's integrity is currently being verified by checking its hash that is in a separate index.md5 file. This can be changed to use a mechanism that is identical to the one described above.
These examples have been formatted for easy viewing, the actual descriptor and package contents are serialized with the PHP serialize() function.
SampleModule-0.1.0-20050828145408.descriptor
<?php
$header = array (
'id' => 'sample',
'version' => '0.1.0',
'requiredCoreApi' => array(6, 0),
'requiredModuleApi' => array(0, 12),
'stringsRevision' => '1.7',
'buildTimestamp' => '20050828145408'
)
$descriptions = array (
'en_EN' => array (
'name' => 'Sample Module',
'description' => 'This is a sample module.',
'group' => 'Group Name'),
)
$directories = array (
'classes' => array (
'package' => 'base'
),
'test' => array (
'package' => 'test'
),
'test/phpunit' => array (
'package' => 'test'
)
)
$files = array (
'module.inc' => array (
'size' => 123456,
'hash' => 'hash here',
'package' => 'base' ),
'classes/class1.class' => array (
'size' => 423565,
'hash' => 'hash here',
'package' => 'base' ),
'test/phpunit/ModuleTest.php' => array (
'size' => 198565,
'hash' => 'hash here',
'package' => 'test' ),
)
$languages = array (
'1.7' => array (
'en_US' => '1.12'
)
)
?>
SampleModule-0.1.0-20050828145408-base.package
<?php
function unpack_zkbrg($outputDir) {
/* Create directory structure. */
@mkdir($outputDir . 'classes');
/* Recreate individual files. */
expand_zkbrg($outputDir, 'module.inc', 'encoded contents here');
expand_zkbrg($outputDir, 'classes/class1.class', 'encoded contents here');
}
?>
function expand_zkbrg($outputDir, $relativePath, $data) {
$fd = fopen($outputDir . $relativePath, 'w');
fwrite($fd, base64_decode($data));
fclose($fd);
}
$unpackFunction = "unpack_zkbrg";
?>
SampleModule-0.1.0-20050828145408-test.package
<?php
function unpack_asdfg($outputDir) {
/* Create directory structure. */
@mkdir($outputDir . 'test');
@mkdir($outputDir . 'test/phpunit');
/* Recreate individual files. */
expand_asdfg($outputDir, 'test/phpunit/ModuleTest.class', 'encoded contents here');
}
?>
function expand_asdfg($outputDir, $relativePath, $data) {
$fd = fopen($outputDir . $relativePath, 'w');
fwrite($fd, base64_decode($data));
fclose($fd);
}
$unpackFunction = "unpack_asdfg";
?>
A repository is a directory accessible through HTTP containing module and theme packages, a repository index and a file containing the index's hash. The repository index is a text file containing the $header, $descriptions and $languages parts of each plugin's descriptor. The only difference is that the index generator adds data for all available strings.raw revisions to the $languages section. This has to be done during index generation, because, unlike the descriptor and package generators which work on individual plugins, it has the knowledge of the repository's contents. The entire contents of the repository are automatically generated with the repository tools.
Repository directory structure:
repository_path/index reposiroty_path/index.hash repository_path/modules/SampleModule-0.1.0-20050828145408.descriptor repository_path/modules/SampleModule-0.1.0-20050828145408-base.package repository_path/modules/SampleModule-0.1.0-20050828145408-test.package repository_path/modules/SampleModule-lang-de_DE-1.8-1.17.package repository_path/themes/SomeTheme-0.9.6-20050829135632.descriptor repository_path/themes/SomeTheme-0.9.6-20050829135632-base.package
index
$modules = array (
'sample' => array ('header' => $header, 'descriptions' => $descriptions, 'languages' => $languages),
)
$themes = array (
'sometheme' => array ('header' => $header, 'descriptions' => $descriptions, 'languages' => $languages),
)
$header, $descriptions and $languages are taken from the descriptor and are not reproduced here.
The repository tools are simple tools for managing the repository. Since end-users will not use them, the repository tools' interface is minimal. They are web-based, utilize the G2 API and reside in gallery2/lib/tools/repository. Their source code structure is very similar to a Gallery module. The only significant difference is a custom RepositoryControllerAndView class which has the role of both GalleryController and GalleryView, but is much simpler. It would not be very difficult to convert the repository tools to a G2 module if we ever want to package them for users who want to maintain their own repositories. The code is covered with unit tests and a patch to the unit test runner includes them in the standard G2 test run.
The tools can be used to package, browse and remove plugins from the repository and generate the index. There is also an option to package all plugins that don't already exist in the repository. Packaging a plugin is done by selecting a single plugin to be packaged or by entering a regexp filter. A package manager creates the selected plugins' descriptors and all possible packages in the repository. After the repository has been changed, the index generator needs to be executed, which scans the repository descriptors and creates a new index and its hash. After this point, users will see the new packages after they do an index update in their administrative tools.
This is the user story that guided the development of the repository tools. It is the simplest fully functional scenario that is initially supported. The idea is to have only the necessary features first, and expand later as new requirements are identified during regular usage.
Alice is the repository administrator. She wants to add a new module to the repository, and to also remove an old one that has not been updated for a long time. She opens the Repository Tools, and is presented with the following options: Package Plugin, Browse Repository and Generate Index. She selects Package Plugin and is presented with a list of available themes and modules that can be packaged. She clicks on Package beside the new one. The plugin directory is parsed and the descriptor and packages are created and placed in the repository. The index file has not been automatically regenerated to allow her to do other tasks first. Alice now wants to remove the old package, so she clicks on Browse Repository and is presented with a list of packages in the repository. She finds the one she wants to remove and clicks on the Remove link beside it. She confirms this on a new page and the package is removed from the repository. Finally, she chooses Generate Index, and the metadata from all available plugins is read and added to a new index file.
The administrative tools for interacting with the repository are located in the Gallery group in the Site Admin. The main features of the tools are:
When the repository tools are accessed for the first time, users are informed that a gallery2/plugins directory needs to be created and its permissions set appropriately. If the user sets up the directory properly and refreshes the page, all additional directories are automatically created and the repository tools view is presented. It is a tab based user interface with the following tabs: Common Tasks, Modules and Themes.
The Common Tasks tab displays information about the local index -- number of modules and themes and the time it was last updated, and has an Update command for updating it. Updating the repository index consists of downloading the index and the index hash file from the remote repository, verifying the integrity of the downloaded index and finally placing the index in the local repository data directory. After this has been done for the first time, the rest of the options are presented.
There is a section that lets the user know whether there is a Gallery core upgrade available. If there is, links to the download page and upgrade instructions are shown. The Gallery core cannot be upgraded through the repository tools interface because it consists of more than just the core module. This may be changed in a future version. It is possible to review the compatibility of the plugins with the new core version in the repository browser.
The final section of this tab is the Upgrade All command, which most users will use to keep their plugins up-to-date. By clicking on this button, all installed plugins and all downloaded translations will be upgraded to the latest available versions.
The Modules and Themes tabs display information about the available plugins in the repository. Their interface is very similar to the current module management interface in the Site Admin. It contains a table with columns for plugin name, local version, repository version, description and actions. By default, only compatible plugins are displayed, but there is a link at the top of the page that displays all available plugins for the latest Gallery core version. Since the API version requirements of each plugin are known, incompatible ones are flagged, and detailed information about the incompatibility cause is written. Depending on whether a plugin is available locally, a download or upgrade action is presented. They are functionally identical -- they both display the download page for the selected plugin. If no upgrades are available for a plugin or a plugin is not compatible with the installed core module, the action columns remains empty.
This view shows what downloads or upgrades are available for the selected plugin. When a plugin is downloaded for the first time, the base package is required and is preselected. If there are any translations available, a list of checkboxes with the names of the languages is displayed. If unit tests exist, the option to download them is shown in a separate section. The available commands are Download and Cancel. The former downloads and installs all of the selected packages. A progress bar shows information about the current activity during this process. Cancel returns the user to the repository browser.
If the selected plugin has been downloaded before, translations that are already installed but have newer versions available in the repository are put in an Upgrade translation section, while translations that have not been downloaded yet are in a separate Download translations section. Similarly, if the base or test packages can be upgraded, there will be options to do so. It is possible to update the translations without upgrading the plugin base files.
Previously, the core module did not support plugins in directories other than g2_base/[modules|themes]. Since the downloaded packages are placed into the plugins directory, the core needed to be modified to add support for using plugins in both this directory and the default directory. The changes that needed to be made in order to implement this functionality:
Originally, the plan was to put downloaded plugins in g2data/plugins, but while working on adding support for themes in g2data/plugins/themes, I realized that there was a hidden complexity I wasn't aware of. Support for themes in g2data was relying on the assumption that Apache can access it, which is of course wrong. What needed to be implemented to make themes work in g2data was routing of theme data through Gallery. It was decided that this would be too complex, and the decision was made to move the downloaded plugins directory to g2_base_directory/plugins.
Outstanding issues:
There are several new files and directories used by Gallery after the downloadable plugins code has been added. Their paths relative to the G2 base directory and purpose are given in the table below.
| Path | Purpose |
|---|---|
| plugins/modules/ plugins/themes/ |
Unpacked downloaded plugins. |
| plugins/index.modules plugins/index.themes |
List of modules and themes located in the plugins directory. |
| g2data/plugins_data/modules/core/repository/index.repository | The downloaded repository index. |
| g2data/plugins_data/modules/core/repository/cache/ | Downloaded packages. |
Although the originally proposed repository feature set has been fully implemented, there are a lot of possibilities for expanding it. This is a list of possible features that can be worked on, in no particular order.
