Gallery2:Progressbar - Gallery Codex
Personal tools

Gallery2:Progressbar

From Gallery Codex

Progress Bar in Gallery 2

Progress bars are used in Gallery for long-running operations. The two principal reasons to use them are:

  • To guarantee that the request handling is not terminated by the webserver.
  • To give the user feedback on the current state of the progress of the operation.

PHP scripts running on a webserver can be killed either because the PHP maximum execution time has been exceeded or because the webserver terminates the execution. We combat the PHP execution time limit with dynamic calls to $gallery->guaranteeTimeLimit(60); which calls set_time_limit(60) to ensure that the script can run another 60 seconds. But we can't control the webserver with PHP functions that way. We need to keep the connection between the client and the webserver alive, else we are terminated. And we need to signal the webserver that we're still doing stuff, else the webserver times out the request handling (default of apache is after 5 minutes). Both can be achieved by sending some output from the server to the client every once in a while. This resets apache's timeout and keeps the connection alive.

Writing Code to Use the Progress Bar

  • Tell G2 to use the ProgressBar view in one of these ways:
    • An AdminMaintenance task has function requiresProgressBar() which can return true.
    • An ItemEditPlugin has function handleRequest(), the 4th return value from this is a boolean specifying whether to use ProgressBar.
    • From your own controller, delegate to core.ProgressBar. See the section below for details.
  • Tell G2 what action to perform once the ProgressBar is displayed:
    • An AdminMaintenance task will invoke its run() function.
    • All other cases use code like this:
  $templateAdapter =& $gallery->getTemplateAdapter();
  $templateAdapter->registerTrailerCallback(array($this, 'runMyTask'), array($form));

The parameters here are a callback followed by an array of parameters for the callback. In the above case you'd need function runMyTask($form) in this class.

  • Update the ProgressBar while the task runs:
  $templateAdapter =& $gallery->getTemplateAdapter();
  ...
  $templateAdapter->updateProgressBar('top heading', 'sub heading', $numProcessed / $total);
  ...
  $urlGenerator =& $gallery->getUrlGenerator();
  $redirect = array('view' => 'my.View');
  $templateAdapter->completeProgressBar($urlGenerator->generateUrl($redirect));

While your task runs, periodically call updateProgressBar(). Also use $gallery->guaranteeTimeLimit() to avoid PHP time limits. When the task is complete, call completeProgressBar() which shows a "Continue" link with the given URL.

How to Use a Progress Bar in a Controller

In short, you will have to delegate the request to the progress bar view and the progress bar view will then call a callback function that you registered before. In your callback function, you should periodically update the status, telling the progress bar to how much percent you're done with the task.

With these steps you can convert a normal controller to a controller that uses the progress bar view:

  1. Move the actual body of your controller into a new function (the callback function). All handleRequest() should do is calling that function and returning the results.
  2. Now change it such that instead of calling your new function, you register that function as a trailer callback with the template adapter.
  3. Finally, instead of redirecting at the end of handleRequest(), you should delegate to the ProgressBar view template.
  4. Your callback function should periodically call $templateAdapter->updateProgressBar(...) to update the status.
  5. Finally, your callback function should call $templateAdapter->completeProgressBar($continueUrl) to complete the task.

An example:

Before using a progress bar, the whole work was done in a handleRequest() function of a controller.
 class ExampleBatchRenameController extends GalleryController {
     function handleRequest($form) {
         // some batch task, operation on many items
         // set $result redirect to some success page
         return array(null, $results);
     }
 }
After converting the controller to use the progress bar, it looks like this:
  class ExampleBatchRenameController extends GalleryController {
      function handleRequest($form) {
          global $gallery;
          $templateAdapter =& $gallery->getTemplateAdapter();

          /*
           * Probably you need to do some input checking first. Depending on that, either show an error page
           * or delegate to the progress bar to actually do the work.
           */
          if (isset($form['action']['save'])) {
              /* Let's say we need an albumId for our operation. We should check for it before we delegate to the progress bar view! */
              $albumId = (int) GalleryUtilities::getRequestVariables('albumId');
              $renamePrefix = (string) GalleryUtilities::getRequestVariables('itemNamePrefix');
              if (empty($albumId) || empty($renamePrefix)) {
                  /* Let the view render a nice error message for the missing album id. */
                  $results['error'] = empty($albumId) ? array('form[error][missingAlbumId]') : array('form[error][emptyItemPrefix]');
                  $results['redirect']['view'] = 'core.ExampleBatchRenameView';
              } else {
                  /* Register our batch operation as a trailer callback function for the progress bar view. */
                  /* NOTE: The second array are the parameters that we pass to our callback function. */
                  $templateAdapter->registerTrailerCallback(array($this, 'runRenameOperation'), array($form, albumId, $renamePrefix ));

                  /* Delegate to the progress bar view which will then call our callback function. */
                  $results['delegate']['view'] = 'core.ProgressBar';
              }
          } else {
              $results['redirect']['view'] = 'core.SomeOtherPage';
          }

          return array(null, $results);
      }

      /**
       * Callback function for the progress bar view that does the actual work.
       * In this example, we rename all items under a specific album in some way.
       *
       * @param $form (see GalleryController::handleRequest)
       * @param int $albumId just an example to show how to pass parameters to this callback function
       * @param string $itemTitlePrefix we add this string as a prefix to all item titles
       * @return object a GalleryStatus object or null (all trailer callbacks should return a GalleryStatus only)
       */
      function runRenameOperation($form, $albumId, $itemTitlePrefix) {
          global $gallery;
          $templateAdapter =& $gallery->getTemplateAdapter();
          $storage =& $gallery->getStorage();

          $heading = $module->translate('Adding a prefix to all item titles');
          $gallery->guaranteeTimeLimit(60);
          /* The "0" indicates 0 percent done, it will render the progress bar with 0% done. */
          $templateAdapter->updateProgressBar($heading, $module->translate('Preparing...'), 0);

          /* This call could potentially return a million item ids, it's a whole subtree of our Gallery. */
          list ($ret, $itemIds) = GalleryCoreApi::fetchDescendentItemIds($item, null, null, 'core.edit');
          if ($ret) {
	    return $ret;
	  }

          /* Now let's process all the items! */
          /*
           * To handle the operation efficiently, we're processing the items in batches. The batch size should be small enough
           * such that it can be handled in a short time though. The batch should also be reasonably small (around 100) since
           * the unix "open files" limit (we open a file for each locked item if flock based locking is used).
           */
          $batchSize = 100;
          $total = count($itemIds);
          $ind = 0;
          /* We'd like that the progress bar updates at least in 5% steps (100 / 20 = 5). */
          $step = min(200, intval($total / 20) + 1);
          while (!empty($itemIds) {
              $currentItemIds = array_splice($itemIds, 0, $batchSize);
              /* Since we want to change the item title, we need to acquire a write lock for the items. */
              list ($ret, $lockId) = GalleryCoreApi::acquireWriteLock($currentItemIds);
              if ($ret) {
                  return $ret;
              }

              list ($ret, $$items) = GalleryCoreApi::loadEntitiesById($currentItemIds);
              if ($ret) {
                  return $ret;
              }

              foreach ($items as $$item) {
                  /* Now update the progress bar every once in a while. */
                  if (!(++$ind % $step) || $ind == $total) {
                      $message = $module->translate(array('text' => 'Processing item %d of %d',
                                                          'arg1' => $ind, 'arg2' => $total));
                      $templateAdapter->updateProgressBar($heading, $message, $ind / $total);

                      /* Also prevent PHP timeouts */
                      $gallery->guaranteeTimeLimit(60);
                  }

                  /* Do the actual work :) */
                  $item->setTitle($itemTitlePrefix . $item->getTitle);
                  $ret = $item->save();
                  if ($ret) {
                      GalleryCoreApi::releaseLocks($lockId);
                      return $ret;
                  }
              }

              $ret = GalleryCoreApi::releaseLocks($lockId);
              if ($ret) {
                  return $ret;
              }

              /* Periodic checkpoints ensure that the work that has been done until now is not lost due to a later error. */
              $ret = $storage->checkPoint();
              if ($ret) {
                  return $ret;
              }
          }

          /* Yay, we're done! Ensure the progress bar is at 100% and show a continue link. */
          $redirect = array('view' => 'core.ItemAdmin', 'subView' => 'core.ItemEdit', 'itemId' => $albumId);
          $urlGenerator =& $gallery->getUrlGenerator();
          $templateAdapter->completeProgressBar($urlGenerator->generateUrl($redirect));

          return null;
      }
  }

I'd like to highlight a few things:

  • Use $gallery->guaranteeTimeLimit()
  • Use $gallery->checkPoint()
  • Use batches smaller or around 100 to combat open files limit
  • Use regular progress bar updates to combat connection timeouts
  • Update in 5-10% steps
  • Acquire write locks when changing items
  • Don't forget to release your locks on error
  • Check for the correct permission in API calls (can the current user see a specific subitem, can the user edit it, ...)

How the Page Rendering Works

Example: Maintenance tasks in G2 (controller = modules/core/AdminMaintenance.class)

  1. User requests to execute a long-running maintenance task, e.g. rebuilding thumbnails.
  2. AdminMaintenanceController::handleRequest() is called
  3. We register the actual task as a "trailer". The task can be a function or a class method.
  4. We render the progress bar template (themes/matrix/templates/progressbar.tpl)
  5. progressbar.tpl is included in the theme.tpl
  6. the (almost) last thing theme.tpl renders is {g->trailer}, that means it starts the actual task (building the thumbnails)
  7. the task periodically sends a JavaScript snippet to the browser, the connection is never terminated, we're still in the same HTTP response as in step 1.

Comment from progressbar.tpl:

*
* After the template is sent to the browser, Gallery will send a
* Javascript snippet to the browser that calls the updateStatus()
* function below.  It'll call it over and over again until the task is
* done.  The code below should update the various elements of the page
* (title, description, etc) with the values that it receives as arguments.
*

And these JS snippets basically say to how many percent the task is completed and what the current description is. The JS then updates the page that is displayed. All things that we update have a HTML id. We use a table with a single row and 2 columns with 10% width for the first column and 90% percent for the 2nd column to show that the task is 10% complete. The next JS update sent from G2 to the browser changes the 1st column to e.g. 15% and the 2nd to 85%. etc.