Difference between revisions of "Gallery2:Modules:imagenotes" - Gallery Codex
Personal tools

Difference between revisions of "Gallery2:Modules:imagenotes"

From Gallery Codex

(Trying to bring up to date with where I'm at.)
(Imagenotes Module: Added more design notes and restructured page.)
Line 1: Line 1:
 
=Imagenotes Module=
 
=Imagenotes Module=
 
Select regions of your images and apply annotations, tags, etc.
 
Select regions of your images and apply annotations, tags, etc.
 
==Status==
 
Design and manufacture (Only client side region selection currently implemented, no annotation). More design work being done. See Dev status below for more info.
 
  
 
==Description==
 
==Description==
 
This module is intended to allow the selection of regions within your images for the purposes of annotation and tagging.  
 
This module is intended to allow the selection of regions within your images for the purposes of annotation and tagging.  
 +
 +
==Status==
 +
* Continued design. More work to do here. When finished, back to the implementation.
 +
* Slight restructuring of this wiki page and my thoughts.
 +
* Investigating extensibility.
 +
* Investigating non-rectangular (and complex 3d) image region selection (use SVG/VML with YUI abstraction?).
 +
* Investigating browser tab selection ordering and rendering layers. (YUI OverlayManager manages the z-order of Overlays, but this does not change the tab order as would be desired. Need to modify OverlayManager to re-order document)
 +
* Client side rectangular region selection and editing (move, resize) complete.
  
 
==Use cases==
 
==Use cases==
- A user wants to annotate an image by tagging parts of the image that represent the faces of his/her friends. When someone views the image the tags are drawn over the image and by hovering over the tag the image region that represents it is highlighted. It is already known which tags the user has used before, which tags have been used in the album and tags already attached to the image, so a list of suggestions for tags is available for choosing from.(The tags suggestion system can also hook into external systems to get relevant suggestions)
+
* A user wants to annotate an image by tagging parts of the image that represent the faces of his/her friends. When someone views the image the tags are drawn over the image and by hovering over the tag the image region that represents it is highlighted. It is already known which tags the user has used before, which tags have been used in the album and tags already attached to the image, so a list of suggestions for tags is available for choosing from.(The tags suggestion system can also hook into external systems to get relevant suggestions)
 +
* Someone who doesn't have permission to annotate images recognises someone in a photograph and wants to suggest that an annotation is created. Just like creating an actual annotation the user creates a 'suggestion' for an image regon and tag annotation. A user with sufficient priviledges will be able to view all pending suggestions (possibly even notified via e-mail) and accept or reject the suggestion. In the meantime, the user that created the suggestion can modify their suggestion, until the point the suggestion is accepted (how does this work with anonymous users?).
 +
*: If the suggestion is rejected the user who created it is somehow notified, maybe when they go to view their suggestions or maybe by e-mail.
 +
* Someone who doesn't have priviledge notices that an image tag is incorrect. They suggest an ammendment, which can be accepted or rejected by a user with sufficient priviledge.
 +
* An image recognition program trawls through the image database and recognises some faces and objects. The program makes suggestions on what various regions of the images are to a user with sufficient priviledge to view them. The privilidged user can verify the image recognition programs analysis and choose to notify the program that it's recognition is correct or false, the user can also choose to add annotations based on any correct recognition and notify the program what the actual image area was when recognition was false.
 +
* A user wants to annotate an image with notes about a specific region of the image. Notes are more detailed than simple tags and give describtions as to what lies within a certain region of the image. When the image is viewed a user can see a region has been marked up and on giving the region focus (mouse or keyboard) the note relevant to that region will be displayed. The user who creates the annotation can place the annotation and decide if the text is always shown or just when focus is given to the image region.
 +
* A user wants to spice up an image by adding some speech bubbles containing some text. The user can choose to have the speech bubble always displayed or triggered to display on certain events (such as mouseover a certain region). The speech bubble display in itself can trigger other events that react with other annotation objects (so a sequence of speech can be displayed).
  
- Someone who doesn't have permission to annotate images recognises someone in a photograph and wants to suggest that an annotation is created. Just like creating an actual annotation the user creates a 'suggestion' for an image regon and tag annotation. A user with sufficient priviledges will be able to view all pending suggestions (possibly even notified via e-mail) and accept or reject the suggestion. In the meantime, the user that created the suggestion can modify their suggestion, until the point the suggestion is accepted (how does this work with anonymous users?).
+
==Investigations==
If the suggestion is rejected the user who created it is somehow notified, maybe when they go to view their suggestions or maybe by e-mail.
+
; Investigating browser tab selection ordering and rendering layers:
 +
: YUI OverlayManager manages the z-order of Overlays, but this does not change the browser tab order. The browser tab order rely's on an elements position in the DOM. Making the tab order the same as the layer order would give a better feel.  
 +
: Need to modify OverlayManager to re-order document rather than manipulate z-index. As well as benefitting tab ordering, SVG (Scalable Vector Graphics) does not handle z-ordering so requires document ordering to be rendered at the correct layer.
  
- Someone who doesn't have priviledge notices that an image tag is incorrect. They suggest an ammendment, which can be accepted or rejected by a user with sufficient priviledge.
+
==Design==
 +
===Overview===
 +
* Keep extensible.
 +
*: It is hoped that this module will be extensible so that features not envisioned here can be included later and so that administrators can enable only the features they desire, minimising page load time, client side operation time and resource usage.
  
- An image recognition program trawls through the image database and recognises some faces and objects. The program makes suggestions on what various regions of the images are to a user with sufficient priviledge to view them. The privilidged user can verify the image recognition programs analysis and choose to notify the program that it's recognition is correct or false, the user can also choose to add annotations based on any correct recognition and notify the program what the actual image area was when recognition was false.
+
===Components===
 +
; Image regions:
 +
* An image region is a specific area of an image that has some importance. This image region exists on all versions of the same image (i.e. on all derivatives of the source image).
 +
* An image region is represented by a rectangle or other 2d shape within the bounds of the image.
 +
* ?? An image region usually represents a complex 3 dimensional object in 2 dimensional space. Rather than just representing a 2d area, the image region may define the 3d properties of the object in the 2d space, such as the rotation of a face, head and torso. ?? Should this instead be an annotation/other association?
 +
; Image region sets:
 +
* An image region set is a grouping of related image regions. Because a 3d object may be represented by disjoint regions in 2d space, we need to be able to link separate image regions together. (This becomes important when, for instance, a person is being tagged and their arm goes behind someone else. We tag the image region set rather than each individual 2d image region)
 +
; Annotation:
 +
* An annotation is something that is drawn in the browser (may not be visible due to style, but added into the DOM) and is associated with the image.
 +
* The annotation, unlike an image region, is not necessarily desired on all sizes of the same image.
 +
* An annotation may be, but is not necessarily associated with an image region.
 +
: i.e. Tags and notes are associated with an image region, but a quirky speech bubble isn't.
 +
* An annotation may be shown, hidden, or animated by some event.
 +
* An annotation can be displayed in various positions:
 +
** Absolutely positioned, relative to the image.
 +
** Absolutely positioned, relative to another anotation.
 +
** Absolutely positioned, relative to the cursor.
 +
** In the imagenotes block area?
 +
** In a special annotation region for the annotation type.
 +
** In other special block areas of the correct type?
 +
; Annotation sets:
 +
* We may have lots of data about an image and not want to display all the data at the same time. Also, we may want different behaviours for interacting with the image when it is being viewed.
 +
* An annotation belongs in one or more annotation sets.
 +
* Annotations may have different properties per annotation set.
 +
* Only one annotation set is viewable at a given time, otherwise different properties of an annotation being displayed may conflict. It should be possible to merge sets into one another, giving the opportunity to resolve conflicts.
 +
; Image region associations:
 +
* There may be reasons for creating associations with an image region that aren't visual markup. Not sure what yet though. :-)
  
- A user wants to annotate an image with notes about a specific region of the image. Notes are more detailed than simple tags and give describtions as to what lies within a certain region of the image. When the image is viewed a user can see a region has been marked up and on giving the region focus (mouse or keyboard) the note relevant to that region will be displayed. The user who creates the annotation can place the annotation and decide if the text is always shown or just when focus is given to the image region.
+
===Client side===
 +
====Goals====
 +
* Make all actions responsive
 +
* Minimise resource usage
 +
* Minimise page load time
 +
* Only load required features. Load features as required?
 +
* Degrade gracefully?
  
- A user wants to spice up an image by adding some speech bubbles containing some text. The user can choose to have the speech bubble always displayed or triggered to display on certain events (such as mouseover a certain region). The speech bubble display in itself can trigger other events that react with other annotation objects (so a sequence of speech can be displayed).
+
====Notes====
 +
* This all really only works when javascript is enabled. It may be possible to render the markup server side without to atleast some degradeble functionality, but for now this is not being considered (would require the theme to put things in the right place).
 +
* Image notes manager object handles all interactions with any given image (of course, within the scope of the imagenotes module) and is responsible for any page regions associated with the imagenotes (image, block area, toolkit)
 +
* The image notes manager will be instanciated when the imagenotes block is included on photo pages.
 +
* The image notes manager has to be pointed to the image it is managing. Currently a utility function searches out an <img> tag matching the item being displayed (this may not always work), with the image src URL passed in through the imagenotes block smarty template.
 +
* The image notes manager is responsible for putting the image notes into the normal flow of browser focus, so when tabbing through the document editable regions and keyboard input works correctly.
 +
* The image notes manager is responsible for client/server communication, plug in modules will communicate through it (is this bad design?, maybe it's more efficient for plugins to do their own thing to save unnecessary stuff being loaded, but structure the calls through an interface method).
 +
* The image notes manager has a notion of 'mode'. The mode can be changed to allow editing of image regions, creation of annotations, and to generally change the view and behaviour of the imagenotes. An interface class will define the mode and other G2 modules will be able to add modes (collected through factory methods server side).
 +
* The image notes manager will have rubber band functionality (just within the image area?) that is available to modes. Initially intended only for dragging out an image region, it may be useful for selection.
 +
* Mode types, view, edit(/add/delete), suggest.
 +
* Themes can define placement of toolbox, dock areas, annotation areas.
 +
* A class will define an interface for a handler for imagenotes types. It handles a specific imagenote type. (e.g. for image tagging, the handler will look after the tag/image region relationship, even when an annotation doesn't exist)
 +
* A class will define an interface for imagenotes types. An imagenote type is something that may be rendered to the page? and will probably fit into the imagenotes manager mode system. Will have events for it's selection, mouseover, etc.
 +
* Can create annotation areas that fit into page flow and are not absolute(?) Can we determine this with javascript? Does it have to be done per item being displayed? Definitely theme specific. Will it be easily upset by adding other blocks? Sounds too complicated! :-)
  
 +
===Server side===
 +
====Goals====
 +
* Minimise page load time, whilst being extensible
  
==Features==
+
====Notes====
===Implemented===
+
* Image region at client end should stick to co-ordinates on local image. Server side should convert co-ordinates to full size image co-ordinates for storage and to derivative size co-ordinates for client display.
===Planned===
+
* Only include javascript in page when necessary. Restrict to needed JS based on user permissions (i.e. edit code is not needed for someone that cannot edit)
- Imagenotes manager component that handles all interactions (for imagenotes) with any given image
+
  
- Selection of multiple regions on an image.
+
===Database structure===
 +
====Goals====
 +
* Minimise page load time
 +
* Design so it's easy and quick to search for interesting things
 +
====Notes====
 +
* Image region needs a unique ID, probably built from a key on the image it's attached to and it's own unique ID for the particular image. Maybe image region ID should have a single unique key to simplify cross referencing from other tables. Image region needs layer information to ensure the desired z-ordering is applied  on rendering.
  
- AJAX communication with the server so page reloads are not required.
+
<!-- ===Hacks, or areas otherwise needing improvement=== -->
  
- Associate tags (from the [Tags] module) with one of more regions.
+
===Implementation===
  
- Associate some arbitrary text with an area of the image and display it either in a separate block or over the top of the image (potentially stylised as a speech or thought bubble).
+
====Client side====
 +
* Created utility methods for creating instances of templates. That is, cloning a DOM node and applying IDs to the new node and it's children.
 +
* Created TemplatedRubberBandRegion which allows a rubber band to be dragged over a specified region of the document and on mouseup a custom event is fired with the selected region
 +
* Created BoxModelRegionOverlay which extends the YUI Overlay widget and allows placement of an overlay based on the margin, border, padding or content parts of the box model. This is so that image region overlays can be positioned and sized by the content area.
 +
* Created TemplatedResizeOverlay which extends the BoxModelRegionOverlay widget and adds YUI DragDrop functionailty and the ability to resize the region. The code searches a template passed in client side for certain classes to turn into resize and drag handles. Custom events are fired at the start and end of drag and resize operations.
 +
* Created an initial image region manager that sets up a TemplatedRubberBandRegion and listens for area selected events, then creates a TemplatedResizeOverlay. Multiple regions can be created, but only a single region can be focused at a time.
 +
* The image region manager ensures that the image regions fit into the page focus flow.
 +
* Created utility function for finding an <img> tag in the page with a specific src attribute to allow the image region manager to be attached.
  
- Integration with external systems. Tags, images and objects exist in other systems. We may want to export information about our image regions or import/link to additional data in other systems.
+
====Server side====
 +
* Block created for addition to item view page
 +
* No database structure created yet
 +
* No server endpoint for AJAX communication created yet
 +
 
 +
 
 +
==Features==
 +
<!-- ===Implemented=== -->
 +
===Planned===
 +
* Imagenotes manager component that handles all interactions (for imagenotes) with any given image
 +
* Selection of multiple regions on an image.
 +
* AJAX communication with the server so page reloads are not required.
 +
* Associate tags (from the [Tags] module) with one of more regions.
 +
* Associate some arbitrary text with an area of the image and display it either in a separate block or over the top of the image (potentially stylised as a speech or thought bubble).
 +
* Integration with external systems. Tags, images and objects exist in other systems. We may want to export information about our image regions or import/link to additional data in other systems.
 
E.g. If we tag a person in an image, we may want to pull contact details from a CMS when we hover over or click on that person in the image.
 
E.g. If we tag a person in an image, we may want to pull contact details from a CMS when we hover over or click on that person in the image.
 
+
* Search for tagged image regions and potentially display just the selected portion of the image in the search results.
- Search for tagged image regions and potentially display just the selected portion of the image in the search results.
+
  
 
===Planned, but may never happen===
 
===Planned, but may never happen===
- Contextual search about the location or size of a region. E.g. Where a person is tagged, we want them to  take up a certain percentage of the image, or be a certain distance from another person in the image.
+
* Contextual search about the location or size of a region. E.g. Where a person is tagged, we want them to  take up a certain percentage of the image, or be a certain distance from another person in the image.
 
+
* Hook in open source image recognition software for automatically suggestion image regions to tag. E.g. It searches through all your pictures, finds similarities in faces, lets you tell it who they are and then magically all your photos are tagged. :-)
- Hook in open source image recognition software for automatically suggestion image regions to tag. E.g. It searches through all your pictures, finds similarities in faces, lets you tell it who they are and then magically all your photos are tagged. :-)
+
  
 +
<!--
 
==Known bugs==
 
==Known bugs==
  
 
==Frequently Asked Questions==
 
==Frequently Asked Questions==
 +
-->
  
==Dev stuff==
+
==Reference==
===Current status===
+
===Feature requests===
- Been thinking through the use cases to try and produce a design that can be extended in the future whilst getting all the core concepts in in the first place.
+
* [https://sourceforge.net/tracker/index.php?func=detail&aid=1053895&group_id=7130&atid=357130 Imagemap notes]
 +
* [http://sourceforge.net/tracker/index.php?func=detail&aid=1741079&group_id=7130&atid=357130 Fotonotes]
 +
* [http://sourceforge.net/tracker/index.php?func=detail&aid=972082&group_id=7130&atid=357130 Labelling areas of images]
  
====Client side====
+
===Forum/Codex topics===
- Created utility methods for creating instances of templates. That is, cloning a DOM node and applying IDs to the new node and it's children.
+
* [http://gallery.menalto.com/node/47274 Facebook like image tagging]
 
+
* [http://gallery.menalto.com/node/71919 Image Notes]
- Created TemplatedRubberBandRegion which allows a rubber band to be dragged over a specified region of the document and on mouseup a custom event is fired with the selected region
+
 
+
- Created BoxModelRegionOverlay which extends the YUI Overlay widget and allows placement of an overlay based on the margin, border, padding or content parts of the box model. This is so that image region overlays can be positioned and sized by the content area.
+
 
+
- Created TemplatedResizeOverlay which extends the BoxModelRegionOverlay widget and adds YUI DragDrop functionailty and the ability to resize the region. The code searches a template passed in client side for certain classes to turn into resize and drag handles. Custom events are fired at the start and end of drag and resize operations.
+
 
+
- Created an initial image region manager that sets up a TemplatedRubberBandRegion and listens for area selected events, then creates a TemplatedResizeOverlay. Multiple regions can be created, but only a single region can be focused at a time.
+
 
+
- The image region manager ensures that the image regions fit into the page focus flow.
+
 
+
- Created utility function for finding an <img> tag in the page with a specific src attribute to allow the image region manager to be attached.
+
 
+
====Server side====
+
- Block created for addition to item view page
+
- No database structure created yet
+
- No server endpoint for AJAX communication created yet
+
 
+
===Design (Draft)===
+
====Client side====
+
- This all really only works when javascript is enabled. It may be possible to render the markup server side without to atleast some degradeble functionality, but for now this is not being considered (would require the theme to put things in the right place).
+
 
+
- Image notes manager object handles all interactions with any given image (of course, within the scope of the imagenotes module) and is responsible for any page regions associated with the imagenotes (image, block area, toolkit)
+
- The image notes manager will be instanciated when the imagenotes block is included on photo pages.
+
- The image notes manager has to be pointed to the image it is managing. Currently a utility function searches out an <img> tag matching the item being displayed (this may not always work), with the image src URL passed in through the imagenotes block smarty template.
+
- The image notes manager is responsible for putting the image notes into the normal flow of browser focus, so when tabbing through the document editable regions and keyboard input works correctly.
+
- The image notes manager is responsible for client/server communication, plug in modules will communicate through it (is this bad design?, maybe it's more efficient for plugins to do their own thing to save unnecessary stuff being loaded, but structure the calls through an interface method).
+
 
+
- The image notes manager has a notion of 'mode'. The mode can be changed to allow editing of image regions, creation of annotations, and to generally change the view and behaviour of the imagenotes. An interface class will define the mode and other G2 modules will be able to add modes (collected through factory methods server side).
+
- The image notes manager will have rubber band functionality (just within the image area?) that is available to modes. Initially intended only for dragging out an image region, it may be useful for selection.
+
- Mode types, view, edit(/add/delete), suggest.
+
 
+
- Themes can define placement of toolbox, dock areas, annotation areas.
+
 
+
- A class will define an interface for a handler for imagenotes types. It handles a specific imagenote type. (e.g. for image tagging, the handler will look after the tag/image region relationship, even when an annotation doesn't exist)
+
- A class will define an interface for imagenotes types. An imagenote type is something that may be rendered to the page? and will probably fit into the imagenotes manager mode system. Will have events for it's selection, mouseover, etc.
+
 
+
 
+
 
+
- Can create annotation areas that fit into page flow and are not absolute(?) Can we determine this with javascript? Does it have to be done per item being displayed? Definitely theme specific. Will it be easily upset by adding other blocks? Sounds too complicated! :-)
+
 
+
====General====
+
Need to make sure it's possible for regions to be non rectangular in the future. Try and make the code pluggable, or at least extensible.
+
 
+
Do the image regions apply to all versions of the image, or just the one that was marked up? It depends on use, but possibly annotations that aren't linked to a specific region will not be wanted on larger versions of the same image.
+
 
+
As there are a few different but related purposes for imagenotes, it will probably make sense to have a few different database tables.
+
 
+
For tagging regions of an image, there will certainly be an 'image region' that identifies what it is that's being tagged.
+
 
+
For image annotations, there are two possible styles. One style where a region is marking something up and the annotation is attached to that (so potentially an event is fired when the region is selected, unhiding the associated image annotation), essentially so you can pass comment on something in the image without obscuring it.
+
 
+
The other style is where the annotations aren't specifically related to any particular image region, but laid over the top. E.g. placing speech bubbles at a specific location on the image that don't need a trigger region to display.
+
 
+
It is also possible that different image regions may want to be linked into groups, for a couple of possible reasons. A group of regions may define one object. As the images are generally photos, they represent 3d space. A person in an image may have their arm around someone else, giving two regions showing one person within the same image. If we tag that person, we only want to associate one tag with the multiple regions.
+
 
+
Another possible use of grouping regions is for creating multiple sets of annotations. The user will be allowed to select the set of annotations they want to display.
+
 
+
 
+
====Database structure====
+
- Image region needs a unique ID, probably built from a key on the image it's attached to and it's own unique ID for the particular image. Maybe image region ID should have a single unique key to simplify cross referencing from other tables. Image region needs layer information to ensure the desired z-ordering is applied  on rendering.
+
 
+
===Hacks, or areas otherwise needing improvement===
+
- TemplatedResizeOverlay doesn't take into account CSS border sizes or padding when determining position and constraints. To make sure the content area is always highlighted independent of template used, this may need to be reworked a little.
+
 
+
- The method for attaching to the image for markup is theme specific so will break easily until other methods are developed. The safest way to attach would be to create a specific view for image region markup and viewing, but it's nicer to see the markup on the regular item view page.
+
 
+
- The TemplatedResizeOverlay makes use of private functions, which apparently are created for every instance of the overlay. This means unnecessary resource consumption, probably without gaining anything from it. The resize functions can probably be pulled out into the util namespace.
+
 
+
==Related feature requests==
+
[https://sourceforge.net/tracker/index.php?func=detail&aid=1053895&group_id=7130&atid=357130 Imagemap notes]
+
 
+
[http://sourceforge.net/tracker/index.php?func=detail&aid=1741079&group_id=7130&atid=357130 Fotonotes]
+
 
+
[http://sourceforge.net/tracker/index.php?func=detail&aid=972082&group_id=7130&atid=357130 Labelling areas of images]
+
 
+
==Related forum/codex topics==
+
[http://gallery.menalto.com/node/47274 Facebook like image tagging]
+
 
+
[http://gallery.menalto.com/node/71919 Image Notes]
+
  
 
Because image tagging is one of the goals, general tag discussions are also of interest:
 
Because image tagging is one of the goals, general tag discussions are also of interest:
 
+
* [http://codex.gallery2.org/Gallery2:Modules:tags Gallery2:Modules:tags]
[http://codex.gallery2.org/Gallery2:Modules:tags Gallery2:Modules:tags]
+
* [http://gallery.menalto.com/node/69224 TagsSearch] For putting tags into categories
 
+
[http://gallery.menalto.com/node/69224 TagsSearch] For putting tags into categories
+
  
 
Because within an image, not only do you have a viewing location, you can see various places:
 
Because within an image, not only do you have a viewing location, you can see various places:
 
+
* [http://codex.gallery2.org/Gallery2:Modules:Map Gallery2:Modules:Map]
[http://codex.gallery2.org/Gallery2:Modules:Map Gallery2:Modules:Map]
+
 
+
  
  

Revision as of 13:01, 23 February 2008

Imagenotes Module

Select regions of your images and apply annotations, tags, etc.

Description

This module is intended to allow the selection of regions within your images for the purposes of annotation and tagging.

Status

  • Continued design. More work to do here. When finished, back to the implementation.
  • Slight restructuring of this wiki page and my thoughts.
  • Investigating extensibility.
  • Investigating non-rectangular (and complex 3d) image region selection (use SVG/VML with YUI abstraction?).
  • Investigating browser tab selection ordering and rendering layers. (YUI OverlayManager manages the z-order of Overlays, but this does not change the tab order as would be desired. Need to modify OverlayManager to re-order document)
  • Client side rectangular region selection and editing (move, resize) complete.

Use cases

  • A user wants to annotate an image by tagging parts of the image that represent the faces of his/her friends. When someone views the image the tags are drawn over the image and by hovering over the tag the image region that represents it is highlighted. It is already known which tags the user has used before, which tags have been used in the album and tags already attached to the image, so a list of suggestions for tags is available for choosing from.(The tags suggestion system can also hook into external systems to get relevant suggestions)
  • Someone who doesn't have permission to annotate images recognises someone in a photograph and wants to suggest that an annotation is created. Just like creating an actual annotation the user creates a 'suggestion' for an image regon and tag annotation. A user with sufficient priviledges will be able to view all pending suggestions (possibly even notified via e-mail) and accept or reject the suggestion. In the meantime, the user that created the suggestion can modify their suggestion, until the point the suggestion is accepted (how does this work with anonymous users?).
    If the suggestion is rejected the user who created it is somehow notified, maybe when they go to view their suggestions or maybe by e-mail.
  • Someone who doesn't have priviledge notices that an image tag is incorrect. They suggest an ammendment, which can be accepted or rejected by a user with sufficient priviledge.
  • An image recognition program trawls through the image database and recognises some faces and objects. The program makes suggestions on what various regions of the images are to a user with sufficient priviledge to view them. The privilidged user can verify the image recognition programs analysis and choose to notify the program that it's recognition is correct or false, the user can also choose to add annotations based on any correct recognition and notify the program what the actual image area was when recognition was false.
  • A user wants to annotate an image with notes about a specific region of the image. Notes are more detailed than simple tags and give describtions as to what lies within a certain region of the image. When the image is viewed a user can see a region has been marked up and on giving the region focus (mouse or keyboard) the note relevant to that region will be displayed. The user who creates the annotation can place the annotation and decide if the text is always shown or just when focus is given to the image region.
  • A user wants to spice up an image by adding some speech bubbles containing some text. The user can choose to have the speech bubble always displayed or triggered to display on certain events (such as mouseover a certain region). The speech bubble display in itself can trigger other events that react with other annotation objects (so a sequence of speech can be displayed).

Investigations

Investigating browser tab selection ordering and rendering layers
YUI OverlayManager manages the z-order of Overlays, but this does not change the browser tab order. The browser tab order rely's on an elements position in the DOM. Making the tab order the same as the layer order would give a better feel.
Need to modify OverlayManager to re-order document rather than manipulate z-index. As well as benefitting tab ordering, SVG (Scalable Vector Graphics) does not handle z-ordering so requires document ordering to be rendered at the correct layer.

Design

Overview

  • Keep extensible.
    It is hoped that this module will be extensible so that features not envisioned here can be included later and so that administrators can enable only the features they desire, minimising page load time, client side operation time and resource usage.

Components

Image regions
  • An image region is a specific area of an image that has some importance. This image region exists on all versions of the same image (i.e. on all derivatives of the source image).
  • An image region is represented by a rectangle or other 2d shape within the bounds of the image.
  •  ?? An image region usually represents a complex 3 dimensional object in 2 dimensional space. Rather than just representing a 2d area, the image region may define the 3d properties of the object in the 2d space, such as the rotation of a face, head and torso. ?? Should this instead be an annotation/other association?
Image region sets
  • An image region set is a grouping of related image regions. Because a 3d object may be represented by disjoint regions in 2d space, we need to be able to link separate image regions together. (This becomes important when, for instance, a person is being tagged and their arm goes behind someone else. We tag the image region set rather than each individual 2d image region)
Annotation
  • An annotation is something that is drawn in the browser (may not be visible due to style, but added into the DOM) and is associated with the image.
  • The annotation, unlike an image region, is not necessarily desired on all sizes of the same image.
  • An annotation may be, but is not necessarily associated with an image region.
i.e. Tags and notes are associated with an image region, but a quirky speech bubble isn't.
  • An annotation may be shown, hidden, or animated by some event.
  • An annotation can be displayed in various positions:
    • Absolutely positioned, relative to the image.
    • Absolutely positioned, relative to another anotation.
    • Absolutely positioned, relative to the cursor.
    • In the imagenotes block area?
    • In a special annotation region for the annotation type.
    • In other special block areas of the correct type?
Annotation sets
  • We may have lots of data about an image and not want to display all the data at the same time. Also, we may want different behaviours for interacting with the image when it is being viewed.
  • An annotation belongs in one or more annotation sets.
  • Annotations may have different properties per annotation set.
  • Only one annotation set is viewable at a given time, otherwise different properties of an annotation being displayed may conflict. It should be possible to merge sets into one another, giving the opportunity to resolve conflicts.
Image region associations
  • There may be reasons for creating associations with an image region that aren't visual markup. Not sure what yet though. :-)

Client side

Goals

  • Make all actions responsive
  • Minimise resource usage
  • Minimise page load time
  • Only load required features. Load features as required?
  • Degrade gracefully?

Notes

  • This all really only works when javascript is enabled. It may be possible to render the markup server side without to atleast some degradeble functionality, but for now this is not being considered (would require the theme to put things in the right place).
  • Image notes manager object handles all interactions with any given image (of course, within the scope of the imagenotes module) and is responsible for any page regions associated with the imagenotes (image, block area, toolkit)
  • The image notes manager will be instanciated when the imagenotes block is included on photo pages.
  • The image notes manager has to be pointed to the image it is managing. Currently a utility function searches out an <img> tag matching the item being displayed (this may not always work), with the image src URL passed in through the imagenotes block smarty template.
  • The image notes manager is responsible for putting the image notes into the normal flow of browser focus, so when tabbing through the document editable regions and keyboard input works correctly.
  • The image notes manager is responsible for client/server communication, plug in modules will communicate through it (is this bad design?, maybe it's more efficient for plugins to do their own thing to save unnecessary stuff being loaded, but structure the calls through an interface method).
  • The image notes manager has a notion of 'mode'. The mode can be changed to allow editing of image regions, creation of annotations, and to generally change the view and behaviour of the imagenotes. An interface class will define the mode and other G2 modules will be able to add modes (collected through factory methods server side).
  • The image notes manager will have rubber band functionality (just within the image area?) that is available to modes. Initially intended only for dragging out an image region, it may be useful for selection.
  • Mode types, view, edit(/add/delete), suggest.
  • Themes can define placement of toolbox, dock areas, annotation areas.
  • A class will define an interface for a handler for imagenotes types. It handles a specific imagenote type. (e.g. for image tagging, the handler will look after the tag/image region relationship, even when an annotation doesn't exist)
  • A class will define an interface for imagenotes types. An imagenote type is something that may be rendered to the page? and will probably fit into the imagenotes manager mode system. Will have events for it's selection, mouseover, etc.
  • Can create annotation areas that fit into page flow and are not absolute(?) Can we determine this with javascript? Does it have to be done per item being displayed? Definitely theme specific. Will it be easily upset by adding other blocks? Sounds too complicated! :-)

Server side

Goals

  • Minimise page load time, whilst being extensible

Notes

  • Image region at client end should stick to co-ordinates on local image. Server side should convert co-ordinates to full size image co-ordinates for storage and to derivative size co-ordinates for client display.
  • Only include javascript in page when necessary. Restrict to needed JS based on user permissions (i.e. edit code is not needed for someone that cannot edit)

Database structure

Goals

  • Minimise page load time
  • Design so it's easy and quick to search for interesting things

Notes

  • Image region needs a unique ID, probably built from a key on the image it's attached to and it's own unique ID for the particular image. Maybe image region ID should have a single unique key to simplify cross referencing from other tables. Image region needs layer information to ensure the desired z-ordering is applied on rendering.


Implementation

Client side

  • Created utility methods for creating instances of templates. That is, cloning a DOM node and applying IDs to the new node and it's children.
  • Created TemplatedRubberBandRegion which allows a rubber band to be dragged over a specified region of the document and on mouseup a custom event is fired with the selected region
  • Created BoxModelRegionOverlay which extends the YUI Overlay widget and allows placement of an overlay based on the margin, border, padding or content parts of the box model. This is so that image region overlays can be positioned and sized by the content area.
  • Created TemplatedResizeOverlay which extends the BoxModelRegionOverlay widget and adds YUI DragDrop functionailty and the ability to resize the region. The code searches a template passed in client side for certain classes to turn into resize and drag handles. Custom events are fired at the start and end of drag and resize operations.
  • Created an initial image region manager that sets up a TemplatedRubberBandRegion and listens for area selected events, then creates a TemplatedResizeOverlay. Multiple regions can be created, but only a single region can be focused at a time.
  • The image region manager ensures that the image regions fit into the page focus flow.
  • Created utility function for finding an <img> tag in the page with a specific src attribute to allow the image region manager to be attached.

Server side

  • Block created for addition to item view page
  • No database structure created yet
  • No server endpoint for AJAX communication created yet


Features

Planned

  • Imagenotes manager component that handles all interactions (for imagenotes) with any given image
  • Selection of multiple regions on an image.
  • AJAX communication with the server so page reloads are not required.
  • Associate tags (from the [Tags] module) with one of more regions.
  • Associate some arbitrary text with an area of the image and display it either in a separate block or over the top of the image (potentially stylised as a speech or thought bubble).
  • Integration with external systems. Tags, images and objects exist in other systems. We may want to export information about our image regions or import/link to additional data in other systems.

E.g. If we tag a person in an image, we may want to pull contact details from a CMS when we hover over or click on that person in the image.

  • Search for tagged image regions and potentially display just the selected portion of the image in the search results.

Planned, but may never happen

  • Contextual search about the location or size of a region. E.g. Where a person is tagged, we want them to take up a certain percentage of the image, or be a certain distance from another person in the image.
  • Hook in open source image recognition software for automatically suggestion image regions to tag. E.g. It searches through all your pictures, finds similarities in faces, lets you tell it who they are and then magically all your photos are tagged. :-)


Reference

Feature requests

Forum/Codex topics

Because image tagging is one of the goals, general tag discussions are also of interest:

Because within an image, not only do you have a viewing location, you can see various places:

advertisements