Appearance Model library and tools
The mapm library contains classes to implement statistical texture and appearance models.
The mapm_texture_model class contains:
- An object to perform sampling from image into a vector (si2v_sampler)
- The mean and modes of a linear statistical model describing variation over the training set
The mapm_app_model class contains:
- An msm_shape_model to represent the shape variation
- A mapm_texture_model to represent the texture variation
- Mean and modes of the combined shape+texture parameter vectors (encoding the link between shape and texture)
To train a texture model use:
mapm_build_texture_model -p params.txt
An example of a parameter file is:
builder: {
//: Sampler object - in this case a region defined by a set of triangles
// The n_planes indicates the number of planes in the input images (3 for RGB, 1 for greyscale)
// If tri_path is specified, triangles are loaded from named file. If not, Delauney triangulation
// applied to the reference shape.
sampler: si2v_tri_region_sampler {
tri_path: -
frame_width: 100
use_integral_image: false
n_planes: 3
border_width: 0
image_filter: -
}
tex_transform: ttfn_linear_tex_transform { }
// Maximum number of shape modes
max_modes: 5
// Proportion of shape variation to explain
var_prop: 0.95
// Object to apply limits to shape parameters
param_limiter: msm_ellipsoid_limiter { accept_prop: 0.98 }
}
//: Define renumbering required under reflection
// If defined, a reflected version of each shape is included in build
reflection_symmetry: { 1 0 3 2 7 6 5 4 13 12 11 10 9 8 14 16 15 17 18 19 21 20 23 22 26 27 24 25 }
//: When true, only use reflection. When false, use both reflection and original.
// Only relevant when reflection_symmetry defined.
only_reflect: false
//: Number of image pyramid levels to build
max_im_pyr_levels: 4
image_dir: /home/bim/images/faces/tfc_train1/images/
points_dir: /home/bim/images/faces/tfc_train1/points_28/
images:
{
b_image_0011.pts : b_image_0011.png
a_image_0001.pts : a_image_0001.png
a_image_0002.pts : a_image_0002.png
a_image_0005.pts : a_image_0005.png
a_image_0007.pts : a_image_0007.png
a_image_0008.pts : a_image_0008.png
a_image_0010.pts : a_image_0010.png
a_image_0020.pts : a_image_0020.png
a_image_0025.pts : a_image_0025.png
}
Triangulation
If no triangulation file is supplied (the tri_path: parameter), a Delauney triangulation of
the mean shape is calculated, which will include the convex hull of all the points. This
may not be desirable for some shapes. Alternative triangulations can be created
manually, in the following text file (.tri) format:
version: 1
n_triangles: 3
{
{ v1: 0 v2: 1 v3: 3 }
{ v1: 1 v2: 3 v3: 2 }
{ v1: 1 v2: 2 v3: 4 }
}
It is easier to use a tool such as qapm_tri_editor to create and edit the triangles. This enables easy creation, deletion and swapping of triangles.
The above uses a si2v_tri_region_sampler, which uses a triangular mesh to represent the
region of interest.
Example of resulting mean texture (frame_width: 100):
 |
 |
| n_planes: 1 | n_planes: 3 |
Effect of changing frame_width:
The border_width parameter defines how many pixels outside the triangulated
region will be sampled. Morphological operations are used to expand the triangulated
region in the reference frame by border_width pixels, and these are assigned to the
nearest triangle to work out where they should be sampled from. This helps when edge
information is important.
Effect of changing border_width:
Image Filters
The image_filter object applies a filter to the image in the reference frame.
This allows smoothing, local normalisation or gradient calculations. The filter can generate multiple planes from each input plane (for instance, two gradient planes for each input plane).
Image filters are derived from the mifl_image_filter_f2f base class.
Examples include:
- mifl_abs_grad_f2f { inc_raw_plane: true } // Input + Abs. gradients in x and y directions
- mifl_linear_norm_f2f { } // Linear function applied so mean is zero and variance=1
- mifl_local_z_norm_f2f { smooth_width: 5 min_sd: 1 } // Local z-normalisation
- mifl_sigmoid_grad_f2f { } // Grad in x, y after non-linear normalisation
- mifl_z_norm_abs_grad_f2f { smooth_width: 5 grad_smooth_width: 5 min_sd: 1 } // Z-norm, and x, y gradient
See mifl library documentation for others and full details.
Sampling Patches
An alternative sampler, si2v_patches_sampler uses boxes around each point:
// Sample 21x21 patch around each point, each normalised independently
sampler: si2v_patches_sampler {
frame_width: 100
default_grabber: si2v_box_patch_grabber { ni: 21 nj: 21 n_planes: 1 use_lin_norm: true }
}
Example of resulting mean texture (frame_width: 100):
 |
 |
| n_planes: 1 | n_planes: 3 |
Effect of changing ni,nj in si2v_box_patch_grabber (frame_width: 100)
Effect of changing ni,nj in si2v_box_patch_grabber (frame_width: 70)
To train an appearance model use:
mapm_build_app_model -p params.txt -o app_model.apm
An example of a parameter file is:
builder: {
shape_builder: {
aligner: msm_similarity_aligner
min_modes: 0 max_modes: 99 var_prop: 0.98
param_limiter: msm_ellipsoid_limiter { accept_prop: 0.98 }
}
tex_builder: {
//: Sampler object - in this case a region defined by a set of triangles
// The n_planes indicates the number of planes in the input images (3 for RGB, 1 for greyscale)
// If tri_path is specified, triangles are loaded from named file. If not, Delauney triangulation
// applied to the reference shape.
sampler: si2v_tri_region_sampler {
tri_path: -
// Filter to apply to image in reference frame
image_filter: -
// Approximate width of mean shape in reference frame
frame_width: 100
use_integral_image: false
n_planes: 1
}
// Texture transformation object
tex_transform: ttfn_offset_tex_transform { }
// tex_transform: ttfn_identity_tex_transform { }
// tex_transform: ttfn_linear_tex_transform { }
// Maximum number of modes
max_modes: 99
// Proportion of shape variation to explain
var_prop: 0.95
// Object to apply limits to shape parameters
param_limiter: msm_ellipsoid_limiter { accept_prop: 0.98 }
}
min_modes: 0 max_modes: 77 var_prop: 0.99
param_limiter: msm_ellipsoid_limiter { accept_prop: 0.98 }
verbose: true
}
//: Define renumbering required under reflection
// If defined, a reflected version of each shape is included in build
reflection_symmetry: { 1 0 3 2 7 6 5 4 13 12 11 10 9 8 14 16 15 17 18 19 21 20 23 22 26 27 24 25 }
//: When true, only use reflection. When false, use both reflection and original.
// Only relevant when reflection_symmetry defined.
only_reflect: false
//: Number of image pyramid levels to build
max_im_pyr_levels: 4
image_dir: /home/bim/images/faces/tfc_train1/images/
points_dir: /home/bim/images/faces/tfc_train1/points_28/
images:
{
b_image_0011.pts : b_image_0011.png
a_image_0001.pts : a_image_0001.png
a_image_0002.pts : a_image_0002.png
a_image_0005.pts : a_image_0005.png
a_image_0007.pts : a_image_0007.png
// ...etc...
}
The effect of different samplers is shown here.
To view the modes of the model, either use
- qapm_mode_viewer -m appearance_model.apm (from uomapm/qapm) - See: qapm_mode_viewer
- mapm_make_mode_images - Generate single images showing modes
- mapm_make_mode_movie - Generate sequences of images for each mode (which can be animated)
Single frames with mapm_make_mode_images
To generate one image for each for the first five appearance modes, showing several choices of parameter, use:
mapm_make_mode_images -apm model.apm -t app -nm 5 -nf 3 -r 2.5 -o model_c -it png
This will generate images model_c1.png, model_c2.png ... model_c5.png.
Each image will contain 3 appearance model reconstructions (the -nf flag), spanning the range -2.5 S.D to +2.5 S.D. (the -r flag).
For instance:
To generate texture model frames, use "-t tex"
To generate shape model frames, use "-t shape"
Off-axis Modes
You can generate images along a different line in parameter space using
mapm_make_mode_images -apm model.apm -t app -c centre.txt -d dir_vec.txt -nf 3 -r 2.5 -o model_c.png
where dir_vec.txt is a text file containing the elements of a direction vector, for instance:
0.5 0 1.0
and centre.txt is a text file defining the centre point for the mode.
The above command would generate three examples along a line c+alpha*d, where c is the centre, d is a unit vector along the supplied direction. The values of alpha are defined by the -r flag (in units of S.D. along each axis).
Single image at a given parameter value
To generate a single image of the model at a particular position in parameter space (in the centre.txt file), use:
mapm_make_mode_images -apm model.apm -t app -c centre.txt -nf 1 -o new_image.png
Movie frames with mapm_make_mode_movie
This tool generates frames which can be collected to form a movie of a mode.
mapm_make_mode_movie -apm model.apm -t [app/shape/tex] -mlo 1 -mhi 5 -o out_base -it png [-d dir_path.txt]
Given a set of labelled images, we can calculate shape, texture and appearance model
parameters with the mapm_get_apm_params tool.
mapm_get_apm_params -p get_apm_params.params
where the parameter file is something like this:
apm_path: model.mapm
//: Base for results files
// Output will be basepath-shape.txt, basepath-tex.txt, basepath-app.txt for
// shape, texture and combined appearance modes
// Results saved one vector per line
output_basepath: basepath
//: When true, include points file name at start of each line
include_ptsname: true
//: When defined, defines class value for all examples
// When defined save binary files containing parameter vectors and class
// value, in a format suitable for the imsci_mlrn tools.
class_value: +1
//: Define renumbering required under reflection
// If defined, a reflected version of each shape is included in build
reflection_symmetry: { 1 0 3 2 7 6 5 4 13 12 11 10 9 8 14 16 15 17 18 19 21 20 23 22 26 27 24 25 }
//: When true, only use reflection. When false, use both reflection and original.
// Only relevant when reflection_symmetry defined.
only_reflect: false
//: Number of image pyramid levels to build
max_im_pyr_levels: 4
image_dir: /home/bim/images/faces/tfc_train1/images/
points_dir: /home/bim/images/faces/tfc_train1/points_28/
images:
{
a_image_0001.pts : a_image_0001.png
a_image_0002.pts : a_image_0002.png
a_image_0005.pts : a_image_0005.png
a_image_0007.pts : a_image_0007.png
a_image_0008.pts : a_image_0008.png
// ...
}
The tool will then save out three text files, basepath-shape.txt, basepath-tex.txt and basepath.app.txt, containing the relevant parameter vectors, one per line.
If the class_value value is defined, then the tool also saves out four binary files:
- basepath-shape.bfs : Containing {shape_vector,class_value} pairs
- basepath-tex.bfs : Containing {tex_vector,class_value} pairs
- basepath-app.bfs : Containing {app_vector,class_value} pairs
- basepath-s+t.bfs : Containing {shape_vector+tex_vector,class_value} pairs
The last contains vectors which are the contatenation of shape and texture parameter vectors for each example.
Vectors are saved as vnl_vector<float> objects, class_value as float. The format is that used by the other tools, so can be used to create training data for them.
The mapm_tps_mean_image tool warps a set of images to the mean shape using a thin plate spline.
Useful for quickly checking whether correspondences are sensible.
