opentl::modalities::Modality Class Reference

Inherits opentl::core::util::ParameterContainer.

Inherited by opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, opentl::modalities::Motion, and opentl::modalities::TemplateMap.

List of all members.

Public Types
Public Member Functions
virtual void	addChild (modalities::Modality *childFeature, core::cvdata::T_LEVEL childOutputLevel)
	Add one child at a time to this class adding more than one child is currently only possible for data fusion class instances child feature pointer and child output level for measurement.
virtual modalities::Modality *	clone () const =0
	Clone this class and all potential childs RECURSIVELY!! (deep copy).
virtual void	getCamIdx (std::set< int > *outCamIdx)
	returns all used camera indexes of descendant child incl. camIdx from this class instance
int	getCamIdx ()
	returns the camera index of this class instance
modalities::Modality *	getChildModalityFeature (std::size_t m)
modalities::Modality *	getChildModalityObject (std::size_t m)
modalities::Modality *	getChildModalityPixel (std::size_t m)
	access to the internal modalities
int	getIter ()
	Reset the iteration counter (useful for some optimization algorithms).
int	getNofChildren () const
	Return overall number of children modalities.
virtual void	init ()
	Needed init() method for parameter handling.
virtual void	invalidateFeatures ()
	Set the mForceFeatureUpdate flag in the feature data to TRUE.
virtual int	matchFeatLevel (const TargetPtrVector &targets, T_MEAS_FEATPtrVector &outputMeas, std::size_t partitionIdx)
	Matching on feature level match projected model features with detected image features.
virtual int	matchObjLevel (const TargetPtrVector &targets, T_MEAS_OBJPtrVector &outputMeas, std::size_t partitionIdx)
	Matching on object level compute a local, maximum-likelihood estimate of pose (evtl. state), using either pixel- or feature-level likelihood functions.
virtual int	matchPixLevel (const TargetPtrVector &targets, T_MEAS_PIXPtrVector &outputMeas, std::size_t partitionIdx)
	Matching on pixel level compare pixel-level image data with expected map, by projecting model data under a predicted state.
	Modality (const Modality &c)
	Abstract class - copy constructor.
	Modality (opentl::modelprojection::WarpBack *warpBack, int camIdx)
	Modality (opentl::modelprojection::Warp *warp, int camIdx)
	Constructor.
virtual int	preProcess (const opentl::core::cvdata::CvData &data)
	Pre-processing. Model-independent processing operations on given sensor data.
void	resetIter ()
	Reset the iteration counter (useful for some optimization algorithms).
virtual int	sampleModelFeatures (const TargetPtrVector &targets)
	Sample visible model features M_off, from the off-line given ShapeAppearance model (i.e. off-line).
virtual int	updateModelFeatures (const TargetPtrVector &targets)
	Update model features M_on, from the on-line image stream.
virtual	~Modality ()
Protected Member Functions
virtual int	callChildrenMatches (const TargetPtrVector &targets, std::size_t partitionIdx)
	calls match() function of all children (added with addChild()) in a recursive way
virtual void	cloneChildrenFrom (const modalities::Modality &feature)
	clones the children in a recursive way
virtual void	setParent (Modality *parent)
	sets the parent of this class instance
Protected Attributes
int	mCamIdx
	The sensor index related to this feature.
std::vector< std::pair < modalities::Modality *, T_MEAS_FEATPtrVector > >	mChildModalityFeature
std::vector< std::pair < modalities::Modality *, T_MEAS_OBJPtrVector > >	mChildModalityObject
std::vector< std::pair < modalities::Modality *, T_MEAS_PIXPtrVector > >	mChildModalityPixel
	Holds pointers to the child features and the related measurement structures per target, allocated by the calling class instance. ATTENTION: only used by the calling Feature class -> the calling class instance allocates the structures, and the called class (matchXXXLevel(...)) writes into these structures.
bool	mDeleteChildren
	Flag if children are owned by this instance => needed for deletion.
double	mDtimestamp
	The timestamp related to D (D = Unassociated, pose-independent output of pre-processing).
int	mIter
	This is the current iteration number, for optimization algorithms (output = object-level pose estimate).
unsigned int	mNofChildren
	Total number of children this feature is parent of.
Modality *	mParent
	pointer to _ONE_ parent node (no cycle within tree/pipeline allowed!)
std::set< int >	mRecusiveCamIndexes
	Holding all camera indexes for all descendant children (WITHOUT ourown camIdx!).
opentl::modelprojection::Warp *	mWarp
	Pointer to the global warp instance.
opentl::modelprojection::WarpBack *	mWarpBack
	Pointer to the global back-warp instance.

---

Member Enumeration Documentation

enum opentl::modalities::Modality::OfflineParameters

Common flags to all modalities: they specify which field will be filled in the output measurement, inside the matchXLevel() function. Default values = all true.

Enumerator:

MeasComputeFlags
OFFLINE_COUNT

Reimplemented in opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, opentl::modalities::Motion, and opentl::modalities::TemplateMap.

enum opentl::modalities::Modality::OnlineParameters

Enumerator:

ONLINE_COUNT

Reimplemented in opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, opentl::modalities::Motion, and opentl::modalities::TemplateMap.

enum opentl::modalities::Modality::TypeMeasComputeFlags

Enumerator:

COMPUTE_Z
COMPUTE_H
COMPUTE_JAC
COMPUTE_E
COMPUTE_R

---

Constructor & Destructor Documentation

opentl::modalities::Modality::Modality	(	opentl::modelprojection::Warp *	warp,
		int	camIdx
	)

Constructor.

opentl::modalities::Modality::Modality	(	opentl::modelprojection::WarpBack *	warpBack,
		int	camIdx
	)

virtual opentl::modalities::Modality::~Modality

(

)

[virtual]

opentl::modalities::Modality::Modality

(

const Modality &

c

)

Abstract class - copy constructor.

---

Member Function Documentation

virtual void opentl::modalities::Modality::addChild	(	modalities::Modality *	childFeature,
		core::cvdata::T_LEVEL	childOutputLevel
	)			[virtual]

Add one child at a time to this class adding more than one child is currently only possible for data fusion class instances child feature pointer and child output level for measurement.

virtual int opentl::modalities::Modality::callChildrenMatches	(	const TargetPtrVector &	targets,
		std::size_t	partitionIdx
	)			[protected, virtual]

calls match() function of all children (added with addChild()) in a recursive way

virtual modalities::Modality* opentl::modalities::Modality::clone

(

)

const [pure virtual]

Clone this class and all potential childs RECURSIVELY!! (deep copy).

Implemented in opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, opentl::modalities::Motion, and opentl::modalities::TemplateMap.

virtual void opentl::modalities::Modality::cloneChildrenFrom

(

const modalities::Modality &

feature

)

[protected, virtual]

clones the children in a recursive way

virtual void opentl::modalities::Modality::getCamIdx

(

std::set< int > *

outCamIdx

)

[virtual]

returns all used camera indexes of descendant child incl. camIdx from this class instance

int opentl::modalities::Modality::getCamIdx

(

)

[inline]

returns the camera index of this class instance

Reimplemented in opentl::modalities::ContourCCD.

modalities::Modality* opentl::modalities::Modality::getChildModalityFeature

(

std::size_t

m

)

modalities::Modality* opentl::modalities::Modality::getChildModalityObject

(

std::size_t

m

)

modalities::Modality* opentl::modalities::Modality::getChildModalityPixel

(

std::size_t

m

)

access to the internal modalities

int opentl::modalities::Modality::getIter

(

)

[inline]

Reset the iteration counter (useful for some optimization algorithms).

int opentl::modalities::Modality::getNofChildren

(

)

const [inline]

Return overall number of children modalities.

virtual void opentl::modalities::Modality::init

(

)

[inline, virtual]

Needed init() method for parameter handling.

Implements opentl::core::util::ParameterContainer.

Reimplemented in opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, opentl::modalities::Motion, and opentl::modalities::TemplateMap.

virtual void opentl::modalities::Modality::invalidateFeatures

(

)

[virtual]

Set the mForceFeatureUpdate flag in the feature data to TRUE.

Reimplemented in opentl::modalities::ContourCCD, and opentl::modalities::ContourPointsGPU.

virtual int opentl::modalities::Modality::matchFeatLevel	(	const TargetPtrVector &	targets,
		T_MEAS_FEATPtrVector &	outputMeas,
		std::size_t	partitionIdx
	)			[inline, virtual]

Matching on feature level

• match projected model features with detected image features.

Parameters:

	states	Predicted state
	outputMeas	Feature-space measurements and residuals

Reimplemented in opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, and opentl::modalities::TemplateMap.

virtual int opentl::modalities::Modality::matchObjLevel	(	const TargetPtrVector &	targets,
		T_MEAS_OBJPtrVector &	outputMeas,
		std::size_t	partitionIdx
	)			[inline, virtual]

Matching on object level

• compute a local, maximum-likelihood estimate of pose (evtl. state), using either pixel- or feature-level likelihood functions.

Parameters:

	states	Predicted state
	outputMeas	State-space measurement and residuals (expected state = prediction, observed state = ML estimate)

Reimplemented in opentl::modalities::ContourCCD, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, and opentl::modalities::TemplateMap.

virtual int opentl::modalities::Modality::matchPixLevel	(	const TargetPtrVector &	targets,
		T_MEAS_PIXPtrVector &	outputMeas,
		std::size_t	partitionIdx
	)			[inline, virtual]

Matching on pixel level

• compare pixel-level image data with expected map, by projecting model data under a predicted state.

Parameters:

	states	Predicted state
	outputMeas	Pixel-space measurement (can be a pixel map of residuals, or a unique residual value)

Reimplemented in opentl::modalities::BackgroundSub, opentl::modalities::ColourGMMGPU, opentl::modalities::DataFusion, and opentl::modalities::Motion.

virtual int opentl::modalities::Modality::preProcess

(

const opentl::core::cvdata::CvData &

data

)

[virtual]

Pre-processing. Model-independent processing operations on given sensor data.

• process the raw image (or for more process steps the preProcessed image, e.g. a binary image) to obtain an information suitable for matching
• examples: detect SIFT key-points, edge detection, color space conversion, optical flow computation, ...
• we need the full sensor data class because also the timestamp is required
• the resulting data is stored in a T_MEAS_PIX structure
• IMPORTANT: the output of the preProcess function must be stored as a static member (within a std::vector) in the derived class in order to be able to share the pre-processed data among multiple threads. The dim of the std::vector is the camera index.
• IMPORTANT: the preprocess function can communicate with other functions via static variables ONLY -> especially important if an initialize() is called within preProcess()

void opentl::modalities::Modality::resetIter

(

)

Reset the iteration counter (useful for some optimization algorithms).

virtual int opentl::modalities::Modality::sampleModelFeatures

(

const TargetPtrVector &

targets

)

[virtual]

Sample visible model features M_off, from the off-line given ShapeAppearance model (i.e. off-line).

• Compute visible parts (edges, surfaces, ...) of object model X, from camera Y, at PREDICTED pose P by rendering the complete scene (GLScene) from the given camera view

• Sample good features for tracking (contour points, keypoints, color histograms, ...)

Q: How do we decide if a single scene should consist of multiple rendered objectModels (= elements of "states" vector represent different objectModels) in order to deal with (partial) object occlusion or if just a single object should be rendered?

A: all elements of the std::vector<boost::shared_ptr<core::State> >* states should be rendered into a single scene e.g. if Kalman filters are used the vector of states is made by a "bank of Kalman filters" (one filter per target)

IMPORTANT: This operation is done at a predicted pose hypothesis, BEFORE matching

Reimplemented in opentl::modalities::ColourHist2D, opentl::modalities::ContourCCD, opentl::modalities::ContourPointsGPU, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, and opentl::modalities::TemplateMap.

virtual void opentl::modalities::Modality::setParent

(

Modality *

parent

)

[protected, virtual]

sets the parent of this class instance

virtual int opentl::modalities::Modality::updateModelFeatures

(

const TargetPtrVector &

targets

)

[virtual]

Update model features M_on, from the on-line image stream.

• Take the estimated pose P_est (AFTER tracker correction)
• Match at feature level (h,z) = color histograms, keypoints, etc.
• Use z for updating the reference model database: z(P_est) -> M_on

IMPORTANT: This operation is done at the estimated pose, AFTER tracking (correction)

Reimplemented in opentl::modalities::ColourHist2D, opentl::modalities::DataFusion, opentl::modalities::HarrisKeypoints, opentl::modalities::HistoOrientedGrad, and opentl::modalities::TemplateMap.

---

Member Data Documentation

int opentl::modalities::Modality::mCamIdx [protected]

The sensor index related to this feature.

std::vector<std::pair<modalities::Modality*, T_MEAS_FEATPtrVector > > opentl::modalities::Modality::mChildModalityFeature [protected]

std::vector<std::pair<modalities::Modality*, T_MEAS_OBJPtrVector > > opentl::modalities::Modality::mChildModalityObject [protected]

std::vector<std::pair<modalities::Modality*, T_MEAS_PIXPtrVector > > opentl::modalities::Modality::mChildModalityPixel [protected]

Holds pointers to the child features and the related measurement structures per target, allocated by the calling class instance. ATTENTION: only used by the calling Feature class -> the calling class instance allocates the structures, and the called class (matchXXXLevel(...)) writes into these structures.

Main index: modality-measurement data pair Data pair:

1. Child modality (=lower level) class -> works also for multithreaded childs 2. Double vector of measurement data (1. index = Target)

bool opentl::modalities::Modality::mDeleteChildren [protected]

Flag if children are owned by this instance => needed for deletion.

double opentl::modalities::Modality::mDtimestamp [protected]

The timestamp related to D (D = Unassociated, pose-independent output of pre-processing).

int opentl::modalities::Modality::mIter [protected]

This is the current iteration number, for optimization algorithms (output = object-level pose estimate).

unsigned int opentl::modalities::Modality::mNofChildren [protected]

Total number of children this feature is parent of.

Modality* opentl::modalities::Modality::mParent [protected]

pointer to _ONE_ parent node (no cycle within tree/pipeline allowed!)

std::set<int> opentl::modalities::Modality::mRecusiveCamIndexes [protected]

Holding all camera indexes for all descendant children (WITHOUT ourown camIdx!).

opentl::modelprojection::Warp* opentl::modalities::Modality::mWarp [protected]

Pointer to the global warp instance.

opentl::modelprojection::WarpBack* opentl::modalities::Modality::mWarpBack [protected]

Pointer to the global back-warp instance.