A Higher-Order Structure

Tensor
Thomas Schultz, Joachim Weickert,
and Hans-Peter Seidel
MPII20074005 July 2007
Authors Addresses
Thomas Schultz
Max-Planck-Institut f ur Informatik
Stuhlsatzenhausweg 85
66123 Saarbr ucken
Germany
[email protected]
Joachim Weickert
Mathematical Image Analysis Group
Faculty of Mathematics and Computer Science
Saarland University, Building E2 4
66041 Saarbr ucken
Germany
[email protected]
Hans-Peter Seidel
Max-Planck-Institut f ur Informatik
Stuhlsatzenhausweg 85
66123 Saarbr ucken
Germany
[email protected]
Acknowledgements
We would like to thank Holger Theisel, who is with Bielefeld University, for
discussions at all stages of this project. Discussions with Torsten Langer, who
is with the MPI Informatik, helped in developing parts of the mathematical
toolbox in Chapter 4.
Our implementation uses the CImg library by David Tschumperle, avail-
able from http://cimg.sf.net/.
This research has partially been funded by the Max Planck Center for
Visual Computing and Communication (MPC-VCC).
Abstract
Structure tensors are a common tool for orientation estimation in image pro-
cessing and computer vision. We present a generalization of the traditional
second-order model to a higher-order structure tensor (HOST), which is able
to model more than one signicant orientation, as found in corners, junctions,
and multi-channel images. We provide a theoretical analysis and a number
of mathematical tools that facilitate practical use of the HOST, visualize it
using a novel glyph for higher-order tensors, and demonstrate how it can be
applied in an improved integrated edge, corner, and junction detector.
Keywords
Structure Tensor, Higher-Order Tensors, Corner Detection, Multivalued Im-
ages
Contents
1 Introduction 2
2 A Higher-Order Structure Tensor 4
3 Glyphs for Higher-Order Tensors 6
3.1 Generalized Ellipses as Higher-Order Tensor Glyphs . . . . . . 6
3.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 7
4 A Mathematical Toolbox 10
4.1 Ecient Representation . . . . . . . . . . . . . . . . . . . . . 10
4.2 Relation to Truncated Fourier Series . . . . . . . . . . . . . . 10
4.3 Generalized Tensor Trace . . . . . . . . . . . . . . . . . . . . . 12
4.4 Generalized Eigenvector Decomposition . . . . . . . . . . . . . 13
4.5 Extrema of the Contrast Function . . . . . . . . . . . . . . . . 14
4.5.1 Accelerating the Brute Force Method . . . . . . . . . . 15
4.5.2 A Faster Method . . . . . . . . . . . . . . . . . . . . . 16
4.5.3 Implementation of the Faster Method . . . . . . . . . . 17
5 Integrated Edge and Junction Detection 18
6 Conclusions and Future Work 21
1
1 Introduction
The second-order structure tensor, formed as the outer product of the image
gradient with itself, is a common tool for local orientation estimation. Since it
was rst introduced for edge and corner detection [9], it has been applied to a
wide variety of problems in image processing and computer vision, including
optic ow estimation [2], image diusion [23], texture segmentation [21],
image inpainting [22], and image compression [10].
Two popular extensions are its generalization to vector- and tensor-valued
images, which goes back to an idea of Di Zenzo [7], and the introduction of
nonlinear local averaging [24], which lead to nonlinear structure tensors [4].
It is a known limitation of the traditional structure tensor that it can
only represent a single dominant orientation. Recently, there have been at-
tempts to overcome this: Arseneau and Cooperstock [1] have placed second-
order structure tensors in discrete directional bins and derived parameters
of multimodal directional distribution functions from them. Their work con-
centrates on lifting the constraint of antipodal symmetry (i.e., they treat
direction v dierently than direction v), a property which our approach
preserves. Moreover, they use the structure tensors only as an intermedi-
ate representation, nally reducing them to two scalar parameters for each
direction.
Herberthson et al. [12] have used outer products to handle pairs of orien-
tations. However, their approach is specic to the case of two orientations:
It neither generalizes to more than two directions, nor does it indicate cases
in which representing a single orientation is sucient.
In our present work, we present a generalization of the second-order struc-
ture tensor to a higher-order tensor model, which is able to capture the ori-
entations of more complex neighborhoods, for example corners, junctions,
and multivalued images. The tensor order allows to specify the maximum
complexity the structure tensor can represent and can be chosen based on
the requirements of a given application.
This report is structured as follows: Chapter 2 introduces our new higher-
2
order structure tensor (HOST). In Chapter 3, we present a novel glyph for
higher-order tensors and use it to visualize rst experimental results. A
theoretical analysis and a number of mathematical tools that help to use
the HOST in practice are presented in Chapter 4. They include an e-
cient representation of the structure tensor, an alternative representation as
a truncated Fourier Series, a generalization of the matrix trace and the eigen-
vector decomposition, and an algorithm to extract contrast extrema from a
higher-order tensor representation. Chapter 5 shows a proof-of-concept ap-
plication, in which the HOST is used for an improved integrated edge and
junction detection. Finally, Chapter 6 concludes this chapter and points out
directions of future research.
3
2 A Higher-Order Structure
Tensor
The standard second-order structure tensor J is given by the outer product
of the image gradient f with itself [9]:
J := ff
T
(2.1)
It is typically averaged over a neighborhood to obtain a descriptor of local im-
age structure. Our generalization to a higher-order tensor J simply repeats
the outer product. For a vector v, taking the outer product with itself l times
will be written v
l
. It yields an order-l tensor, indexed by {i
1
, i
2
, . . . , i
l
}:
_
v
l
_
i
1
i
2
...i
l
:= v
i
1
v
i
2
v
i
l
(2.2)
To ensure antipodal symmetry of the resulting tensor, we choose l to be even.
Interpretation of the higher-order structure tensor requires a contrast
function J, which species the local contrast in a given direction. When
the direction is represented by a unit-length vector u, J(u) is dened by
repeating the inner tensor-vector product of J and u until a scalar is left,
i.e., l times. In n dimensions, this can be written as
J(u) :=
n

i
1
=1
n

i
2
=1

n

i
l
=1
(J)
i
1
i
2
...i
l
u
i
1
u
i
2
u
i
l
(2.3)
This denition is inspired by a work of

Ozarslan and Mareci [18], who have
derived a diusivity function D(u) in the same manner from higher-order
diusion tensors D in the context of generalized diusion tensor magnetic
resonance imaging (DT-MRI). For a second-order structure tensor, J is uni-
modal, which reects the fact that it is suitable to model only one dominant
orientation. For higher orders, J can become multimodal, which allows a
more accurate representation of corners, junctions, and multi-images.
4
We consider it a sensible requirement that the values of the contrast
function should remain comparable, independent of the tensor order that
we use. When evaluated in direction of the gradient, the contrast function
yields the squared gradient magnitude in the second-order case. However,
taking the outer product l times would raise the gradient magnitude to the
lth power. We compensate this by scaling the gradient vector beforehand.
An order-l structure tensor J that reduces to the well-known second-order
tensor J for l = 2 is then given by
J :=
_
f
|f|
l2
l
_
l
(2.4)
In some applications, it is benecial to have a contrast function that gives
the non-squared gradient magnitude [3]. This can be achieved by replacing
the exponent
l2
l
by
l1
l
in Equation (2.4).
5
3 Glyphs for Higher-Order
Tensors
In the literature on higher-order diusion tensors, generalized Reynolds glyphs
constitute the only glyph-based visualization technique [18, 15]. Let S be the
unit sphere and J the contrast function as dened above. Then, these glyphs
are formed by the set of points {J(u)u| u S}, which directly depicts the
contrast prole of the tensor. However, these glyphs have a round shape
around their maxima, which makes their exact orientation dicult to see.
To compensate this problem, Hlawitschka and Scheuermann [15] suggest to
add arrows that point to the maxima.
3.1 Generalized Ellipses as Higher-Order Ten-
sor Glyphs
While the diusion ellipsoid is accepted as the standard glyph for second-
order diusion tensors, the Reynolds glyph does not reduce to it for l = 2.
Since the tensor ellipsoid can be constructed by transforming the unit sphere
under the linear mapping induced by the tensor, it is natural to generalize it
by taking the inner tensor-vector product l 1 times, until a vector is left.
We denote the inner product J u, where
(J u)
i
1
i
2
...i
l1
:=
n

i
l
=1
(J)
i
1
i
2
...i
l
u
i
l
(3.1)
and use the shortcut notation J
l1
u to indicate that we repeat it l1 times.
Then, the surface of our glyph is given by the points {J
l1
u| u S}.
As the 2D examples in Figure 3.1 illustrate, the extrema of the generalized
ellipses coincide with the extrema of the Reynolds glyphs. However, they
6
Figure 3.1: Three tensors of order six, visualized with Reynolds glyphs (a)
and our generalized ellipses (b). In (b), maxima of the contrast function
appear more localized.
develop sharp features around the maxima, at the cost of a smoother shape
around the minima. In examples two and three, the generalized ellipses
immediately make clear that the respective tensors are not axially symmetric,
a fact which the Reynolds glyph may not reveal at rst glance. Since we are
generally more interested in the maxima than in the minima of the contrast
function, we will use the new glyphs in the remainder of this chapter.
3.2 Experimental Results
We will now present some experiments to conrm that higher-order struc-
ture tensors indeed give a more accurate representation of junctions and
multivalued images. Our rst experiment uses simple junctions in synthetic
grayscale images. Derivatives are calculated by convolution with a derivative-
of-Gaussian lter ( = 0.7). After HOSTs of dierent order l have been
computed, their information is propagated to a local neighborhood by con-
volution with a Gaussian kernel ( = 1.4).
Figure 3.2 shows the test images, with the position of the displayed struc-
ture tensor marked by a cross. The results show that a HOST of order l = 4
is sucient to represent two edges that cross orthogonally, while the tradi-
tional structure tensor (l = 2) does not distinguish any particular direction.
In the non-orthogonal case, the traditional model indicates a principal direc-
tion which does not correspond to any gradient found in the image. While
the generalized ellipse of order four gives an impression of the involved direc-
tions, a clear separation of the maxima in the contrast prole now requires
higher orders. However, Chapter 4.4 will show that the generalized eigen-
vectors of the HOST give a good approximation of the gradient directions
already with l = 4.
The second experiment is based on a natural color image. Derivatives are
now calculated channel-wise and according to the conventional generalization
to multi-channel images, the HOSTs of the red, green, and blue color channels
7
(a) Orthogonal edges are clearly distinguished with order l = 4.
(b) For non-orthogonal edges, higher orders give more accurate repre-
sentations.
Figure 3.2: Two junctions in grayscale images and the corresponding struc-
ture tensors. For orders l > 2, the directions of the meeting edges can be
represented.
Figure 3.3: In a color image, the channel-wise gradients may point into
dierent directions. Higher-order structure tensors can be used to model
this situation accurately.
8
are added. In this case, we do not propagate the structure information
( = 0).
For comparison, Figure 3.3 also shows the gradients of the individual
color channels. Again, the structure tensor of order four already gives a much
better impression of the dominant directions than the traditional model. To
demonstrate the feasibility of going to very high tensor orders, we also present
the representation with l = 50.
9
4 A Mathematical Toolbox
4.1 Ecient Representation
An order-l tensor in n dimensions has n
l
tensor channels, which becomes
impractical already for moderate l. However, higher-order structure tensors
are totally symmetric, i.e., invariant under permutation of their indices. This
reduces the number of independent channels to N =
_
n+l1
l
_
, which means
merely linear growth for n = 2 (N = l + 1) and quadratic growth for n = 3.
With some additional notation, it is possible to evaluate J(u) directly
from this non-redundant representation: Call the ith non-redundant element
[J]
i
, stored in a zero-based linear array [J]. Let
i,k
{0, 1, . . . , l} denote
the number of times k {1, 2, . . . , n} appears as an index of the i-th element.
The multiplicity of element i, denoted
i
, is the number of times it appears
as a channel of the original tensor. For n = 2,
i
=
_
l

i,1
_
, for n = 3,

i
=
_
l

i,1
__
l
i,1

i,2
_
. Then, Equation (2.3) can be rewritten as
J(u) =
N1

i=0

i
[J]
i
u

i,1
1
u

i,2
2
u

i,n
n
(4.1)
For n = 2, we chose indices such that
i,1
= l i (e.g., [J
1111
, J
1112
, . . .]).
4.2 Relation to Truncated Fourier Series
From generalized DT-MRI, it is known that using a higher-order tensor model
in 3D is equivalent to approximating the diusivity prole with a truncated
Laplace series [18]. We will now show that the corresponding result in 2D is
a relation of higher-order tensors to truncated Fourier Series. This fact will
serve as the basis of the methods in Chapters 4.3 and 4.5.
10
Consider a Fourier Series, truncated after order l:
f() =
1
2
a
0
+
l

k=1
a
k
cos(k) +
l

k=1
b
k
sin(k) (4.2)
Setting a
k
:= b
k
:= 0 for odd k leaves a l + 1 dimensional vector space of
functions. For n = 2, Equation (4.1) can be rewritten in polar coordinates:
J() =
l

i=0
[J]
i
_
l
i
_
cos
li
sin
i
(4.3)
Let us regard [J]
i
as coecients and
_
l
i
_
cos
li
sin
i
as basis functions. We
will now show that these basis functions span the same space as the truncated
Fourier Series.
Proof by induction on order l. Let {f
k
} denote the basis functions of a
truncated Fourier Series in which only even multiples of are allowed:
f
k
:=
_

_
0.5 if k = 0
cos((k + 1)) if k odd
sin(k) if k even (k = 0)
Likewise, t
l
k
is the k-th basis function of an order-l tensor:
t
l
k
:=
_
l
k
_
cos
lk
sin
k

For l = 0, both the Fourier Series and the tensor basis represent constant
functions and f
0
= 0.5t
0
0
. Assume that the functions that can be represented
using {f
k
} with k l are equivalent to the functions represented by {t
l
k
}.
Further, assume that we know how to express the Fourier basis in terms of
the tensor basis. Then, we can show that the same assumption also holds for
l + 2: Observe that
cos
li
sin
i
=
_
cos
2
+ sin
2

_
cos
li
sin
i

= cos
l+2i
sin
i
+ cos
li
sin
i+2

and that the latter functions are proportional to functions in {t

l+2
k
}. Thus,
we can express the rst l + 1 Fourier basis functions in terms of {t
l+2
k
} by
replacing each occurence of t
l
k
in their known representation by
t
l
k
=
(l + 2 k)(l + 1 k)
(l + 2)(l + 1)
t
l+2
k
+
(k + 2)(k + 1)
(l + 2)(l + 1)
t
l+2
k+2
11
l = 2 a
0
= [J]
0
+ [J]
2
a
2
=
1
2
[J]
0

1
2
[J]
2
b
2
= [J]
1
l = 4 a
0
=
3
4
[J]
0
+
3
2
[J]
2
+
3
4
[J]
4
a
2
=
1
2
[J]
0

1
2
[J]
4
b
2
= [J]
1
+ [J]
3
a
4
=
1
8
[J]
0

3
4
[J]
2
+
1
8
[J]
4
b
4
=
1
2
[J]
1

1
2
[J]
3
Table 4.1: Relation of Fourier coecients and tensor components for orders
l = 2 and l = 4. A method to compute these relations for general l is given
in the text.
It remains to be shown how to express f
l+1
and f
l+2
in terms of {t
l+2
k
}.
For this, we use trigonometric identities for multiple angles:
f
l+1
= cos((l + 2))
=
l/2+1

i=0
(1)
i
_
l + 2
2i
_
cos
l+22i
sin
2i
=
l/2+1

i=0
(1)
i
t
l+2
2i
f
l+2
= sin((l + 2))
=
l/2

i=0
(1)
i
_
l + 2
2i + 1
_
cos
l+12i
sin
2i+1
=
l/2

i=0
(1)
i
t
l+2
2i+1
Our proof is constructive in the sense that it implies a recursive method
to construct a change-of-basis matrix. For reference, Table 4.1 presents the
relations for l = 2 and l = 4.
4.3 Generalized Tensor Trace
The second-order tensor trace has been used as a substitute of the squared
gradient magnitude [8]. For the higher-order case,

Ozarslan et al. [19] have
proposed a generalized trace operation gentr in 3D, which is based on
integrating J over the unit hemisphere and reduces to the standard matrix
trace for l = 2:
gentr(J) :=
3
2
_

J(u) du (4.4)
In the 2D case, is one half of the unit circle and the normalization factor
3
2
is to be replaced with
2

. Since the generalized trace of an order-l 2D tensor

12
equals its Fourier coecient a
l
0
, we can use the results from Chapter 4.2 to
verify that
gentr(J) = a
l
0
= 2
l/2

i=0
[J]
2i
(l 1)!!
(l 2i)!! (2i)!!
(4.5)
where l!! is the double factorial, i.e., the product of integers in steps of two.
In the denition of J, we scaled the gradient magnitude such that the
maximum value of J is invariant to the tensor order. However, maxima
become narrower with increasing l, so the generalized trace decreases. It
follows from Equation (4.5) that the generalized trace of an order-l structure
tensor equals
gentr(J) = 2
(l 1)!!
l!!
|f|
2
(4.6)
4.4 Generalized Eigenvector Decomposition
Many applications of the second-order structure tensor depend on its spectral
decomposition into eigenvectors and eigenvalues (e.g., [23, 16, 22, 8, 10]).
In this section, we introduce the Cand (Canonical Decomposition), which
can be regarded as a generalized eigendecomposition for higher-order tensors
and has rst been studied by Hitchcock [13, 14]. A review in the context of
higher-order statistics and some new results are given by Comon et al. [6, 5].
We will concentrate on the symmetric Cand (sCand), which decomposes
a symmetric order-l tensor J into a sum of r outer powers v
l
i
of unit vectors
v
i
, i {1, 2, . . . r}, scaled with
i
:
J =
r

i=1

i
v
l
i
(4.7)
For l = 2, Equation (4.7) reduces to the spectral decomposition, where
i
are
the eigenvalues and v
i
the eigenvectors. It can be shown that any symmetric
higher-order tensor J has a sCand [5]. In analogy to the matrix rank, the
symmetric rank R
S
of J is dened as the smallest number r for which a
sCand exists. In dimension n = 2, it holds that R
S
l [6].
The Cand is a current research topic and while some theoretical results
have been obtained, practical algorithms for ecient computation are rare.
Fortunately, Comon et al. [6] present an algorithm that works for n = 2 and
thus can be applied to our HOSTs. It is outside the scope of this chapter to
review the full theory required to derive the algorithm. For our experiments,
13
Figure 4.1: Generalized eigenvectors can be used to recover individual direc-
tions from a higher-order structure tensor.
we have simply re-implemented the Matlab code given in [6] in C, using
routines from Lapack
1
and the Numerical Recipes [20].
The algorithm returns pairs of
i
and v
i
, where the v
i
are not normalized.
While it appears trivial to convert this result to the canonical form, the
algorithm proves numerically unstable for vectors v
i
which are nearly aligned
with the y-axis: In such cases,
i
tends to zero, while the magnitude of v
i
tends to innity. We work around this problem by reconstructing a tensor J

only from those v

i
which have a reasonable magnitude. Then, the residual

J := J J

can be rotated by 90

to obtain the remaining v

i
. Note that the
tensor rotation only requires a simple permutation of its elements and some
sign changes: In the array representation from Section 4.1, it is sucient to
reverse the array and to multiply entries [J]
i
with an odd index i by 1.
Figure 4.1 visually represents Equation (4.7) for a particular HOST. Even
though the gradient directions in the neighborhood of the considered pixel
are too close to be resolved in the contrast prole of an order-four tensor,
they are well approximated by the two largest generalized eigenvectors.
Our prototype implementation found the sCand of 160 000 order six
structure tensors from a natural color image in around 2.5 s on a 2 GHz
Athlon 64.
4.5 Extrema of the Contrast Function
A frequent problem when dealing with higher-order structure tensors will be
to nd the angles at which the contrast function J attains an extremum.
In the second-order case, maxima and minima are given by the directions
of the major and minor eigenvectors, respectively. However, the generalized
eigenvectors do not in general coincide with maxima in the contrast function.
To nd the extrema of the contrast function, we use the Fourier Series
1
http://www.netlib.org/lapack/
14
representation from Section 4.2. This makes it easy to take derivatives of J,
since they are again Fourier Series of the same order, whose coecients are
straightforward to compute.
4.5.1 Accelerating the Brute Force Method
Extracting extrema requires to nd angles at which J

() = 0. The obvious
method to nd such points for an arbitrary dierentiable function J is to
sample its derivative with some resolution r to identify intervals in which
it changes sign and to rene the result to a desired accuracy a by a binary
search for the sign change on these intervals.
This method may miss pairs of extrema whose distance is less than the
sampling resolution r. Fortunately, such pairs are usually only minor local
variations in the contrast function, which are of no practical interest (cf.
Figure 4.2), so we found r = 2

sucient for structure tensors of order six.

For higher orders, denser sampling will be necessary, as peaks in the contrast
function become sharper.
The computational cost of this method is dominated by the cost of eval-
uating the derivative J

. Evaluating Equation (4.2) directly involves l sines

and l cosines. A recursive formulation exists which is known alternatively as
Clenshaws algorithm or as the Goertzel-Watt algorithm and requires only a
single sine and cosine. It is given by the recursion rule
u
l+1
= u
l+2
= 0 (4.8a)
u
r
= f
r
+ 2u
r+1
cos u
r+2
with r = l, l 1, . . . , 1 (4.8b)
where f
r
:= a
r
if the sum of cosines is to be computed, f
r
:= b
r
for the sum
of sines. From u
1
and u
2
, the nal result is determined as
l

k=1
a
k
cos(k) = u
1
cos u
2
and
l

k=1
b
k
sin(k) = u
1
sin (4.9)
Gentleman [11] has shown that this method magnies roundo errors
when evaluated near = k (k Z) and can produce unusable results for
large orders l. Newbery [17] suggests to adaptively perform a phase shift
of /2 to avoid such cases. However, an experimental comparison of direct
evaluation, the original and the modied version of Clenshaws algorithm
indicates that the error is tolerable for the moderate values of l that occur in
our context: For 160 000 structure tensors of order six from a natural color
image, all methods produced identical results (with r = 2

and a = (2
7
)

, at
single precision). Even with l = 50, all methods gave the same extrema, now
15
(a) cos + (
1
2
+ ) sin 2 (b) cos +
1
2
sin 2 (c) cos + (
1
2
) sin 2
Figure 4.2: In a Fourier Series, a small (here, = 10
10
) can make the
dierence between a pair of extrema, a saddle, and no stationary point. The
top row shows one full period, while the bottom row gives a close-up of the
aected extrema.
with a maximum angular deviation of (2
7
)

, in less than 1% of the cases.

Thus, we chose the unmodied Clenshaw algorithm, which gave a speedup
factor of 2.2 for l = 6.
4.5.2 A Faster Method
We will now present a more ecient algorithm which exploits the fact that
even higher derivatives of J are easy to evaluate. The basic idea of the
method is to expand J

into a Taylor series

J

, which is terminated after

degree three to obtain a polynomial that can be solved analytically in a
numerically stable manner [20]. Then, the error bounds of the expansion
dene a corridor around the x-axis. For intervals in which

J

() is outside
this corridor, we can be sure that J

() = 0, i.e., they do not contain an

extremum.
It is possible to recurse on the (now shorter) intervals in which

J

()
lies within the corridor; this allows to identify extrema which are so close
that nding them with the brute force approach would be computationally
infeasible. For example, nding the pair of extrema which is shown in Fig-
ure 4.2(a) would require sampling at r (2
10
)

. In contrast, our algorithm

needs only 10 Taylor expansions to identify all four extrema.
While it is nice to be able to nd such small extrema, they are not im-
16
portant in our context, so we would prefer to nd the major ones more
eciently. As intervals get down to a certain width, the computational eort
of reducing them further via the Taylor expansion exceeds the cost of the
brute force method. A simple way to combine both methods is to use the
Taylor expansion once to cut down search space, and to run the brute force
method on the remaining intervals. In the above experiment with order six
tensors (r = 2

, a = (2
7
)

), this gave a speedup of factor two, reducing the

total time for processing all 160 000 tensors on a 2 GHz Athlon 64 processor
to 2.2 seconds, which is about as fast as a sCand on the same data. For
smaller r, the speedup was much higher (factor 7.5 for r = 0.1

); for r = 5

,
it vanished.
4.5.3 Implementation of the Faster Method
Since we cannot expect a polynomial of degree three to reasonably approxi-
mate a sine of frequency l in an interval larger than /l, we initially partition
[0, 2) into 2l equal intervals for a Fourier Series of order l. For each of these
intervals, the Taylor expansion is performed by evaluating the higher deriva-
tives at its center
0
. The required Fourier coecients are pre-computed
once. For a third-order approximation, the error bound is
=
J
(5)
()
4!
(
0
)
4
(4.10)
for some within the interval. Taking the order-ve derivative in this ex-
pression is appropriate, since we approximate J

. To obtain a simple, safe

estimate of , the Fourier coecients a
k
and b
k
of J
(5)
() are used to state
that
J
(5)
()
l

k=1
_
a
2
k
+ b
2
k
(4.11)
and half the interval length is taken for (
0
).
Now, the roots of

J

and

J

+ give the intersections with the upper

and lower error bound, respectively. Starting from the value of

J

at the left
interval boundary, we can go through the sorted error bound intersections to
determine the intervals in which

J

is within the error corridor. Within some

intervals,

J

may lie fully inside or outside the corridor.

17
5 Integrated Edge and
Junction Detection
Given a grayscale image, edges can be seen as lines across which local contrast
is high. Then, corners are points in which edges bend sharply, and junctions
are points in which two or more edges meet. Corner detection has been one
of the rst applications of the structure tensor [9]. More recently, K othe [16]
has presented an algorithm that uses the structure tensor for integrated
edge and junction detection.
While this method is a considerable improvement over previous approa-
ches, it nally decomposes the structure tensor eld into an edge and a
junction part, which are processed by separate algorithms to produce the
respective edge and junction maps. Because of this, edges break down near
junctions, it is not always clear which edges are connected by a junction, and
isolated or duplicate junctions can occur (cf. Figure 5.1).
Unlike the second-order structure tensor, the HOST at a junction holds
enough information to nd the adjacent edges. This allows to extract edges,
corners, and junctions in a single, fully integrated process. The fundamental
idea is similar to tracking lines in higher-order tensor elds [15], except that
we assume that edges are orthogonal to contrast maxima.
Figure 5.2 illustrates the process of tracking edges in a HOST eld: Start-
ing from a seed with locally maximal generalized trace, we integrate edges
orthogonal to the major contrast indicated by the tensor (red arrow). When
a secondary peak in the contrast function attains a local maximum, we insert
a junction and start new edges from it (yellow arrows).
While standard hyperstreamline integration techniques can be applied
by interpolating the higher-order tensors channel-wise, we are typically not
interested in edge maps that have a greater resolution than the structure
tensor grid (note, however, that it is advisable to sample the structure tensors
at twice the image resolution [16]).
Thus, we implemented a simple tracking algorithm that works on the
18
(a) (b) (c) (d)
Figure 5.1: Subgure (b) shows edges and junctions extracted from (a) using
second-order structure tensors. (c) and (d) illustrate that edges break down
as the tensor becomes isotropic near junctions (marked with circles).
Figure 5.2: Scheme of the tracking process which extracts edges and junctions
simultaneously.
19
(a) (b) (c)
Figure 5.3: With higher-order tensors, edges can be traced through and
assigned to junctions, which allows easy classication of corners (green) and
junctions (red).
given grid. For each edge, it outputs a list of sub-pixels that belong to it,
and each junction holds references to the edges it connects. This information
allows an easy classication of corners and junctions, which are shown in
dierent colors in Figure 5.3, based on the number of incident edges.
Integration over a Gaussian neighborhood propagates the contrast from
edges beyond corner points, which leads to short phantom edges that make
corners appear as junctions. To avoid such artifacts, we reject edges along
which the HOST indicates multiple directions in all sub-pixels. Also, edges
may reach existing junctions through a cycle (like in Figure 5.3(c)), and we
need to connect them explicitly to such junctions.
Figure 5.3 presents a result of our method with HOSTs of order l = 6.
The present algorithm gives superior results on synthetic images and demon-
strates the applicability of the methods introduced in Chapter 4. However,
a version which is robust enough for natural images has to be left for future
research. In particular, we plan to explore the potential of using generalized
eigenvectors instead of contrast maxima to steer the tracking.
Our implementation of K othes algorithm took 0.3 s for the shown ex-
ample image (including his anisotropic averaging), while our method needed
0.9 s, again including computation and integration of the tensors. The orig-
inal image size was 300 300 pixels, giving a 599 599 sub-pixel structure
tensor eld.
20
6 Conclusions and Future
Work
In the present work, we have shown how higher-order tensors can be used to
represent the average of orientations in greater detail than it is possible using
traditional second-order structure tensors. We have introduced the notions,
denitions and mathematical tools required to work with such higher-order
structure tensors eciently and to visualize them appropriately.
While Chapter 5 demonstrates the advantages of the HOST for integrated
edge and junction detection, it is intended as a proof of concept for the
introduced methods, not as the ultimate goal of our research. Consequently,
our next step will be to explore the potential of the HOST in several image
processing and computer vision applications. We have already conducted
some promising experiments on using HOSTs for texture segmentation and
for steering image diusion.
All results in this chapter are in 2D. Some of the theory (Chapter 2) and
practical methods (Chapters 3 and 4.1) easily carry over to three dimensions,
or equivalents can be taken from the literature (Chapters 4.2 and 4.3). How-
ever, ecient methods for the sCand with n = 3 and for nding the maxima
of 3D contrast functions are still missing and probably require substantial
research.
21
Bibliography
[1] S. Arseneau and J. R. Cooperstock. An improved representation of
junctions through asymmetric tensor diusion. In G. Bebis, R. Boyle,
B. Parvin, D. Koracin, P. Remagnino, A. V. Nean, M. Gopi, V. Pas-
cucci, J. Zara, J. Molineros, H. Theisel, and T. Malzbender, editors,
Advances in Visual Computing, volume 4291 of Lecture Notes in Com-
puter Science, pages 363372. Springer, 2006.
[2] J. Big un, G. Granlund, and J. Wiklund. Multidimensional orientation
estimation with applications to texture analysis and optical ow. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 13(8):775
790, 1991.
[3] T. Brox and J. Weickert. A TV ow based local scale measure for
texture discrimination. In T. Pajdla and J. Matas, editors, Proc. 8th
European Conference on Computer Vision (ECCV04), volume 3022 of
LNCS, pages 578590. Springer, May 2004.
[4] T. Brox, J. Weickert, B. Burgeth, and P. Mr azek. Nonlinear structure
tensors. Image and Vision Computing, 24(1):4155, 2006.
[5] P. Comon, G. Golub, L.-H. Lim, and B. Mourrain. Symmetric tensors
and symmetric tensor rank. Technical Report SCCM-06-02, Stanford
Scientic Computing and Computational Mathematics (SCCM), 2006.
[6] P. Comon and B. Mourrain. Decomposition of quantics in sums of powers
of linear forms. Signal Processing, 53(2):96107, September 1996.
[7] S. Di Zenzo. A note on the gradient of a multi-image. Computer Vision,
Graphics, and Image Processing, 33:116125, 1986.
[8] C. Feddern, J. Weickert, and B. Burgeth. Level-set methods for tensor-
valued images. In O. D. Faugeras and N. Paragios, editors, Proc. Second
22
IEEE Workshop on Geometric and Level Set Methods in Computer Vi-
sion, pages 6572, Nice, France, October 2003.
[9] W. F orstner and E. G ulch. A fast operator for detection and precise
location of distinct points, corners and centres of circular features. In
International Society for Photogrammetry and Remote Sensing (ISPRS)
Intercomission Conference on Fast Processing of Photogrammetric Data,
pages 281305, Interlaken, 1987.
[10] I. Galic, J. Weickert, M. Welk, A. Bruhn, A. Belyaev, and H.-P. Seidel.
Towards PDE-based image compression. In N. Paragios, O. D. Faugeras,
T. Chan, and C. Schn orr, editors, Variational, Geometric, and Level Set
Methods in Computer Vision, Third International Workshop (VLSM
2005), volume 3752 of LNCS, pages 3748. Springer, 2005.
[11] W. Gentleman. An error analysis of Goertzels (Watts) method for
computing Fourier coecients. The Computer Journal, 12(2):160164,
1969.
[12] M. Herberthson, A. Brun, and H. Knutsson. Pairs of orientations in the
plane. In Proceedings of the SSBA Symposium on Image Analysis, pages
97100, Umea, Sweden, March 2006. SSBA.
[13] F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of
products. Journal of Mathematics and Physics, 6(1):164189, 1927.
[14] F. L. Hitchcock. Multiple invariants and generalized rank of a p-way
matrix or tensor. Journal of Mathematics and Physics, 7(1):3979, 1927.
[15] M. Hlawitschka and G. Scheuermann. HOT-lines: Tracking lines in
higher order tensor elds. In C. Silva, E. Gr oller, and H. Rushmeier, edi-
tors, Proceedings of IEEE Visualization 2005, pages 2734, Minneapolis,
MN, USA, October 2005.
[16] U. K othe. Edge and junction detection with an improved structure
tensor. In B. Michaelis and G. Krell, editors, Pattern Recognition. 25th
DAGM Symposium, volume 2781 of Lecture Notes in Computer Science,
pages 2532. Springer, 2003.
[17] A. Newbery. Error analysis for Fourier series evaluation. Mathematics
of Computation, 27(123):639644, 1973.
23
[18] E.

Ozarslan and T. Mareci. Generalized diusion tensor imaging and
analytical relationships between diusion tensor imaging and high an-
gular resolution diusion imaging. Magnetic Resonance in Medicine,
50:955965, 2003.
[19] E.

Ozarslan, B. C. Vemuri, and T. H. Mareci. Generalized scalar mea-
sures for diusion MRI using trace, variance, and entropy. Magnetic
Resonance in Medicine, 53:866876, 2005.
[20] W. H. Press, S. A. Teukolsky, and W. T. Vetterling. Numerical Recipes
in C++: The Art of Scientic Computing, Second Edition. Cambridge
Univ. Press, Cambridge, 2002.
[21] M. Rousson, T. Brox, and R. Deriche. Active unsupervised texture
segmentation on a diusion based feature space. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pages 699706,
Madison, Wisconsin, USA, June 2003.
[22] D. Tschumperle and R. Deriche. Vector-valued image regularization
with PDEs: A common framework for dierent applications. In Proc.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR
2003), pages 651656, 2003.
[23] J. Weickert. Anisotropic Diusion in Image Processing. Teubner,
Stuttgart, 1998.
[24] J. Weickert and T. Brox. Diusion and regularization of vector- and
matrix-valued images. In M. Nashed and O. Scherzer, editors, Inverse
Problems, Image Analysis, and Medical Imaging, volume 313 of Con-
temporary Mathematics, pages 251268. AMS, Providence, 2002.
24
Below you nd a list of the most recent technical reports of the Max-Planck-Institut f ur Informatik. They
are available by anonymous ftp from ftp.mpi-sb.mpg.de under the directory pub/papers/reports. Most
of the reports are also accessible via WWW using the URL http://www.mpi-sb.mpg.de. If you have any
questions concerning ftp or WWW access, please contact [email protected]. Paper copies (which
are not necessarily free of charge) can be ordered either by regular mail or by e-mail at the address below.
Max-Planck-Institut f ur Informatik
Library
attn. Anja Becker
Stuhlsatzenhausweg 85
66123 Saarbr ucken
GERMANY
e-mail: [email protected]
.
MPI-I-2007-RG1-002 T. Hilldenbrand, C. Weidenbach Superposition for Finite Domains
MPI-I-2007-5-002 S. Bedathur, K. Berberich,
T. Neumann, G. Weikum
A Time Machine for Text Search
MPI-I-2007-5-001 G. Ifrim, G. Kasneci, M. Ramanath,
F.M. Suchanek, G. Weikum
NAGA: Searching and Ranking Knowledge
MPI-I-2007-4-004 C. Stoll A Volumetric Approach to Interactive Shape Editing
MPI-I-2007-4-003 R. Bargmann, V. Blanz, H. Seidel A Nonlinear Viseme Model for Triphone-Based Speech
Synthesis
MPI-I-2007-4-002 T. Langer, H. Seidel Construction of Smooth Maps with Mean Value
Coordinates
MPI-I-2007-4-001 J. Gall, B. Rosenhahn, H. Seidel Clustered Stochastic Optimization for Object
Recognition and Pose Estimation
MPI-I-2007-2-001 A. Podelski, S. Wagner A Method and a Tool for Automatic Veriication of
Region Stability for Hybrid Systems
MPI-I-2007-1-001 E. Berberich, L. Kettner Linear-Time Reordering in a Sweep-line Algorithm for
Algebraic Curves Intersecting in a Common Point
MPI-I-2006-5-006 G. Kasnec, F.M. Suchanek,
G. Weikum
Yago - A Core of Semantic Knowledge
MPI-I-2006-5-005 R. Angelova, S. Siersdorfer A Neighborhood-Based Approach for Clustering of
Linked Document Collections
MPI-I-2006-5-004 F. Suchanek, G. Ifrim, G. Weikum Combining Linguistic and Statistical Analysis to
Extract Relations from Web Documents
MPI-I-2006-5-003 V. Scholz, M. Magnor Garment Texture Editing in Monocular Video
Sequences based on Color-Coded Printing Patterns
MPI-I-2006-5-002 H. Bast, D. Majumdar, R. Schenkel,
M. Theobald, G. Weikum
IO-Top-k: Index-access Optimized Top-k Query
Processing
MPI-I-2006-5-001 M. Bender, S. Michel, G. Weikum,
P. Triantalou
Overlap-Aware Global df Estimation in Distributed
Information Retrieval Systems
MPI-I-2006-4-010 A. Belyaev, T. Langer, H. Seidel Mean Value Coordinates for Arbitrary Spherical
Polygons and Polyhedra in R
3
MPI-I-2006-4-009 J. Gall, J. Pottho, B. Rosenhahn,
C. Schnoerr, H. Seidel
Interacting and Annealing Particle Filters:
Mathematics and a Recipe for Applications
MPI-I-2006-4-008 I. Albrecht, M. Kipp, M. Ne,
H. Seidel
Gesture Modeling and Animation by Imitation
MPI-I-2006-4-007 O. Schall, A. Belyaev, H. Seidel Feature-preserving Non-local Denoising of Static and
Time-varying Range Data
MPI-I-2006-4-006 C. Theobalt, N. Ahmed, H. Lensch,
M. Magnor, H. Seidel
Enhanced Dynamic Reectometry for Relightable
Free-Viewpoint Video
MPI-I-2006-4-005 A. Belyaev, H. Seidel, S. Yoshizawa Skeleton-driven Laplacian Mesh Deformations
MPI-I-2006-4-004 V. Havran, R. Herzog, H. Seidel On Fast Construction of Spatial Hierarchies for Ray
Tracing
MPI-I-2006-4-003 E. de Aguiar, R. Zayer, C. Theobalt,
M. Magnor, H. Seidel
A Framework for Natural Animation of Digitized
Models
MPI-I-2006-4-002 G. Ziegler, A. Tevs, C. Theobalt,
H. Seidel
GPU Point List Generation through Histogram
Pyramids
MPI-I-2006-4-001 A. Efremov, R. Mantiuk,
K. Myszkowski, H. Seidel
Design and Evaluation of Backward Compatible High
Dynamic Range Video Compression
MPI-I-2006-2-001 T. Wies, V. Kuncak, K. Zee,
A. Podelski, M. Rinard
On Verifying Complex Properties using Symbolic Shape
Analysis
MPI-I-2006-1-007 H. Bast, I. Weber, C.W. Mortensen Output-Sensitive Autocompletion Search
MPI-I-2006-1-006 M. Kerber Division-Free Computation of Subresultants Using
Bezout Matrices
MPI-I-2006-1-005 A. Eigenwillig, L. Kettner, N. Wolpert Snap Rounding of Bezier Curves
MPI-I-2006-1-004 S. Funke, S. Laue, R. Naujoks, L. Zvi Power Assignment Problems in Wireless
Communication
MPI-I-2005-5-002 S. Siersdorfer, G. Weikum Automated Retraining Methods for Document
Classication and their Parameter Tuning
MPI-I-2005-4-006 C. Fuchs, M. Goesele, T. Chen,
H. Seidel
An Emperical Model for Heterogeneous Translucent
Objects
MPI-I-2005-4-005 G. Krawczyk, M. Goesele, H. Seidel Photometric Calibration of High Dynamic Range
Cameras
MPI-I-2005-4-004 C. Theobalt, N. Ahmed, E. De Aguiar,
G. Ziegler, H. Lensch, M.A. Magnor,
H. Seidel
Joint Motion and Reectance Capture for Creating
Relightable 3D Videos
MPI-I-2005-4-003 T. Langer, A.G. Belyaev, H. Seidel Analysis and Design of Discrete Normals and
Curvatures
MPI-I-2005-4-002 O. Schall, A. Belyaev, H. Seidel Sparse Meshing of Uncertain and Noisy Surface
Scattered Data
MPI-I-2005-4-001 M. Fuchs, V. Blanz, H. Lensch,
H. Seidel
Reectance from Images: A Model-Based Approach for
Human Faces
MPI-I-2005-2-004 Y. Kazakov A Framework of Refutational Theorem Proving for
Saturation-Based Decision Procedures
MPI-I-2005-2-003 H.d. Nivelle Using Resolution as a Decision Procedure
MPI-I-2005-2-002 P. Maier, W. Charatonik, L. Georgieva Bounded Model Checking of Pointer Programs
MPI-I-2005-2-001 J. Homann, C. Gomes, B. Selman Bottleneck Behavior in CNF Formulas
MPI-I-2005-1-008 C. Gotsman, K. Kaligosi,
K. Mehlhorn, D. Michail, E. Pyrga
Cycle Bases of Graphs and Sampled Manifolds
MPI-I-2005-1-007 I. Katriel, M. Kutz A Faster Algorithm for Computing a Longest Common
Increasing Subsequence
MPI-I-2005-1-003 S. Baswana, K. Telikepalli Improved Algorithms for All-Pairs Approximate
Shortest Paths in Weighted Graphs
MPI-I-2005-1-002 I. Katriel, M. Kutz, M. Skutella Reachability Substitutes for Planar Digraphs
MPI-I-2005-1-001 D. Michail Rank-Maximal through Maximum Weight Matchings
MPI-I-2004-NWG3-001 M. Magnor Axisymmetric Reconstruction and 3D Visualization of
Bipolar Planetary Nebulae
MPI-I-2004-NWG1-001 B. Blanchet Automatic Proof of Strong Secrecy for Security
Protocols
MPI-I-2004-5-001 S. Siersdorfer, S. Sizov, G. Weikum Goal-oriented Methods and Meta Methods for
Document Classication and their Parameter Tuning
MPI-I-2004-4-006 K. Dmitriev, V. Havran, H. Seidel Faster Ray Tracing with SIMD Shaft Culling
MPI-I-2004-4-005 I.P. Ivrissimtzis, W.-. Jeong, S. Lee,
Y.a. Lee, H.-. Seidel
Neural Meshes: Surface Reconstruction with a Learning
Algorithm
MPI-I-2004-4-004 R. Zayer, C. R ossl, H. Seidel r-Adaptive Parameterization of Surfaces
MPI-I-2004-4-003 Y. Ohtake, A. Belyaev, H. Seidel 3D Scattered Data Interpolation and Approximation
with Multilevel Compactly Supported RBFs
MPI-I-2004-4-002 Y. Ohtake, A. Belyaev, H. Seidel Quadric-Based Mesh Reconstruction from Scattered
Data