EE5121: Convex Optimization: Assignment 5

EE5121: Convex Optimization
Assignment 5
1. Show that the following functions are convex using properties that preserve convexity:
(a) f (x1 , x2 ) = (x1 − 4x2 )4 + 2e2x1 +3x2 −9
(b) KL Divergence: g(x, y) = y log y − y log x on R2++
2. Here are some of the commonly used objective functions in Neural Networks. Determine whether they are
convex or not? A Neural Network model takes in N input samples, where xi ∈ Rd is the feature vector of the
ith input sample. w ∈ Rd is the weight associated with each sample and ri = < w, xi >∈ R; yi is the output
for the ith input sample.
(a) Hinge Loss: H([r1 , r2 , · · · , rN ]) = ∑Ni=1 max(0, 1 − yi ri )

(b) Relu: R(ri ) = max(0, ri )
(c) Logistic function: L(ri ) = −log(1 + e−ri )
1
(d) Cross Entropy Loss: C([r1 , r2 , · · · , rN ]) = ∑Ni=1 yi − log + (1 − yi ) − log 1 − 1+e1−ri

1+e−ri
3. (a) Let h be a convex function in Rn , and let
f (x) = h(A(x − a)) + hx, a∗ i + α
, where A is a one-to-one linear transformation from Rn to Rn , a and a∗ are vectors in Rn , and α ∈ R.

Then show that
f ∗ (x∗ ) = h∗ (A∗−1 (x∗ − a∗ )) + hx∗ , ai + α ∗ ,
where A∗ is the adjoint of A and α ∗ = −α − h a, a∗ i.
Hint: Use suitable substitution variable y and properties of inner product to arrive at the result.
(b) Show that the function f (x) = ∑ri=1 |x|[i] on Rn , where |x| denotes the vector with |x|i = |xi | and |x|[i]
is the ith component of |x| is convex. In other words, |x|[1] , . . . , |x|[n] are the absolute values of the
components of x, sorted in the non-increasing order.
4. Show that for p > 1,
|x1 | p + . . . + |xn | p kxk pp
f (x,t) = =
t p−1 t p−1
is convex on {(x,t)|t > 0}
5. (a) Show that if the functions f and g are log-convex, f + g is also log-convex. Give a counter example to
show that this is not true for log-concave functions.
(Hint: log( f + g) = log(elog f + elog g ). Show that this is convex by the second-order test for convexity.)
(b) Suppose f is convex, g is non-decreasing and log-convex. Show that h(x) = g( f (x)) is log-convex.
(Hint: Use the definition of log-convex functions.)
(c) Show that the harmonic mean of two log-concave functions is log-concave.
(Note: Harmonic mean of a, b is defined as 1 2 1 .)
a+b
6. (a) Give a visual meaning of computing the Legendre-Frenchel conjugate in 1-D case when f is a convex
function from R to R
(b) Find the Legendre-Frenchel conjugate for the convex function f (x) = |x|
(c) Sketch roughly the Legendre-Frenchel conjugate of the general piecewise-linear convex function sketched
below in Figure 1.
f (x) = max{a1 x + b1 , a2 x + b2 , a3 x + b3 }
where
a1 < a2 < a3
1
y
a1 ≤ a2 ≤ a3
f3(x) = a3x + b3
g(x) = yx
f2(x) = a2x + b2 x
f1(x) = a1x + b1
Figure 1: Piecewise linear function
(HINTS: Use the visual meaning in (a). f (x) = |x| = max{x, −x}. So, use that intuition for extending
it to the general case here. Consider various regions for evaluating the Legendre-Frenchel conjugate
f ∗ (y) as y < a1 , a1 < y < a2 ,a2 < y < a3 ,a3 < y)
7. (a) In the last tutorial, we proved that a differentiable convex function defined on whole of Rn cannot be
bounded. What about quasiconvex functions?
(b) In the last tutorial, we saw that convex functions are continuous in the interior of their domains. What
about quasiconvex functions?
(c) For a differentiable convex function, using first order conditions, it was shown that when the derivative
or gradient vanishes, then that point is the global minimum. Is it true for quasiconvex functions?
Hint: Think of simple one dimensional counter examples by imagining how all can the graphs of
quasiconvex functions look like.
EXTRA QUESTIONS:
8. If fi : Rn →
− (−∞, ∞] be given functions for i ∈ I, where I is an arbitrary index set, and let f : Rn →
− (−∞, ∞]
be the function given by
f (x) = sup fi (x)

i∈I
If fi , i ∈ I, are convex, then f is also convex, while if fi , i ∈ I, are closed, then f is also closed.
(Definition: A functions f is define to be closed, if the epigraph epi( f ) is closed)
9. Consider the gamma function

Z ∞
Γ : R+ → R, Γ(x) = t x−1 e−t dt ∀x > 0.
0
Show that Γ is log-convex. You can use the Holder’s inequality: for any two functions f , g : R → R, if
p, q ≥ 1 and 1p + 1q = 1, then
Z +∞ Z +∞
1/p Z +∞
1/q
| f (t)g(t)|dt ≤ | f (t)| p dt |g(t)|q dt .
−∞ −∞ −∞
(Hint: Show that Γ (θ x + (1 − θ )y) ≤ Γ(x)θ Γ(y)(1−θ ) ∀θ ∈ (0, 1).)

(Food for thought: The gamma function is an extension of the factorial function (which is only defined for
non-negative integers) to all positive real numbers. It is the only log-convex function with this property.)

EE5121: Convex Optimization: Assignment 5

Uploaded by

Copyright:

Available Formats

EE5121: Convex Optimization: Assignment 5

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EE5121: Convex Optimization: Assignment 5

Uploaded by

Copyright:

Available Formats

EE5121: Convex Optimization

(a) Hinge Loss: H([r1 , r2 , · · · , rN ]) = ∑Ni=1 max(0, 1 − yi ri )

3. (a) Let h be a convex function in Rn , and let

f (x) = h(A(x − a)) + hx, a∗ i + α

, where A is a one-to-one linear transformation from Rn to Rn , a and a∗ are vectors in Rn , and α ∈ R.

Figure 1: Piecewise linear function

f (x) = sup fi (x)

9. Consider the gamma function

(Hint: Show that Γ (θ x + (1 − θ )y) ≤ Γ(x)θ Γ(y)(1−θ ) ∀θ ∈ (0, 1).)

You might also like