← All notes

Low Rank Tensor Recovery for Joint Probability Distribution

Contents
  1. Transformation rules of tensors
  2. Einstein Notation for Tensors
  3. Linear algebra of Tensors

Tensors

  • A scalar can be represented by a single component with zero basis vectors for each component.
  • A vector of 3 dimensions can be specified using three components and one basis vector for each component.
  • A stress matrix can be represented by nine components (we are considering 3 dimensions). That is two basis vectors for each component.

All of the above mathematical objects fall under a broad class of Tensors.

Tensors: In an mm-dimensional space, a tensor of order nn has nn indices and mnm^n components in a chosen basis, with components that obey the tensor transformation rules.

Generally, m=3m=3 in real-life scenarios. m=4m=4 is used while discussing relativity.

CP rank is the minimum number of rank-11 tensors whose sum equals the tensor.

Transformation rules of tensors

  • A tensor is an object that transforms like a tensor
  • A tensor is an object that is an object that is invariant under a change of coordinate system with components that change according to a special set of mathematical formulae

In simpler terms, the actual tensor is invariant to a change in the coordinate systems. For example, the displacement vector pointing from AA to BB ( a 11 rank tensor) does not change when coordinates change. The vector will still be pointing from AA to BB. However, the way we represent the vector, i.e., the components changes.

Matrices ≠\neq Tensors

Matrices are a form of representation of rank-22 tensors. They are just an array of numbers. Although, Tensors require detailed specification and are invariant under change of coordinate systems.

Einstein Notation for Tensors

Both subscripts and superscripts are used for indices in tensors.

  • Any twice repeated index in a single term is summed over. Index =1,2,⋯ ,n= 1, 2, \cdots , n. Typically, n=3n = 3

    For example, aijbj=∑j=1naijbj=ai1b1+⋯+ainbna_{ij}b_j = \sum_{j=1}^{n} a_{ij}b_j = a_{i1}b_1 + \cdots + a_{in}b_n

    Here, jj is a dummy variable and ii is a free variable.

    Dummy indices can be replaced, whereas free indices can’t be. A free index takes only one value in an expression.

  • No index may occur 33 or more times in a given term.

    aijbij✓a_{ij}b_{ij} \checkmark aiibij×a_{ii}b_{ij} \pmb\times aijbjj×a_{ij}b_{jj} \pmb\times

    We count subscripts and superscripts together. That is,

    ajj a_j^j - jj is a dummy variable

  • In an equation involving Einstein notation, the free indices on both sides must match.

Brackets in Einstein Notation

  1. Combine terms outside parentheses with each term inside parentheses separately.
  2. From each of the combined terms, use the largest count of each index as the final count in the overall term.

Kronecker Delta

δij≡δji≡δij≡{1i=j0i≠j\delta_{ij} \equiv \delta_j^i \equiv \delta^{ij} \equiv \begin{cases}1 & i=j\\ 0 & i \neq j \end{cases}

Linear algebra of Tensors

image-20210617172019075

image-20210617172120225

image-20210617172135319

For the following analysis, we shall be using the superscript notation to denote each element of a vector.

Rectangular coordinate system - A coordinate system (xix^i) in Rn\mathbb R^n is rectangular if the distance between 22 points C(x1,x2,⋯ )C(x^1, x^2, \cdots) and D(x1,x2,⋯ )D(x^1, x^2, \cdots) is given by : δijΔxiΔxj\sqrt{\delta_{ij} \Delta x^i \Delta x^j}

Curvilinear coordinates

  • PP is a point in a coordinate system (xix^i) in Rn\mathbb R^n given by P:(x1,x2,⋯ )P : (x^1,x^2, \cdots)
  • (xˉi)(\bar x^i) is another coordinate system in Rn\mathbb R^n such that the coordinates of PP in this system are (xˉ1,xˉ2,⋯ )(\bar x^1,\bar x^2, \cdots)
  • Suppose that xˉi=xˉi(x1,x2,⋯ ):F\bar x^i = \bar{x}^i(x^1, x^2, \cdots) : \mathcal{F}. If the functions xˉi(x1,x2,⋯ )\bar x^i(x^1, x^2, \cdots) are all real valued, have continuous 22nd partial derivatives everywhere, and are all invertible then F\mathcal{F} is called a coordinate transformation.

The Jacobian is non-zero over a region UU in Rn\mathbb R^n iff the corresponding transformation T\mathcal{T} is locally bijective in that region UU.

The Jacobian matrix of the inverse transformation T−1\mathcal{T}^{-1} is the inverse of the Jacobian matrix of T\mathcal{T}

The above was taken from here

Recovering Joint Probability of Discrete Random Variables from Pairwise Marginals

Tensor Algebra

Canonical Polyadic Decomposition

If an NN-way tensor X∈RI1×I2×⋯×INX \in \mathbb R^{I_1 \times I_2 \times \cdots \times I_N} has Canonical Polyadic rank FF, it can be written as

X(i1,i2,⋯ ,iN)=∑f=1Fλ(f)∏n=1NAn(in,f)X(i_1, i_2, \cdots, i_N) = \sum_{f = 1}^{F} \lambda(f) \prod_{n = 1}^N A_n(i_n, f)

where An∈RIn×FA_n \in \mathbb R^{I_n \times F} is called the mode-nn latent factor. In the above, λ=[λ(1),⋯ ,λ(F)]T\lambda = [\lambda(1), \cdots, \lambda(F)]^T with ∥λ∥0=F\|\lambda\|_0 = F is employed to ‘absorb’ the norms of columns.

In other words an NN-way tensor Z∈RI1×I2×⋯×IN\mathcal{Z} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N} representing the joint PMF of NN discrete RVs where Z(x1,x2,⋯ ,xN)=p(X1=x1,X2=x2,⋯ ,XN=xN)\mathcal{Z}(x_1, x_2, \cdots, x_N) = p(X_1 = x_1 , X_2 = x_2, \cdots, X_N = x_N), admits a CPD if it can be decomposed as a sum of FF rank-11 tensors. Denoting a⊗ba \otimes b as the outer-product (Kronecker Product) of two vectors, the CPD model is:

Z=∑f=1Fλ(f)A1(:,f)⊗A2(:,f)⊗⋯⊗AN(:,f)\mathcal{Z} = \sum_{f = 1}^F \lambda(f)A_1(:,f)\otimes A_2(:,f) \otimes \cdots \otimes A_N(:,f)

Here, FF is the rank of the tensor.

Joint PMF Recovery: A Tensor Perspective

Any joint PMF admits a naive Bayes (NB) model representation; i.e., any joint PMF can be generated from a latent variable model with just one hidden variable. It follows that the joint PMF of {Zn}n=1N\{Z_n\}^N_{n=1} can always be decomposed as:

Pr(i1,i2,⋯ ,iN)=∑f=1FPr(f)∏n=1NPr(in∣f)Pr(i_1, i_2, \cdots, i_N) = \sum_{f = 1}^F Pr(f) \prod_{n=1}^N Pr(i_n | f)
where $$Pr(f) := Pr(H = f)$$ is the prior distribution of a latent variable $$H$$ and $$Pr(i_n|f) := Pr(Z_n = z_n^{(i_n)} | H = f)$$ are the conditional distributions.
Type to search the whole site.