Terms & Definitions

The core structures in liner algebra are vectors and matrices.

Vectors

A vector is an ordered sequence of values, typically real numbers. In programming terms, we would represent it as an array (i.e., a 1-“dimensional” NumPy ndarray). We denote vectors either with the vector symbol \(\vec{x}\) or as bold-face lowercase letters \(\mathbf{x}\); to write out the content of a vector, we usually use angle brackets, e.g. \(\mathbf{x} = \langle 1, 5, 0 \rangle\) or \(\mathbf{x} = \langle x_1, x_2, \dots, x_n \rangle\). For a vector \(\mathbf{x}\), we denote individual elements of the vector with a subscripted variable \(x_i\), where \(i\) is the position in the vector (starting from 1).

Mathematics, including linear algebra and statistics, almost universally uses 1-based indices to reference vector members, data points, etc. Python, C, Rust, and many other programming languages typically start from 0 for indexing lists and arrays. This is something you need to be pay careful attention to when translating mathematical formulæ to code. R, Matlab, and Fortran conventionally use 1-based indexing.

The number of values \(n\) is called the dimension of the vector. To say that a vector is a real-valued vector with dimension \(n\), we write \(\mathbf{x} \in \Reals^n\).

Matrices

A matrix is a grid of numbers (a 2D array, in programming terms). For example, the following is a \(4 \times 4\) matrix \(M \in \Reals^{4 \times 4}\):

\[M = \begin{bmatrix} 2 & 3 & 0 & 5 \\ 1 & 9 & -2 & 0 \\ 3 & 2 & -5 & 7 \\ -2 & 6 & 1 & 4 \end{bmatrix}\]

We conventionally use uppercase letters to denote matrices (and sets). Individual elements of the matrix are denoted with subscripted lowercase, with the row number coming first. For the matrix above, \(m_{1,2}=9\). When we are using variables as subscripts, we may omit the comma, as in \(m_{ij}\).

The rows and columns of the matrix form vectors, which we also reference with subscripts while specifying whether we are referring to a row or column vector. For \(M\), the column vector \(\mathbf{m}_3 = \langle 0, -2, -5, 1 \rangle\), while the row vector \(\mathbf{m}_1 = \langle 2, 3, 0, 5 \rangle\).

As with a vector, we denote the dimension of a matrix by using a superscript on the domain of the matrix values: a matrix \(A\) with \(m\) rows and \(n\) columns of real numbers is denoted \(A \in \Reals^{m \times n}\).

There are a few special kinds of matrices. A diagonal matrix is a matrix is zero everywhere except along the diagonal starting at the top left. That is, \(i = j \Rightarrow m_{ij} = 0\), for example:

\[M = \begin{bmatrix} 2 & 0 & 0 & 0 \\ 0 & 9 & 0 & 0 \\ 0 & 0 & -5 & 0 \\ 0 & 0 & 0 & 4 \end{bmatrix}\]

Diagonal matrices are usually square.

An identity matrix, usually denoted \(I \in \Reals^{n \times n}\) or \(\mathbb{1} \in \Reals^{n \times n}\), is a square diagonal matrix whose diagonal is all 1.

Other Domains

Above, we have used the real numbers \(\Reals\) as the domain, or the set of valid valies for a matrix or vector. These are not the only valid values, though; in principle, any set can be used.

If we have a vector matrix of binary values 0 and 1, we may write \(\mathbf{x} \in \{0,1\}^n\). If our values are known to be in the range 0–1, we may write \(\mathbf{x} \in [0,1]^n\) (\([a,b]\) is the standard notation for the closed interval from \(a\) to \(b\): \(\{x \in \Reals: a \le x \le b\}\)).

Term Cross-Reference

The terminological overlap between mathematical dimension (number of elements in a vector) and programming dimension (whether you have a 1D array, 2D matrix, 3D, etc.) is confusing. Vectors also have a mathematical length, which is different from the mathematical dimension (see Vector Norms). When programming languages talk about the length of an array or list, e.g. with the Python len(array) function, that is the dimension in linear algebra.

Dimension

• In linear algebra, the number of elements in a vector or matrix.
• In programming (including NumPy), the number of dimensions along which we can index an array, such as 1D for a vector, 2D for a matrix, and 3D for a tensor.

Length

• In linear algebra, the \(L_2\) norm of a vector (see Vector Norms).
• In programming, the number of elements in an array, along a specific dimension (equivalent to the linear algebra dimension).

To reduce confusion, I rarely use the term “length” to refer to mathematical length, preferring “\(L_2\) norm” or “Euclidean length”.

Beyond Matrices

A tensor is a generalization of a matrix to more than 2 dimensions.

A scalar is a single value (not a vector or matrix).