This article needs attention from an expert in Statistics. Please add a reason or a talk parameter to this template to explain the issue with the article. WikiProject Statistics may be able to help recruit an expert. (November 2008)

The partition of sums of squares is a concept that permeates much of inferential statistics and descriptive statistics. More properly, it is the partitioning of sums of squared deviations or errors. Mathematically, the sum of squared deviations is an unscaled, or unadjusted measure of dispersion (also called variability). When scaled for the number of degrees of freedom, it estimates the variance, or spread of the observations about their mean value. Partitioning of the sum of squared deviations into various components allows the overall variability in a dataset to be ascribed to different types or sources of variability, with the relative importance of each being quantified by the size of each component of the overall sum of squares.

Background[edit]

The distance from any point in a collection of data, to the mean of the data, is the deviation. This can be written as yi−y¯displaystyle y_i-overline y, where yidisplaystyle y_i is the ith data point, and y¯displaystyle overline y is the estimate of the mean. If all such deviations are squared, then summed, as in ∑i=1n(yi−y¯)2displaystyle sum _i=1^nleft(y_i-overline y,right)^2, this gives the "sum of squares" for these data.

When more data are added to the collection the sum of squares will increase, except in unlikely cases such as the new data being equal to the mean. So usually, the sum of squares will grow with the size of the data collection. That is a manifestation of the fact that it is unscaled.

In many cases, the number of degrees of freedom is simply the number of data in the collection, minus one. We write this as n − 1, where n is the number of data.

Scaling (also known as normalizing) means adjusting the sum of squares so that it does not grow as the size of the data collection grows. This is important when we want to compare samples of different sizes, such as a sample of 100 people compared to a sample of 20 people. If the sum of squares was not normalized, its value would always be larger for the sample of 100 people than for the sample of 20 people. To scale the sum of squares, we divide it by the degrees of freedom, i.e., calculate the sum of squares per degree of freedom, or variance. Standard deviation, in turn, is the square root of the variance.

The above information is how sum of squares is used in descriptive statistics; see the article on total sum of squares for an application of this broad principle to inferential statistics.

Partitioning the sum of squares in linear regression[edit]

Theorem. Given a linear regression model yi=β0+β1xi1+⋯+βpxip+εidisplaystyle y_i=beta _0+beta _1x_i1+cdots +beta _px_ip+varepsilon _i including a constant β0displaystyle beta _0, based on a sample (yi,xi1,…,xip),i=1,…,ndisplaystyle (y_i,x_i1,ldots ,x_ip),,i=1,ldots ,n containing n observations, the total sum of squares TSS=∑i=1n(yi−y¯)2displaystyle mathrm TSS =sum _i=1^n(y_i-bar y)^2 can be partitioned as follows into the explained sum of squares (ESS) and the residual sum of squares (RSS):

TSS=ESS+RSS,displaystyle mathrm TSS =mathrm ESS +mathrm RSS ,

where this equation is equivalent to each of the following forms:

‖y−y¯1‖2=‖y^−y¯1‖2+‖ε^‖2,1=(1,1,…,1)T,∑i=1n(yi−y¯)2=∑i=1n(y^i−y¯)2+∑i=1n(yi−y^i)2,∑i=1n(yi−y¯)2=∑i=1n(y^i−y¯)2+∑i=1nε^i2.displaystyle hat varepsilon right

Proof[edit]

∑i=1n(yi−y¯)2=∑i=1n(yi−y¯+y^i−y^i)2=∑i=1n((y^i−y¯)+(yi−y^i)⏟ε^i)2=∑i=1n((y^i−y¯)2+2ε^i(y^i−y¯)+ε^i2)=∑i=1n(y^i−y¯)2+∑i=1nε^i2+2∑i=1nε^i(y^i−y¯)=∑i=1n(y^i−y¯)2+∑i=1nε^i2+2∑i=1nε^i(β^0+β^1xi1+⋯+β^pxip−y¯)=∑i=1n(y^i−y¯)2+∑i=1nε^i2+2(β^0−y¯)∑i=1nε^i⏟0+2β^1∑i=1nε^ixi1⏟0+⋯+2β^p∑i=1nε^ixip⏟0=∑i=1n(y^i−y¯)2+∑i=1nε^i2=ESS+RSSdisplaystyle beginalignedsum _i=1^n(y_i-overline y)^2&=sum _i=1^n(y_i-overline y+hat y_i-hat y_i)^2=sum _i=1^n((hat y_i-bar y)+underbrace (y_i-hat y_i) _hat varepsilon _i)^2\&=sum _i=1^n((hat y_i-bar y)^2+2hat varepsilon _i(hat y_i-bar y)+hat varepsilon _i^2)\&=sum _i=1^n(hat y_i-bar y)^2+sum _i=1^nhat varepsilon _i^2+2sum _i=1^nhat varepsilon _i(hat y_i-bar y)\&=sum _i=1^n(hat y_i-bar y)^2+sum _i=1^nhat varepsilon _i^2+2sum _i=1^nhat varepsilon _i(hat beta _0+hat beta _1x_i1+cdots +hat beta _px_ip-overline y)\&=sum _i=1^n(hat y_i-bar y)^2+sum _i=1^nhat varepsilon _i^2+2(hat beta _0-overline y)underbrace sum _i=1^nhat varepsilon _i _0+2hat beta _1underbrace sum _i=1^nhat varepsilon _ix_i1 _0+cdots +2hat beta _punderbrace sum _i=1^nhat varepsilon _ix_ip _0\&=sum _i=1^n(hat y_i-bar y)^2+sum _i=1^nhat varepsilon _i^2=mathrm ESS +mathrm RSS \endaligned

The requirement that the model includes a constant or equivalently that the design matrix contains a column of ones ensures that ∑i=1nε^i=0displaystyle sum _i=1^nhat varepsilon _i=0.

The proof can also be expressed in vector form, as follows:

SStotal=‖y−y¯‖2=‖y−y¯+y^−y^‖2,=‖(y^−y¯)+(y−y^)‖2,=‖y^−y¯‖2+‖ε^‖2+2ε^T(y^−y¯),=SSregression+SSerror+2ε^T(Xβ^−y¯),=SSregression+SSerror+2(ε^TX)β^−2ε^Ty¯,=SSregression+SSerror.displaystyle beginalignedSS_texttotal=Vert mathbf y -bar mathbf y Vert ^2&=Vert mathbf y -bar mathbf y +mathbf hat y -mathbf hat y Vert ^2,\&=Vert left(mathbf hat y -bar mathbf y right)+left(mathbf y -mathbf hat y right)Vert ^2,\&=Vert mathbf hat y -bar mathbf y Vert ^2+Vert hat varepsilon Vert ^2+2hat varepsilon ^Tleft(mathbf hat y -bar mathbf y right),\&=SS_textregression+SS_texterror+2hat varepsilon ^Tleft(Xhat beta -bar mathbf y right),\&=SS_textregression+SS_texterror+2left(hat varepsilon ^TXright)hat beta -2hat varepsilon ^Tbar mathbf y ,\&=SS_textregression+SS_texterror.endaligned

The elimination of terms in the last line, used the fact that

ε^TX=(y−y^)TX=yT(I−X(XTX)−1XT)X=yT(X−X)=0.displaystyle hat varepsilon ^TX=left(mathbf y -mathbf hat y right)^TX=mathbf y ^T(I-X(X^TX)^-1X^T)X=mathbf y ^T(X-X)=mathbf 0 .

Further partitioning[edit]

Note that the residual sum of squares can be further partitioned as the lack-of-fit sum of squares plus the sum of squares due to pure error.

See also[edit]

• 
  Inner-product space
  
  • 
    Hilbert space
    
    • Euclidean space
  
  
  • Orthogonality
  
  
  • 
    Orthonormal basis
    
    • 
      Orthogonal complement, the closed subspace orthogonal to a set (especially a subspace)
    
    
    • 
      Orthomodular lattice of the subspaces of an inner-product space
    
    
    • Orthogonal projection
    
    
  
  
  • 
    Pythagorean theorem that the sum of the squared norms of orthogonal summands equals the squared norm of the sum.
  
  


• Least squares


• Mean squared error


• Squared deviations

References[edit]

• 
  Bailey, R. A. (2008). Design of Comparative Experiments. Cambridge University Press. ISBN 978-0-521-68357-9..mw-parser-output cite.citationfont-style:inherit.mw-parser-output .citation qquotes:"""""""'""'".mw-parser-output .citation .cs1-lock-free abackground:url("//upload.wikimedia.org/wikipedia/commons/thumb/6/65/Lock-green.svg/9px-Lock-green.svg.png")no-repeat;background-position:right .1em center.mw-parser-output .citation .cs1-lock-limited a,.mw-parser-output .citation .cs1-lock-registration abackground:url("//upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Lock-gray-alt-2.svg/9px-Lock-gray-alt-2.svg.png")no-repeat;background-position:right .1em center.mw-parser-output .citation .cs1-lock-subscription abackground:url("//upload.wikimedia.org/wikipedia/commons/thumb/a/aa/Lock-red-alt-2.svg/9px-Lock-red-alt-2.svg.png")no-repeat;background-position:right .1em center.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registrationcolor:#555.mw-parser-output .cs1-subscription span,.mw-parser-output .cs1-registration spanborder-bottom:1px dotted;cursor:help.mw-parser-output .cs1-ws-icon abackground:url("//upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/12px-Wikisource-logo.svg.png")no-repeat;background-position:right .1em center.mw-parser-output code.cs1-codecolor:inherit;background:inherit;border:inherit;padding:inherit.mw-parser-output .cs1-hidden-errordisplay:none;font-size:100%.mw-parser-output .cs1-visible-errorfont-size:100%.mw-parser-output .cs1-maintdisplay:none;color:#33aa33;margin-left:0.3em.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registration,.mw-parser-output .cs1-formatfont-size:95%.mw-parser-output .cs1-kern-left,.mw-parser-output .cs1-kern-wl-leftpadding-left:0.2em.mw-parser-output .cs1-kern-right,.mw-parser-output .cs1-kern-wl-rightpadding-right:0.2em Pre-publication chapters are available on-line.


• 
  Christensen, Ronald (2002). Plane Answers to Complex Questions: The Theory of Linear Models (Third ed.). New York: Springer. ISBN 0-387-95361-2.
  


• 
  Whittle, Peter (1963). Prediction and Regulation. English Universities Press. ISBN 0-8166-1147-5.
  

  Republished as: Whittle, P. (1983). Prediction and Regulation by Linear Least-Square Methods. University of Minnesota Press. ISBN 0-8166-1148-3.
  



• 
  Whittle, P. (20 April 2000). Probability Via Expectation (4th ed.). Springer. ISBN 0-387-98955-2.