8323 Factor Analysis 1 Intro

Factor Analysis: 

Factor Analysis Motivation: why factor analysis The Factor Analysis Model. Assumptions and their consequences Estimation of the Factor Model

Slide2: 

Factor analysis – Motivation We consider variables related to some characteristics of European Regions. GDP_PER_CAPITA: Per capita Domestic Gross Product DENSITY: Nr of inhabitants / area of the region (km2) ROAD_AREA: Km of roads (divided by the area of the region) STUDENT_POP_5_29: students (% of population aged 5-29) ISCED3_STUDENTS: student at the 2° educational level (% of students) SATURATION_MKT: size of local market compared to the potential market (%). REAL_G_RATE_GDP: Growth rate of GDP GFCF_GDP: Gross fixed capital formation - net new investment by enterprises in the domestic economy in fixed capital (as a % of GDP) RD_GDP: R&D expenditures (as a % of GDP) PAT_RD: Nr of patents (divided by the R&D expenditures) DENS_LOCAL_UNIT: Nr of local enterprises (divided by the area of the region) WAGES_GDP: Wages in the manufactory sector (as a % of GDP) EMPL_LOCAL_UNITS: Nr of employees working in local enterprises (average) UNEMPL_RATE: Unemployment Rate (calculated on population with age > 15) We are interested in analyzing the attractiveness of European Regions. This latent concept is supposed to be related to the above variables.

Factor analysis: 

X1 X2 X3 The idea is that the observed correlations between the manifest variables can be regarded as the consequence of the relationship between the manifest vars and some latent variables called factors. Thus, correlations reflect the relationship between the variables and the latent factors. In a sense, the correlations between manifest vars are spurious: they are only due to the effect of the latent factors on the variables. F1 X1 X3 X2 X4 X5 The observed variables (called manifest variables) are correlated. F2 X4 X5 Factor analysis Factor analysis is a multivariate technique aiming at explaining the observed correlations between p variables, X1, …, Xp as functions of a small number of latent unobservable variables, called factors

The factor analysis model: 

X1 U1 X2 X3 U2 U3 We admit possibly different “reactions” of the manifest variables to the factors (or, also, different effects of the factors on the variables): ℓjs = loading – impact of the s-th factor on the j-th variable We also admit the existence of peculiar characteristics/information content of the manifest variables, not depending upon factors and, thus, not shared with the other manifest variables. F1 ℓ11 ℓ21 ℓ31 F2 X4 X5 ℓ32 ℓ42 ℓ52 U4 U5 The factor analysis model The factor analysis model

The factor analysis model: 

The factor analysis model The factor model is defined on the basis of 3 sets of variables: A set of p observable vars, X1, X2 …, Xp , called manifest variables A set of r latent not observable vars, the common factors, F1, …, Fr A set of p not observable vars, the specific factors, U1, U2 …, Up In factor analysis a model is specified relating the manifest variables to some (few) common latent factors. THE FACTOR ANALYSIS MODEL: X1 – 1 = ℓ11F1 + ℓ12F2 +…+ ℓ1rFr + U1 X2 – 2 = ℓ21F1 + ℓ22F2 +…+ ℓ2rFr + U2 ……. Xp – p = ℓp1F1 + ℓp2F2 +…+ ℓprFr + Up

The factor analysis model: 

For the j-th manifest variable: Xj – j = ℓj1F1 + ℓj2F2 +…+ ℓjrFr + Uj The factors F1, F2, …, Fr, are common to/have an effect on all the manifest variables – they appear in all the equations. If a factor has non null coefficients with all the manifest variables, it is called a general factor All the factors – common or specific – are latent , unobservable The factor Uj appears only in the equation modelling Xj – it is specific to Xj – and it represents the information content which is peculiar to the j-th manifest variable and which is not shared with the other manifest variables The factor analysis model

The factor analysis model: 

Xj – j = ℓj1F1 + ℓj2F2 +…+ ℓjrFr + Uj Idea: separate the common component from the specific one. The individuation of common factors provides an explanation/description of the inter-relationships (correlations) between manifest variables. Problem: how to estimate the loadings, measuring the importance of the common factors in explaining the manifest variables. The factor analysis model is a linear model, relating the manifest variables to linear combinations of the factors (explanatory variables) plus an error component. Unlike a linear regression model, the entire right-hand side of the model is unobserved Moreover, the p manifest variables are related to (r + p) random variables (r common factors and p specific factors). It is then impossible to verify empirically the adequacy of the model. Under suitable assumptions on the common and specific factors characteristics the correlation and the covariance matrices of the manifest variables can be expressed in a particular form The factor analysis model

The factor analysis model: Assumptions: 

The specific factors are characterized by. E(Uj) = 0 Var(Uj) = jj The specific factors are mutually uncorrelated: Cov(Uj ,Uh) = 0 The common factors are characterized by: E(Fs) = 0 Var(Fs) = 1 The common factors are mutually uncorrelated: Cov(Fs ,Fw) = 0 All the Common factors are uncorrelated with the specific factors: Cov(Fs , Uj) = 0 Xj – j = ℓj1F1 + ℓj2F2 +…+ ℓjrFr + Uj The factor analysis model: Assumptions

The factor analysis model: consequences of assumptions: 

COMMUNALITY Variance of Xj due to the specific factor Variance not shared with the other vars (related to the specific factor) jj SPECIFICITY or UNIQUENESS Variance of Xj explained by the r common factors. Variance shared with the other vars (related to the common factors) h2j For the j-th manifest (random) variable: Var(Xj) = jj = ℓ 2j1 + ℓ 2j2 +…+ ℓ 2jr + jj = h2j + jj The variance of the j-th manifest variable can be divided into two components The factor analysis model: consequences of assumptions

The factor analysis model: consequences of assumptions: 

The factor analysis model: consequences of assumptions i.e. it is the portion of the variance of Xj explained by Fs Also the loadings are related to the covariances between vars: Cov(Xj,Xk) = ℓj1 ℓk1+ ℓj2 ℓk2+ ℓj3 ℓk3+…. It can be shown that the s-th loading in the j-th equation is The loadings are related to that part of the variance and covariance matrix which is due to the common factors Xj – j = ℓj1F1 + ℓj2F2 +…+ ℓjrFr + Uj

The factor analysis model: consequences of assumptions: 

If we consider standardized (random) manifest variables: Zj = ℓj1F1 + ℓj2F2 +…+ ℓjrFr + Uj Var(Zj) = 1 = ℓ 2j1 + ℓ 2j2 +…+ ℓ 2jr + jj = h2j + jj COMMUNALITY = 1- SPECIFICITY COMMUNALITY = % of the variance of Zj (1) explained by common factors If Xj is informative (with respect to the factorial structure) then the communality is high, otherwise the specificity is high Manifest vars with high communality share high information content with the other ones. The squared loading is the % of the variance of Zj explained by Fs h2j + jj = 1 The loadings. In this case (standardized variables), it is: The factor analysis model: consequences of assumptions

The factor analysis model: consequences of assumptions: 

In the following we will consider standardized variables Given the assumptions, the correlation matrix of the manifest (random) variables can be decomposed into two parts. The factor analysis model: consequences of assumptions The first part – related to the common factors – is related to the matrix of the loadings, i.e., the correlations between manifest variables and factors. The second part – related to the specific factors – is a diagonal matrix whose elements are the variances of the specific factors.

Factor analysis versus Principal Components analysis: 

FA: Hypotheses on the structure of the correlation matrix FA: Factors are unobservable variables, while Principal components are clearly defined transformations of the original variables. FA: Attention is focused on the relation Factors manifest variables, In PCA attention is focused on the transformation Manifest variables  Principal Components FA: The aim is to explain/reconstruct the whole correlation matrix. In PCA, attention is focused on the trace of the correlation matrix (describing the most relevant tendencies of the scatter of the data) FA: There is no hierarchy. The factors are standardized variables variance = 1) having all the same (theoretical) relevance. In PCA the components are extracted in a decreasing order of importance and their variance measures their importance. Factor analysis versus Principal Components analysis

The factor analysis model: indeterminacy: 

The factor analysis model: indeterminacy We are interested in estimating L e  Before estimation: PROBLEM Indeterminacy of the (theoretical) solution Consider now a square (r × r) matrix, such that: C: CCT = I.. It will be: The new loadings L* = LC and the new factors, F * = CTF satisfy the assumptions, and the factor analysis model too. Suppose that two matrices L e  exist satisfying the equation above

The factor analysis model: rotations: 

The factor analysis model: rotations Let C be a square (r × r) matrix, such that: C: CCT = I.. Consider: L* = LC and F * = CTF These factors are obtained by applying an orthogonal transformation to the original ones. Such orthogonal transformations are simply rigid rotations of the axes of the factorial space spanned by the factors in F. Since L* = LC and F * = CTF still satisfy the assumptions and the model, a factorial solution is only unique up to an orthogonal transformation. This indeterminacy of the solution has not deterred the use of factor analysis. Actually, this indeterminacy it permits the examination of a variety of solutions (orthogonal transformations of one solution) for the purpose to select the most useful (according to some specified criteria). Let L e  be two solutions satisfying the factor model

Slide16: 

Estimation of the factor model We now illustrate how the factor model (loadings) can be estimated. Consider the standardized variables and the correlation matrix P. Var(Zj) = 1 = h2j + jj Remember that under the factor model’s assumptions, it is: P* : reduced correlation matrix

Slide17: 

Estimation of the factor model Consider now the sample correlation matrix which is related to the common component underlying R Two problems: How should we estimate the communalities ? How should we estimate the loadings, i.e., how to extract the loadings from the reduced matrix? We are interested to the reduced sample correlation matrix:

Slide18: 

Estimation of the factor model Remember: a square matrix can be decomposed using its eigenvectors and eigenvalues In this way, we would consider p factors. But a factor model should be based upon a nr of factors lower than p, the nr of manifest variables. Hence, consider only the first r eigenvalues/eigenvectors of R* and set: Of course, this approximation of R* based upon r factors is adequate only if the last (p – r) eigenvalues are small) Assume initially that communalities are known (so that all the elements of the reduced correlation matrix are known).

Slide19: 

Estimation of the factor model We can now obtain estimates of the loadings

Slide20: 

The communalities can estimated on the basis of the estimated loadings: Estimation of the factor model Remember that for standardized variables Var(Zj) = 1 = ℓ 2j1 + ℓ 2j2 +…+ ℓ 2jr + jj = h2j + jj And for the specificities we have:

Slide21: 

This is what is not related to factors. Notice that specificities are explicitly ‘admitted’ in the model, since we are prepared to observe some manifest variables isolated and/or not related to the factors. Instead, the factor model theoretically should explain all the observed correlations. Hence the off-diagonal elements in that matrix are errors. Residual matrix (with uniqueness on the diagonal): Estimation of the factor model

Slide22: 

Remember now that in order to obtain the described results we assumed that the communalities (or at least suitable estimates of the communalities) were available. How do we select the initial values for the communalities? We refer here only to two methods Principal Components Method Principal Factors Method (notice that other methods have been proposed in literature, being not based upon the decomposition of the correlation matrix, but we will not consider them) Estimation of the factor model

Slide23: 

Principal Components method The reduced matrix coincides with the correlation matrix. This means that we choose 1 as the initial estimate of the communalities. Notice that this means that we are estimating the factor model by hypothesizing that no specificities are included in the model. This method is then focused on the maximization of the communalities, i.e., on the explanation of the variance. It may result in a residual matrix with lower elements on the diagonal (unexplained variances, specificities) and higher elements out of the diagonal (unexplained correlations) as compared to those characterizing the principal factors method (see later). Estimation of the factor model

Slide24: 

Principal Components method The eigenvalues/eigenvectors obtained by applying the Principal Components method are those characterizing the correlation matrix. Of course, these eigenvectors/eigenvalues are those the Principal Components are based upon. Actually, in this case, if r factors are extracted, the loadings are simply the correlations between the original variables and the first r PC’s. This means that we are substantially extracting the first r Principal Components. These components will “play the role of the factors” with some differences which we will describe later. Nevertheless, to use the PC’s as factors, we have to remember that factors are characterized by variances equal to 1. Thus, the PC’s should be standardized before they can be used to estimate factors. Estimation of the factor model

Slide25: 

Principal Factors method Estimation of the factor model In order to obtain a preliminary estimate of the communalities, consider the variance (1) of one manifest variable. The communality can be seen as the proportion of variance (information) which is shared with the other manifest variables (related to the common factors). Two possible ways to measure this shared proportion of variance are: 1. Squared multiple correlation coefficient (smc). The communality of the j-th variable, Zj , is estimated as the R2 of the regression of Zj on all the other manifest vars. Thus, the communality is the % of variance of Zj which is jointly (linearly) explained by all the other variables. 2. Maximum squared correlation coefficient (max). The squared correlation coefficient between two variables, is the % of the variance of one variable which is explained by the other one. The communality of the j-th variable, Zj , is estimated as the highest proportion of the variance of Zj explained by another manifest variable:

Slide26: 

Principal Factors method Estimation of the factor model The reduced matrix is defined on the basis of estimated communalities. Hence, it is not a correlation matrix / has not the properties of a correlation matrix. In particular, some of the eigenvalues of the reduced matrix may be negative. The Principal factors method leads to an estimation of the loadings and of the specificities, as described above. This method does not immediately provide also an estimation of the factors scores for each observation. Nevertheless, also the factors may be estimated by using proper regression techniques.

Slide27: 

Application of the factor model The main aim of factor analysis is to describe the factor model, i.e. to find and describe the latent factors. Here we illustrated how factor loadings and specificities may be obtained from the correlation matrix, properly reduced. Nevertheless, many relevant questions remain opened. INTERPRETATION OF THE RESULTS How many factors should we consider? (how many latent factors are underlying the manifest variables?) How can we interpret the obtained factors? Which is the meaning of these latent variables? How do the factors reproduce the correlation matrix? How much ‘strong’/’stable’ are factors? Do they depend upon the extraction method? FACTOR SCORES In the previous slides we also mentioned how it is possible to estimate the factors, i.e., to assign factor scores to observations. In many applications we will be interested in evaluating how the observations “perform” with respect to factors. When this is the case, as in PCA, another question arises, i.e., How well do the factors describe the observations?