...  ...  @@ 163,54 +163,53 @@ At this point, after our discussions on regression, classification and clusterin 





Statistical independence is expressed in terms of PDFs. Two variables i & j are independent if the [joint PDF](https://en.wikipedia.org/wiki/Joint_probability_distribution) for i & j is:






´´´math



```math



p_{xy}(i,j)=p_i(i)p_j(j)



´´´



```






In simpler terms, it means if we know the PDFs of i & j, their joint PDF can be constructed. This implies that:






´´´math



```math



E(i^p,j^q) = E(i^p)E(j^q)



´´´



```






Let's start with something we already see in the lecture, p=q=1:






´´´math



```math



E(i,j) = E(i)E(j)



´´´



```






where ´$E(i,j)$´ is the covariance between i and j (both i and j are zero mean):



where `$E(i,j)$` is the covariance between i and j (both i and j are zero mean):






´´´math



```math



E(i,j) = \sum_{n=1}^{N} i_nj_n



´´´



```






"Correlation" is nothing but the normalized covariance:






´´´math



```math



\rho(i,j) = \sum_{n=1}^{N} i_nj_n / \sigma_i\sigma_j






´´´



```






where ´$\sigma$´ is the standard deviation:






´´´math



```math



\sigma_i = E(i,i)^0.5, \sigma_j = E(j,j)^0.5



´´´



```






We already talked in the lecture that this normalization helps us to scale between 1 and 1. To give you vivid examples, let's see the limits. ´$\rho(i,j)=1$´ means that i and j increases in proportion (j in proportion to i). 0 means j does not increase or increase (in average behavior of course) as x increases. If ´$\rho(i,j)=1$´, y decreases in proportion to x, as it increases.



We already talked in the lecture that this normalization helps us to scale between 1 and 1. To give you vivid examples, let's see the limits. `$\rho(i,j)=1$` means that i and j increases in proportion (j in proportion to i). 0 means j does not increase or increase (in average behavior of course) as x increases. If `$\rho(i,j)=1$`, y decreases in proportion to x, as it increases.






To check whether correlation exists or not, we look at the a limited case, the first moment of the joint PDF (p=q=1) and say that they are uncorrelated if:






´´´math



```math



E(i^p,j^q) = E(i^p)E(j^q)



´´´



```






In order to claim that it is independent, the general form of the equation must be satisfied (p>0, q>0; where p & q are integers). So our criteria is:






´´´math



```math



\rho(i^p,j^q) = 0



´´´



```










...  ...  