This webpage corresponds to 10.1 and 10.2 of Session 10 of the English website. Here we show some examples of basic statistics for the relationship between two variables (1) when both or one of X and Y are qualitative data.

In expressing relationships between variables:

X: the cause in the hypothesis, called the independent variable or explanatory variable
Y: the outcome in the hypothesis, called the dependent variable because its values are dependent on some given independent variables, or response variable.

Please refer to the short video below which shows that when XY is both qualitative variables we use cross tabulation, and when X is a qualitative variable and Y is a quantitative variable, we compare means.

When conducting a \(\chi^2\) test, we often use more convenient tools for statistics than Excel, so it is not important to learn the Excel operations. Watch the demonstration video to understand the meaning of \(\chi^2\) test. We use the data we created in Chapters 2 and 3 (3.1.1) (specifically, the downloaded CopyFromHere01.xlsx combined with CopyFromHere03.xlsx). See Pivot Table (Session 03) for how to create a cross-tabulation table in Excel for your reference.

Does the type of car rented vary from city to city? Since cars rented in each city is the population, and the data we have on car rentals is a sample, this task involves statistical inference. Data usually show different percentages. Even if data are taken from the population where rental car types do not vary from city to city, it is highly unlikely that each car type has the exact same percentage in a sample. We perform statistical inference of whether the differences in the data are just a matter of chance caused by the sampling or whether they are due to differences in the population. The null hypothesis is that "car types rented are not different by city". Using theoretical sampling distribution, we test whether the null hypothesis can be rejected. In this case, since both branch and car type are qualitative variables, we create a crosstabulation table and the video begins from here. For reasons we will discuss later, we will create a crosstabulation table with only car types A to C.

Next, create the table expected when XY were not related, that is, if the null hypothesis is true. It is called expected frequency table. The expected frequency of each cell is: \[expected \, frequency \, of \, row \, i, \, column \, j = N×\frac{total \, of \, row \, i}{N}×\frac{total \, of column \, j}{N}\] N is the total number of cells. The hat (^) symbol indicates expected value, f is frequency, i is row i, and j is column j. The expected frequency of the cell in row i, column j \(\hat{f_{ij}}\) can be expressed by the following formula: \[\hat{f_{ij}}=\frac{(f_i.)(f_.j)}{N}\]

How far are the observed values from the expected frequency? If we take the difference of each cell and add them together, the positive and negative values will cancel each other out and become zero, so we square the difference. Then divide it by the expected value to adjust for size, and sum it for all cells in the table.
\[\chi^2=\displaystyle\sum_{i=1}^{R} \sum_{j=1}^{C} \frac{(\hat{f}_{ij}-f_{ij})^2}{\hat{f}_{ij}}\] Where:
 ∑ is total
 R is the number of rows
 C is the number of columns
This value is called chi-square, and the larger this value is, the more the observed frequencies deviate from the expected frequencies.

No matter how large \(\chi^2\) is, however, we cannot say that the value is absolutely improbable even when the null hypothesis is true. Therefore, if the null hypothesis can be said to be implausible to a certain extent, we reject the null hypothesis and adopt the alternative hypothesis that says there is a relationship between X and Y. That's how statistical testing is done. Significance level, the level at which the null hypothesis is rejected, is set by the analyst. The significance level α is set to a close value such as 0.05 or 0.01. We compare the p value (significance probability) with α. The p value is the probability of the test statistic from our sample and values less likely to occur than than that value when the null hypothesis is true,. We reject the null hypothesis if p < α. There is a lot of discussion about this p-centered approach, but it is important to understand this traditional approach in any case. The probability distribution of chi-squared depends on degrees of freedom df.

df in cross tabulation tables: (R-1)×(C-1)

In the video, Chika-chan calculated the chi-squared in a spreadsheet, and from there used the RT function to find the the right tail probability, that is, the probability of taking a value greater than its chi-squared in the chi-squared distribution. Then Chika-chan used CHISQ.TEST function to obtain the p value to show the results are consistent. When p ≧ α, the null hypothesis cannot be rejected. The results are reported as, for example,

"The relationship is not statistically significant by a chi-squared test, \(\chi^2\) = 13.89, df = 12, and p=0.308. Therefore, there is no statistically significant relationship between cities and the types of the rented cars (A to C)."

As you can see, conducting a test itself is easy, and even easier if you use some statistical software. Therefore, what you need to know is the meaning of the analysis, when to use it, and when not to use it. There are some situations in which we should not use the chi-square test: when there are more than 20% of cells with an expected frequency of less than 5 or when there are cells with a minimum expected frequency of less than 1, we should not use the chi-square test. I included only car models A through C in the analysis because including car models D and beyond would violate the criteria.

Since we are now talking about statistical inference of the quantitative variable, we conceptualize what I call three types of distributions of quantitative data.
 The first-type distribution: distribution of values in the population
 The second-type distribution: distribution of values in a sample
 The third-type distribution: probability distribution of a statistic
The first distribution is the distribution that the analyst wants to know about. Numbers that represent population characteristics that the analyst wants to know by conducting analyses is called parameter. Since it is often not possible to conduct a complete enumeration of the entire population, a sample survey is conducted and the parameters are estimated from data extracted as a sample from the population. That is statistical inference. What the analyst can actually observe is the values of the data in a single extracted sample, and the distribution of those values is the second distribution. The third distribution is the theoretical distribution of a statistic, assuming that every possible sample of size N is extracted from the population and that a statistic is calculated from each sample. It is called sampling distribution because such statistic from each sample becomes data. Statistical tests are often conducted using sampling distributions, and the chi-squared distribution used in the chi-square test above is one of the third-type distributions because it is the theoretical probability distribution of the statistic of chi-squared.
The characteristics of symmetrical bell-shaped distributions such as the normal distribution and t-distribution, which are frequently-used third-type distributions, can be summarized with mean and standard deviation.

Let's start with reviewing the two descriptive statistics that you all should have learned in high school, variance and standard deviation, which express the degree of variation of the values of a quantitative variable, using our data taken as complete enumeration (as a second-type distribution with no need for estimation). Suppose your boss asks you a question like this "This year we have had both young and old people using our services, and I think the age variance has increased since last year. To rephrase our task, we need to summerize the magnitude of the variation in statistics and compare the value with that from the last year. Let's experience the concept in Excel. The older people have ages above the mean, the younger people below the mean, and the differences from the mean is called deviation. A large variation means that absolute values of deviations are large overall. The variance based on the entire population s² is the sum of squared deviations divided by N. (Note: If you estimate the variance of the population from a sample, you should use unbiased variance where the sum of squared deviations is divided by degrees of freedom N-1 , not N.) \[s^2=\frac{\sum (Y_i-\hat{Y})^2}{N}\] The standard deviation \(s\) is the square root of variance. \[s=\sqrt{s^2}\]

Experience through the practice that if you would like to have the general degree of variation, that is, the average deviation of age in this year's data, you'll come up with the formula above. Watching the video with the hands-on practice on the "Input Data" worksheet may remind you of the statistics of variance and standard deviation you learned in high school. First, find the column "Age", name the adjacent column "Average" and use AVERAGE function and mixed (or absolute) reference to autofill the average age.

Name the next column D (for deviation), and enter the difference between the individual data and the mean. This difference from the mean is the deviation. Enter a deviation in the top cell of the column, then copy the data by autofill or dragging the fill handle to the bottom, which will result in a relative reference, thus completing the Deviation column. We want to see the total magnitude of the variation from the deviations, but if we simply sum the deviations, the pluses and minuses will cancel each other out.

So, square deviations to make them positive. The next column is D² and enter squared deviation using ^ [caret] operator for exponentiation in Excel. Once you entered a squared deviation in the top cell, fill in the same column in the same way. We want to get the average of the sume of squared deviations, so we will sum them up and divide by N. Enter the sum of squared deviations in any cell using AutoSUM. If you divide the sum of squared deviations by N=200, the returned valued is variance. In another cell, put VAR.P, variance based on the entire population. The returned value of the funccion must match the variance you calculated.

P after the dot in VAR.P is P of population, and this function is used to calculate the variance for complete enumeration data (the data itself is the population). The larger the variance, the larger the scatter of the values, but since it is squared in the calculation process, its value is too large for a measure of standard deviation from the mean.
So, the variance is square-rooted to adjust its size in order to obtain standard deviation. Please compare the value of the standard deviation you calculated yourself with the returned value of the function STDEV.P. In any cells, display the standard deviation using the function and the standard deviation you calculated yourself. See for yourself that they match.

By the way, Excel also has the VAR.S function and STDEV.S function. The S after the dot is the S of sample. When the data are a sample extracted from a population (the first-type distribution) and what you want is not to know the variance/standard deviation of the sample (the second-type distribution) but to estimate the variance/standard deviation of the original population (parameters in the first-type distribution), you should choose this option. It is safe to say that one of the main purposes of statistics is to estimate parameters. What we want to know from a sample are the values that characterize the first distribution: the parameters of the population mean \(\mu_Y\) , the population variance \(\sigma^2\) and the population standard deviation \(\sigma\). What we usually use is unbiased variance, VAR.S, and its square root, the STDEV.S, so STDEV is the same as STDEV. S. In the practice name the column next to D² as STDEV and autofill them with STDEV.S.

Next task. Suppose you are still chatting with your boss. Your boss's nephew is 25-year-old Ren Takahashi, who recently used the services of the company. Your boss is wondering: "How young my nephew Ren is among our service users?" Can you respond to this question?

What statistic would you use to compare positions of given customers in different distributions? You can compare the position of each data within different distributions by adjusting them by making the mean 0 and one standard deviation 1. This is called standard score (z). When you are asked to "standardize values," this means to convert them to standard scores z. Just as z is standardized so that the mean is 0 and the standard deviation is 1, hensachi in Japanese is standardized so that the mean is 50 and the standard deviation is 10. If you are familiar with Japanese hensachi it is easy to conceptualize z by converting the score 50 to 0 and the interval of 10 to 1. Please refer to the image below and try calculating the z scores of Age using the mean and the standard deviation. Compare the z scores you calculated with the returned values of the function STANDARDIZE. Do they match?

Read and understand the Central Limit Theorem and confidence intervals with the t distribution. According to the central limit theorem, the third-type distribution of all possible \(\mu_Y\) is a normal distribution with certain characteristics. Confidence intervals are interval estimates, which can be used to estimate population parameters such as \(\mu_Y\) (the population mean of the variable Y) using t distribution.

First, when trying to estimate mean \(\mu_Y\) and variance \(\sigma_Y^2\), the parameters of the population (the first-type distribution) from the the sample mean \(\overline{Y}\) and variance \(s_Y^2\) of the second-type distribution, the analyst is usually looking at just one sample of size N that happens to be obtained from the population. Suppose that we have all possible combinations of samples of size N from the same population, and we have means \(\overline{Y}\) from all of them. The distribution of \(\overline{Y}\) is a third-type distribution. The standard deviation of this distribution, \(\sigma_\overline{Y}\), has a special name standard error because it is the standard error from the \(\mu_Y\) of \(\overline{Y}\).

According to the Central Limit Theorem, if all possible random samples of size N are drawn from a population with a mean \(\mu_Y\) and a standard deviation \(\sigma_Y^2\), then as N becomes larger, whatever the shape of the first-type distribution, the sampling distribution of sample means becomes approximately normal, with:
  ・ The mean \(\mu_\overline{Y}\) of the third distribution approaches the population mean \(\mu_Y\) of the first distribution
  ・ The standard deviation of the third distribution, the standard error \(\sigma_\overline{Y}\) approaches the following equation
\[\sigma_\overline{Y}=\frac{\sigma_Y}{\sqrt{N}}\]

The z scores of the second-type distribution can be calculated as follows:
\[z=\frac{Y_i-\overline{Y}}{s}\] The z scores of the first-type distribution can be calculated as follows
\[z=\frac{Y_i-\mu}{\sigma}\] Likewise, the z scores of the third-type distribution can be calculated as follows:
\[z=\frac{\overline{Y}-\mu}{\sigma/\sqrt{N}}\] We want to estimate the 95% confidence interval of \(\mu\) based on the z score of the 95% area (probability) of the standard normal distribution, but the problem is that we do not even know the population variance \(\sigma\), so we will use here the t values which uses \(s\) for \(\sigma\) and its probability distribution of t values, i.e., the t distribution . The t distribution changes its shape depending on the degrees of freedom N - 1, and the larger the degrees of freedom, the closer it approaches the standard normal distribution.

As briefly explained in the video, confidence interval of the population mean \(\mu\) is expressed as follows: \[\overline{Y}-t×s/\sqrt{N} ≦ \mu ≦ \overline{Y}+t×s/\sqrt{N}\]

Supposes the following situation. You are still chatting with your boss, and your boss asks another question. "Does the number of card usage differ from branch to branch?" As you know, if X is the branch and Y is the number of times the card is used, then the task is comparison of means. Let's compare means by branch in a bar chart with confidence intervals.

As shown in the video, after creating the pivot table, copy only the mean and standard deviation values and create the columns \(N\), \(\sqrt{N}\), \(\frac{s}{\sqrt{N}}×t_{\alpha/2}\). Although not used in the chart, the LCI (Lower Confidence Interval) and UCI (Upper Confidence Interval) columns are also created for the sake of checking.

After creating the bar chart, customize the width and color of the entire bar chart as you wish. Then, specify an error range of \(\frac{s}{\sqrt{N}}×t_{\alpha/2}\). This completes the bar chart with 95% confidence interval.

In this section, we introduce summary statistics by describe() as descriptive statistics in Python, and in the next section, we introduce t-test as an example of statistical tests in Python. Now, let us display the summary statistics of kingaku.csv with the following code-1.