Evaluating Process Changes

Q. We recently made some process changes. How can we determine whether these changes had a significant impact on our process quality?

A. There are several different tools for visualizing and testing the efficacy of process changes. Control charts with phases are useful, and box plots and t-test comparisons can be used to determine the significance of the process change. To use these tools, it is necessary to assign a variable to indicate which observations are from the original process and which observations are from the changed process. For example, create a variable called "Process," where "Old" indicates the original process and "New" represents the changed process. This variable can then be used to group the data into two individual phases.

The following chart shows a process with a target width of 85. The first ten points are the original process and the second ten points are the changed process. Using this chart, it is difficult to determine whether or not the process changes were effective since the center line and the control limits are calculated using data from both processes.

To more clearly see any differences that occurred based on your change in the process, a control chart with Phases is recommended. Using Statit products, this can be easily implemented by clicking on the "Phases" buttons in the control chart dialog. In the Phases dialog, click the "Calculation categorical variable" box and select the "Process" variable to distinguish the individual phases. Based on this control chart, we can see that the center line of the new process has apparently shifted downward somewhat, closer to the target width of 85.

As with other statistical processes, a shift that is visible on a chart may or may not be statistically significant, depending on the amount of data and its variability. The box plot clearly shows the location and spread of the data. In the box plot below, the data variable is Width and the data is grouped by Process.

From the box plot, it appears that the variances of the two processes are about the same. At the same time, it appears that the median may have decreased a bit. These hypotheses can be tested easily using the standard statistical tests.

To test whether the mean has changed, a standard two-sample t-test is performed. This test is found under Statistics > One and Two-Sample Inference > Two-Sample > Compare Location. The Width variable is tested against the Groups defined by the Process variable ("Old" versus "New"). The results are shown below:

Student's t-test for Variables:                          Width (Old)      N = 50                          Width (New)      N = 50                   Width (Old)   130   Width                             *|1    |                              |     |                                |     |                              |     |                              |     |                              |    1|*          ********************|34 32|******************                      ********|15 17|**********                              |     |                                70   At least one of these samples is quite skewed, and thus the population means may not be very useful as measures of location. It might be better to make some transformation of the data, such as logarithmic, before this type of analysis.   The procedure 'compare' provides diagnostics and suggests options.   However, if one nevertheless wants inference about the means, the t-test is not highly sensitive to lack of normality.    There is no significant difference between the means.    If in fact the populations means were THE SAME, the chance of this much evidence of a greater value for variable Width (Old) than for variable Width (New) would be approximately:          One-sided P-value = 0.1826.     The chance of this much evidence in EITHER direction would be twice that value, that is:         Two-sided P-value = 0.3653.       Inferences for the difference in population means for,        Width (Old) - Width (New)    are:             Estimate                 Standard Error               0.912                       1.003                          Confidence limits                     90% :  -0.753002 to 2.577002                     95% :  -1.077787 to 2.901787                     99% :  -1.721973 to 3.545973            Student's t statistic           Degrees of freedom                    0.910                           98                       Sample Means     Standard Error   Sample Size    Width (Old)          86.0000           0.8978           50    Width (New)          85.0880           0.4465           50

Based on this analysis, the process changes did not result in a statistically significant change in the mean. This conclusion is based on the p-value, which is .3653. This can be interpreted as indicating that we could observe a difference of the size seen about 36% of the time, even when there was no actual difference in the means. The data does therefore not provide enough evidence to conclude that the means are different.

In a similar fashion, the hypothesis about the variance of the data being the same is tested using Statistics > One and Two-Sample Inference > Two-Sample > Compare Dispersion. The Width variable is tested against the Groups defined by the Process variable ("Old" versus "New").

Comparison of variances for Variables:                               Width (Old)      N = 50                             Width (New)      N = 50                           Width (Old)   130   Width (New)                                    *|1    |                                     |     |                                     |     |                                     |     |                                     |     |                                     |    1|*                 ********************|34 32|******************                             ********|15 17|**********                                     |     |                                       70     At least one of these samples is quite skewed, and thus the population variances may not be very useful as measures of dispersion. It may be more useful to use this procedure with the interquartile range option.   Alternatively, it might be better to make some transformation of the data, such as logarithmic, before this type of analysis.     However, if one nevertheless wants inference about the variances, the Levene method used by this procedure is not highly sensitive to lack of normality.     There is no significant difference between the variances.     If in fact the populations variances were THE SAME, the chance of this much evidence of a greater value for variable: Width (Old)   than for variable: Width (New) would be approximately:          One-sided P-value = 0.2348.    The chance of this much evidence in EITHER direction would be twice that value, that is:         Two-sided P-value = 0.4696.      This procedure uses as default Levene's method, for both the test and confidence intervals.  The F-test, which is an option, should not ordinarily be used for this purpose.      Inferences for the ratio of population standard deviations for,         Width (Old) / Width (New)     are:              Estimate                Approx. Standard Error               2.011                       0.284                      Approximate Confidence Limits                     90% :  0.721888 to 1.787993                     95% :  0.627444 to 1.909192                     99% :  0.441655 to 2.164697               Test of equal variance using Levene's method               Levene's t Statistic     Degrees of Freedom                    0.726                       98                  One-sided P-value = 0.2348               Standard Deviations   Approx. Std. Error   Sample Size    Width (Old)           6.3484              0.6348             50    Width (New)           3.1570              0.3157             50

With this procedure we are testing the hypothesis that the standard deviations from the two processes are equal. Based on the two-sided p-value of .4696, we cannot reject this hypothesis. Therefore, the variance did not change when the process changed.

If you would like additional information, please call our Support staff at (541) 752-4100 or send email to .