At the Benjamin Franklin School District, Predictive Analytics was used at their high school to identify how their 11th grade students are predicted to perform on the College Board's SAT. Prior to using Predictive Analytics, the district's data teams examined reports from the College Board on current SAT performance but could not draw connections between the SAT data and curriculum data. Predictive Analytics allowed them to make these connections and develop curricular changes based on the SAT predictions.
After uploading their course data from their Student Information System (SIS) and their SAT data from the College Board, Predictive Analytics was then able to transform these data into something more meaningful. In the sample output below, Benjamin Frankling School District was able to see how their English and Science courses contribute to the "Words in Context" subscore on the SAT.
Words in Context (Base Score: 4.0)
|English III||+ 1.5*||
In this table, we can see that a student who takes English III and Honors Chemistry is predicted to be "in the green." In other words, these students are predicted, on average, to be ready for college per the College Board's standards. To arrive at these students' predicted score, the base score, 4.0, is added to the English III score, 1.5, and lastly to the "Change," 5.4. For students who take English III and Chemistry I, they are predicted to have an average score of 7 for Words in Context, which is "in the yellow." Per the College Board, these students are predicted to not be ready for college but are progressing towards that goal.
The asterisks next to each number help convey the confidence with these predictive data. A number with three asterisks denotes a 1-in-1000 probability of occuring due to chance. Two asterisks are a 1-in-100 probability and one asterisk is a 1-in-10 proability of occurring due to chance. Having one to three asterisks next to a number demonstrates there is a high degree of confidence that these prediction numbers can occur in the future.
For this particular result, the Benjamin Franklin high school can see that English courses AND science courses both contribute to how their students perform on the Words in Context metric. From these data, Benjamin Franklin high school determined that they need to reshape their English and science curricula to focus more on the reading and writing skills outlined by the College Board for "Words in Context." However, how did the high school administrators know that changing the curricula would have the needed impact on SAT performance? By viewing the "R-squared" pie chart in their report (see below), the school knew that it had the power to change student performance:
This pie chart explains all the factors that contribute to how a student at the Benjamin Franklin high school will perform on Words in Context. There are many factors that affect how a student will perform on the different sections of the SAT such as: motivation, confidence, adequate sleep, and the curriculum. The complete picture can be explained by a statistic called, "R-squared." If a school can account for at least 30% of R-squared when predicting scores, then the school can be confident with creating positive outcomes through changing the factors in question. (In this case, the factors are the English and science courses.) In the pie chart, English courses ("ENG") account for 29% of R-squared while science courses ("Additional Groups") account for an extra 7%. Here, we can see exactly how much English and science courses account for the Words in Context scores. Since both of these courses together account for more than 30% of R-squared, Benjamin Franklin high school's administrators were able to be confident that a change in curricula would most likely foster a change in SAT performance.
The following year, actual SAT scores were compared to the previous year's predicted scores. No changes had been made to the curriculum at that time as the administration was in process of determining what aspects needed to be modified. The predicted scores only had between 0% and 5% error when compared to the actual scores. This comparison confirmed that Predictive Analytics can predict SAT scores and provide schools with the needed data to make changes to help improve their students' SAT performance.