Diana Franklin, Phillip Conrad
Scratch programming has risen in prominence, not only as a potential language for K-12 computer science, but also in introductory college courses. Scratch programs are 2-d animations, and can range in complexity from drawing simple shapes to story animations, music videos, musial games, or video games.
While Scratch is excellent for motivating independent learners, there has been more difficulty in its use as an educational language. The main barrier is the time necessary to assess the projects. Manual execution is necessary because of the very reason Scratch is an attractive introductory language - the projects are multimedia in nature, requiring eyes and ears to fully appreciate. In addition, the GUI is disorganized, making it difficult, even in manual execution, to find what instructions were used to implemented what features.
We have created Hairball, an automated system that can be used both by a student to point out potential errors or unsafe practices, and by a grader to assist in inspecting the implementation of Scratch programs. Because automatic analysis will not be able to determine the sensory effect, Hairball focuses instead on the implementation, including safe/robust programming practices, providing a lint-like tool for Scratch. (hence the name, since Scratch's mascot is a cat)
Hairball is the combination of a server infrastructure and individual modules. A plug-in can be created for an individual concept being assessed. For a given project, the user can choose what set of plug-ins should be used.
Our initial system has plug-ins to explore the concepts required for the projects in the Animal Tlatoque Summer Camp - detect proper initialization of Scratch state, synchronization between say and sound blocks, synchronization between broadcast and receive blocks, and proper use of timing and loops for complex animation. Our evaluation shows that Hairball is very useful in conjunction with manual analysis. Overall, Hairball was actually slightly more accurate than manual analysis at classifying errors. Specifically for broadcast/receive, Hairball's analysis correctly classified 95% of the instances, manual analysis only 80%. Overall, if Hairball was only used to identify correctly implemented instances, with manual checks on the rest of the instances, it would remove 75% of the instances for the manual analysis and assist in the rest, with a false positive rate of less than 1%.
"Assessment of Computer Science Learning in a Scratch-Based Outreach Program," Diana Franklin, Phillip Conrad, Bryce Boe, Katy Nilsen, Charlotte Hill, Michelle Len, Greg Dreschler, Gerardo Aldana, Paulo Almeida-Tanaka, Brynn Kiefer, Chelsea Laird, Felicia Lopez, Christine Pham, Jessica Suarez, Robert Waite, SIGCSE 2013.
"Hairball: Lint-inspired Static Analysis of Scratch Projects," Bryce Boe, Charlotte Hill, Michelle Len, Greg Dreschler, Diana Franklin, Phillip Conrad, SIGCSE 2013.
Faculty
Students
Charlotte Hill
Brynn Kiefer
David Borden