Collaborative Research: Seeing Science: Using Computer Vision to Explore the Scientific Principles Behind Everyday Objects

Understanding science is critical for preparing students to make sense of the world around them, make informed decisions, and participate in civic society and in the workforce. However, for many youth, science is a mysterious body of knowledge that feels disconnected from their lives. This project aims to bring science into middle school students’ homes, allowing them to see the science behind everyday objects and transforming lived environments into engaging learning spaces. Students will work on inquiry-based learning units on mobile phones that explore STEM phenomena topics like diffusion, electricity, and simple machines that are present in their kitchens, bedrooms, and local parks. They will be able to take photos and videos of their home and neighborhood, and computer vision algorithms will augment these images with diagrams, models, and simulations that illustrate the principles and mechanisms that explain the STEM phenomena. These overlays will allow students to observe, experiment with, and make predictions for phenomena such as tea diffusing in hot water or heat traveling through walls. The project will capitalize on existing technological devices, such as camera phones, to create “lenses” which enable students to see the science that is all around them. By committing to such low-cost solutions, the project aims to make science education accessible to more students, and enable at-home learning while avoiding undue pressures on family resources. Operating in diverse, low-resource environments motivates fundamental advances in computer vision: First, algorithms must automatically build up a 3D and temporal representation of a scene of a given physical phenomenon. Second, the system must expose hooks for educators to decide which graphics should be overlaid at which time and in which place atop this scene. Further, the project will engage in human-centered design to bring cutting-edge technologies to youth in ways that are accessible, easy to use, and achieve educational goals. Investigators will conduct extensive interviews with parents, students, and teachers about the aspects of students’ out-of-school lives that they would be willing to share with researchers, peers, and teachers. These data will enable the team to realize the benefits of equitable science education that builds on students’ lives and cultures. This research will help foster the development of a more agentic, inclusive way of engaging in science inquiry at home, encouraging students to have a personal connection with science from a young age. This is particularly important for students most at risk to perceive science as disconnected from their lives, and whom can benefit most from seeing science at work in their lives and community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.