You Can Observe a Lot by Watching