Can you solve my favorite interview question? (math + cs)