Abstract— Predicting the trajectory of flying objects with spinis a challenge but an essential task in many fields, especially inmilitary and sports. Robots playing ping-pong is a very goodplatform to validate the trajectory prediction method. Variousvision systems have been proposed, but only position informationwas used in most cases, which limits their capability to predict thetrajectory of the spinning ball. Based on the fact that a spinningball’s motion can be separated into translation movement andspinning with respect to the ball’s center, this paper proposes anovel vision system that can provide both the position and thespin information of a flying ball in a real-time mode with highaccuracy. With a frame difference-based recognition method,the natural brand of a ball can be recognized under normalillumination conditions. Then the 3-D pose of the ball can berestored in ball coordinates. With the observation and analysisthat the axis and angular speed of spin do not change duringflying, the spin state can be estimated using a weighted-randomsample consensus-based plane fitting method. Combining bothposition and spin information in a force-based dynamic model,accurate trajectory prediction can be achieved via an extendedKalman filter. Experimental results show the effectiveness andprecision of the proposed method.