To predict the effects induced by stereo image content on visual health, a new objective Visual Comfort Assessment (VCA) method of stereo image is proposed based on scene modes. Natural scene is abstracted as multiple scene modes according to two position states of foreground object and background region. One is the convex-concave to screen, and the other is the whether locate on zone of comfortable viewing. In the process of mode selection, disparity map is utilized to segment scene into foreground object and background region adaptively. Then, the scene’s mode can be determined by disparity angle features of both foreground object and background region. In the modeling stage, disparity angle features of foreground object and background region, width angle and sinuosity features of foreground object are utilized to build objective VCA models in various scene modes. The experimental results tested on IVY database show that high consistency exists between the proposed model and subjective perception that Pearson linear correlation coefficient is higher than 0.91, Spearman rank-order correlation coefficient is higher than 0.90, Kendall rank-order correlation coefficient is higher than 0.74, Mean Absolute Error (MAE) is lower than 0.24 and Root Mean Squared Error (RMSE) is lower than 0.32. Compared with other existing methods, the proposed model has the better assessment performance and is much closer to the subjective assessment scores.