The bundle adjustment process will perform a minimization of image residuals. Hence, it could lead to modifications on the image positions that are larger than expected, but still leading to good residuals. You should use the "Constrained" mode to set maximum modifications permitted during bundle.