Recent observations by the Fermi-LAT showed that there are delayed arrivals of GeV photons relative to the onset of MeV photons in some gamma-ray bursts (GRBs). In order to avoid a large optical depth, the minimal value of the Lorentz factor has been estimated to be higher than 1000 in some of the brightest bursts. In this paper, we present a detailed calculation of the time delay between the MeV and GeV photons in the framework of the magnetic-dominated jet model. We find that the time delay strongly depends on the saturated bulk Lorentz factor of the jet. Inspired by this fact, we use this model to calculate the Lorentz factors of the four brightest Fermi bursts. The results indicate that the Lorentz factors are much smaller than those obtained from the "single-zone" scenario. The short burst GRB 090510 has a minimal Lorentz factor of 385, while the three long bursts, GRB 080916c, GRB 090902b, and GRB 090926, have almost the same Lorentz factors with an average value near 260. Another interesting result is that, for long bursts, GeV photons are emitted after the bulk Lorentz factor saturates. For the short GRB, however, MeV and GeV photons are emitted at the same phase, i.e., either in the expansion phase or in the coasting phase.