We have shown that the Sun's shadow by high-energy cosmic rays moves year by year and its, behavior is correlated with a time variation of the large-scale structure of the solar and interplanetary magnetic fields. The solar activity was near minimum in the period from 1996 through 1997. Using the data obtained with the Tibet air shower array, we examined the shadowing of cosmic rays by the Sun in this quiet phase of the solar cycle and found that the Sun's shadow was just in the apparent direction of the Sun, though it was observed at the position considerably away from the Sun to the southwest in the period between 1990 and 1993. It is known that the magnetic pole of the equivalent solar dipole was reversed during the previous active phase, and near solar minimum the dipole was aligned with the rotating axis, preserving its N-pole on the north pole side of the Sun. This causes the solar magnetic field to shift the Sun's shadow to the east. Thus, the observed results suggest that the shift of the Sun's shadow due to the solar magnetic field was pushed back by the effect of the geomagnetic field, since the geomagnetic field always makes the shadow shift to the west. We discuss the Sun's shadow observed during the period near solar minimum in 1996-1997 and compare it with the simulation results.