The recently discovered inverse magnetic catalysis around the critical temperature indicates that some important information is missing in our current understanding of conventional chiral dynamics of QCD, which is enhanced by the magnetic field. In this work, we provide a mechanism to explain that the inverse magnetic catalysis around T-c is induced by sphalerons. At high temperatures, sphaleron transitions between distinct classical vacua cause an asymmetry in the chiral number density due to the axial anomaly of QCD. In the presence of a strong magnetic field, the chiral imbalance is enhanced and destroys the pairings between the different chiralities, which naturally lowers the critical temperature of the chiral phase transition for increasing magnetic field. The inverse magnetic catalysis at finite baryon density and the critical end point in the presence of a strong magnetic field is also explored in this work.