The size effect on structural transitions of Mn3O4 has been investigated under pressures by in situ synchrotron X-ray diffraction and Raman technique in a diamond anvil cell. Compared with bulk Mn3O4, Mn3O4 nanoparticles show an obvious elevation of phase transition pressure and different phase transformation routines with the occurrence of a new high-pressure phase at 14.5-23.5 GPa. The new phase most probably has an orthorhombic CaTi2O4-type structure, which is regarded as a metastable phase transforming to the higher pressure marokite-like structure. By the return to ambient pressure, the marokite phase is quenchable in bulk Mn3O4, whereas the coexistence of hausmannite and marokite phase is observed in the recovered Mn3O4 nanoparticles. It is proposed that the unique atomic conformation in Mn3O4 spinel structures, the cation distribution, and the higher surface energy together with the size-induced effect of nanocrystalline Mn3O4 probably play crucial roles in the high-pressure behavior of Mn3O4 nanoparticles.