Quantum size effect (QSE) is of central importance in nanoscience. For semiconductors, it is generally perceived that the QSE raises the band gap of a nanocrystal by effectively increasing the kinetic energies of electrons and holes. Using first-principles calculations and Si nanofilms as a test case, we investigated the impact of surface passivation of nanocrystals on the QSE and extended the view of the traditional QSE, as depicted in the classic effective mass model, to include the quantum boundary effect. We showed that the band gap of Si nanofilms is critically affected by the passivation species at the surface, which could result in not only the commonly observed increase in band gap but also a decrease with respect to the bulk value. The Si nanofilms can have a band gap that is virtually the same as that of bulk Si when film thickness is less than 2 nm. The new understanding of the QSE opens a new degree of freedom in engineering the electronic and optoelectronic properties of nanomaterials.