Those who remain stay secluded in their rooms as authorities paste posters on dwellings around the complex.
The Tibetan spiritual leader's visit is religious and not political in nature, government officials say.
The unidentified protesters call out in support of exiled spiritual leader the Dalai Lama.
The move appears aimed at preventing attendance at the Dalai Lama's Kalachakra teachings next year.
Detained in 2012 for opposing Chinese rule in Tibetan areas, Lobsang Sangye was tortured in custody, a source says.
Authorities ban the prayer and teaching event as they work to reduce the Tibetan Buddhist study center's size, sources say.
Some will be held for up to six months until the classes end, sources say.
Many are forced to leave without being allowed to gather their belongings, a local source says.