Raise your hand (virtually) if your baby ~
thrashes his head back and forth in the crib,
lifts up her feet and slams them down on the mattress,
hums or moans in the crib,
pushes her face hard up against any hard surface,
scoots to the crib railing and sleeps pushed up against them in a seemingly very uncomfortable position.
This is just a partial list of activities that your baby does to self-regulate (get physically, mentally, and emotionally comfortable) so that she can fall asleep and sleep well for a cycle or two and eventually more. So let your baby slam his legs, sing, moan, scoot to the side of the crib; if you give him the space to get himself comfortable on his own then you will be creating a healthy, happy, and independent sleeper. Next time you want to readjust your baby, just imagine how mad you would be if you finally got comfortable and someone moved you to the other end of your bed.
As long as your baby is safe, then let him enjoy and express his free will in the crib.