Smiling in passport photos has indeed been banned in a number of countries. In the United States, the State Department guidelines state that the subject of the photo must have a "neutral" expression, and the eyes must be fully open. Passport applicants may be asked to pose for a new photo if the first one is deemed too distorted by the act of smiling.
The rules against smiling do have some exceptions and clarifications, however. A closed mouth smile may be acceptable, but a smile that exposes the teeth is not. A grimacing "smile" formed with a closed jaw may be tolerated as long as other facial features, such as the eyes and nose, are not distorted unnaturally. It is primarily the open mouth, tooth-filled smile that has become problematic.
The reason smiling in passport photos has been strongly discouraged or banned has to do with international security measures. Many modernized airports now use advanced biometric scanning devices that contain facial recognition software. Ideally, a targeted passenger's face can be scanned electronically and compared against a database of legally obtained passport photos. Distinctive biometric patterns, such as the distance between a person's eyes or the shape of his mouth, can rarely be sufficiently altered to prevent a match.
The passport photographs used for comparison should ideally be consistent and accurate, with no shadows or reflections to distort the facial measurements. Passport applicants may also be required to sweep any hair away from their faces, tilt their eyeglasses to eliminate any glare, and face completely forward with a neutral expression. Smiling can distort the subject's eyes and change the relationship between biometric points.
Although the temptation to smile for a more flattering photograph may always be there, the photographer will likely pointedly ask the subject not to do it. The passport photo may not be very flattering, but a smile may be a small sacrifice to make in exchange for increased personal safety.