Taken literally, a mug shot should be a picture of a cup with a handle. But instead it’s a picture of a criminal. The word mug, as in a drinking vessel, came from Scandinavia. In Swedish it’s mugg, and in Norwegian it’s mugge.
It was first used to mean a face in the early 1700’s. It probably came from the grotesque faces that you’ve no doubt seen adorning a mug. To mug someone originally meant to strike someone in the face. Often this was accompanied by stealing their valuables, and so it took on that meaning as well. So that’s how a mug shot became a mug shot. A mug is an unpleasant looking face, and nobody ever looks that pleasant after being picked up by the police.