Normally a hamster will not need grooming as they are very good at keeping themselves clean and tidy. Short haired hamsters will only need be run through your hands to bring a shine to their coat. A small soft brush may also be used but it may take time to get your hamster used to it.
Long haired females may need a little more attention in that although the fur does not get very long it can become matted around the rear end and should be teased out using a fine comb, we use a cat's flea comb as opposed to a soft brush or toothbrush.
Long haired males may need special treatment if their hair is longer than an inch or two. By trial and error we seem to have found a way to keep the skirts (this is normally what people in the Fancy call the long hair on a hamster) on our males at least most of the time.
Firstly we groom the males about once a week. (We used to do it oftener but found a lot of the coat left on the comb). When we feed we look at the males for matting or tangles - if these are present we groom that hamster that night.
We use sawdust not shavings and this is sieved through an old colander to get the larger pieces out - these act as curlers otherwise. We try to get a sort of coarse plain sawdust which is not too fine.
Paper bedding rather than hay is better (less tangles).
Obviously the really long haired males don't have a wheel but they do have cardboard rolls suspended by wire for them to play in. The only other special treatment they have is that their porridge bowl is placed under their wooden shelf and not out in the open. Before we started doing this it was not unusual to see a male sitting with his skirt in his porridge bowl.His skirt would then have to be washed to remove the sticky porridge!
We start grooming young males before they have started growing a skirt. We usually just sit them on the hand and give a quick groom. Should they then grow a decent skirt they take grooming in their stride.
As proof of this practice ROGUE, a long haired black eyed cream, at 18 months old had a coat of 5 to 6 inches long. He died at the age of 27 months, and during the last 3 months we kept his skirt at 3 inches so that there was no need to groom him regularly in his old age.
If your hamster is purely a pet but has unmanageably long hair, it will do no harm to trim the coat, as long as not too much is taken off at one time, as this could lead to a chill. A healthy well balanced diet will normally give your hamster a nice coat.