1. Fever is a defense mechanisms, not a sickness
Even when always associated to some kind of sickness, fever is just a symptom but not a disease, moreover, fever is the consequence of a lot of substances released to the bloodstream by the immune system cells to fight infections (or any other insult); then having fever is not bad at all because the body is taking actions to defend itself from a disease; the problem is that during that fight the baby will feel very bad, so lowering fever is necessary to make the patient feeling more comfortable.
2. Not all temperature raises are fever
Body temperature might raise a little bid above the normal in many circumstances, from just a minor flu up to exercise, but it's not possible to talk about fever unless body temperature climbs over 38.5 ºC (101.5 ºF).
Below this limit, temperature raises are called "hyperthermia" and usually are a minor concern, but when temperature exceeds that frontier it begins to be called fever and usually medical advice is required.
3. Never estimate fever using your hands
It's usual for parents and grandparent "to estimate" children's fever by touching their faces or hans, and even when this could give a gross idea about body temperature, certainly is the less accurate method because it doesn't allow to determine if core temperature raised over the 38.5 ºC (101.5 ºF) to be called fever.