It sounds as though your houseplant is in deep trouble. If only a few leaves are affected, remove them, wash the remaining foliage, and keep a close eye out for repeat problems. If most of the plant is showing these signs and symptoms, however, it’s probably not worth the effort to save it.
The rusty things that can be rubbed off are most likely scale insects. For confirmation, look for honeydew, a shiny, sticky residue these insects excrete. Honeydew accumulates on leaves and on the surface where the plant sits. If the “fuzzy stuff” is white, it may be mealybugs. They’re usually found on larger leaf-veins on leaf undersides as well as in axils, where leaf stems join larger stems.
Both scale insects and mealybugs are difficult to eliminate. You have to pick off scales, then inspect and spray repeatedly with a houseplant insecticide to catch the “crawler” stage of the insects. Wipe off mealies with a cotton swab dipped in rubbing alcohol, then monitor the plant frequently for their reappearance.
I’m not sure about the white spots you mention, but unless this plant has huge sentimental value, I’d ditch it before you inadvertently spread its many problems to other plants in your home.