Answer
Why keep the colour profile and orientation tag?
Because they describe the image rather than you. Dropping them makes photos display sideways or with washed-out colours, and it protects nothing.
Updated
The orientation tag records how the camera was held. Phone sensors have a fixed physical orientation, so a photo taken in portrait is stored as a landscape image plus a tag saying "rotate this". Remove the tag and viewers stop rotating: portrait photos display on their side. Nothing about the picture changed, but it now looks broken.
The ICC colour profile describes how the numbers in the file map to actual colours. Without it, viewers guess — usually at sRGB — and a photo in a wider gamut comes back visibly flat and desaturated. For anyone who edited the image deliberately, that undoes the work.
Neither identifies you. Orientation says which way up the picture goes; a colour profile is one of a small set of industry-standard descriptions shared by millions of files. Removing them costs display quality and buys no privacy at all, which is the definition of a bad trade.
This is where indiscriminate stripping shows its cost. A tool that drops every non-pixel section without knowing what any of them are will take these too, and the result is a technically cleaner file that displays worse. JPEG has a related trap: the APP14 segment declares the colour transform, and removing it turns CMYK and YCCK JPEGs into colour-inverted messes — so it is deliberately kept here.
If you do want an absolutely bare file, that option exists and the trade is now an informed one. The engineering write-up lists exactly which sections are kept for each format and why.
More on this: Metadata remover, How it works, Supported formats.
Terms used here: Orientation tag.