Not sure if I understand you quite right.
The downshifts are not that harsh, but noticeable. Upshifts are worse when the car is cold (first 3 gears at least).
However, yes - the car does slow down when releasing the throttle at low speeds in the city, especially when the car is not in operating temperature.
But I think this is normal due to the recuperation to charge the battery.
Is your problem the shifting itself or is the gear engaged and then you slow down?