In 5.10.1:

What kind of compression does Calibre use here?
That is NOT a bug. That is a USER ERROR.

What is happening is that the setting to update the metadata is set. So when the eBook is loaded again in the editor, it again modifies the metadata. The original cover is restored because it's part of the metadata that gets updates.

The solution is to turn off automatic metadata updating in the editor. This option is totally not needed because it is USER ERROR!
That didn't answer my question.
Last time I looked, calibre used mozjpeg and libvips for image compression.
Yes mozjpeg for covers since they are jpeg files.
Where is this option to compress the cover?
Edit metadata in bulk > Basic metadata (tab) ... at the bottom.
What I do after editing an eBook is go to the metadata editor and update the metadata and the cover with the edited cover and any metadata changes. If the person who put in the request did that, there would have been no problem with the cover getting replaced with the original cover.

Though one change to Calibre I would like to see is the default for updating the metadata in the editor to be turned off by default.