I stumbled upon problem with running calibre server after I attempted to add some more libraries.
● calibre.service - Calibre Service Loaded: loaded (/lib/systemd/system/calibre.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2018-09-10 14:22:24 CEST; 22min ago Main PID: 8673 (code=exited, status=1/FAILURE)
Sep 10 14:22:23 vps systemd[1]: Started Calibre Service.
Sep 10 14:22:24 vps calibre-server[8673]: There is no calibre library at: /home/username/Library
Sep 10 14:22:24 vps systemd[1]: calibre.service: Main process exited, code=exited, status=1/FAILURE
Sep 10 14:22:24 vps systemd[1]: calibre.service: Unit entered failed state.
Sep 10 14:22:24 vps systemd[1]: calibre.service: Failed with result 'exit-code'.
Sep 10 14:25:20 vps systemd[1]: Stopped Calibre Service.
Sep 10 14:28:27 vps systemd[1]: [/lib/systemd/system/calibre.service:11] Missing '='.
It looks like calibre library called "Library" is missing. Strange as I didn't add library called like that or had it setup earlier either.
I tried to resolve the issue few hours later since my last post and server seems to be working fine.
Still I am interested why it didn't work in the first place.