Mobileread
Plugins cross-influencing??
#1  jindroush 09-26-2020, 02:26 PM
Hi,
because of change to 5.0.1, one of the metadata plugins I use (Databazeknih) stopped working. I made some fixes, it works.

BUT: I use two metadata sources - one for Databazeknih with priority 1, the other one is built-in Google Images with priority 2.

When there is only Google Images enabled, it works.
As soon as I enable Databazeknih, Databazeknih works and Google Images stops working (in log are missing imports, for example: ImportError: cannot import name 'Browser' from 'calibre.utils.browser')

How could these two influence each other? I know the solution (Add some import statements to Databazeknih __init__, but I can't understand why this happens, and I still think this is somewhat rotten behavior and wrong thing to fix).
Reply 

#2  kovidgoyal 09-26-2020, 10:04 PM
All plugins are loaded into the same process, so they can in theory do anything to each other. calibre does not treat plugins as potentially hostile code, it assumes plugin developers "do the right thing".
Reply 

#3  jindroush 09-27-2020, 05:20 AM
I'm not criticizing this, as I also don't think there is a real need for separation.

I just can't understand how can Google Images work on its own, but when I load some code BEFORE it, it stops working because some imports are suddenly missing.

I attached the plugin. To test it, just configure DatabazeKnih with priority 1, and GoogleImages with priority 2 and try to download some covers.

With "FIXED" GoogleImages don't work, with FIXED2 it works. But I still don't know where the actual problem lies, as I fixed only the symptoms, not the root of the problem.
[zip] databazeknih1_4_16_FIXED.zip (7.2 KB, 24 views)
[zip] databazeknih1_4_16_FIXED2.zip (7.2 KB, 23 views)
Reply 

#4  kovidgoyal 09-27-2020, 05:24 AM
Are you on windows? If so, then the problem will be because python's zipimport is not threadsafe, and metadata download happens in multiple threads. So unless importing is done in __init__ or at module level things will break. See for example: I just finished dealing with this problem in another part of the code: https://github.com/kovidgoyal/calibre/commit/9eb68319cdbf7a54dcdc0267bb632fc76534bd53
Reply 

#5  kovidgoyal 09-27-2020, 05:30 AM
Apparently its even worse. importing in python 3 is generally not thread safe, on all OSes and even when not using zipimport. Sigh.

https://bugs.python.org/issue38884
Reply 

#6  jindroush 09-27-2020, 06:21 AM
Yes, forgot to state - running latest 5.0.1 64bit on Windows 10 x64.

I downloaded latest tarball, looked up the missing import - importing is done deep there on line 311 of global __init__.py, in function browser
from calibre.utils.browser import Browser
ImportError: cannot import name 'Browser' from 'calibre.utils.browser' (C:\Program Files\Calibre2\\app\pylib.zip\calibre\utils\browse r.pyc)
so this fails when there is DatabaseKnih plugin loaded. If it's because loading order in multi-threaded environment, I can't think of different solution than to put all import statements somewhere in global init.

BTW: Is it possible for the debugging purposes to unpack pylib.zip to see if the problem vanishes?
Reply 

#7  jindroush 09-27-2020, 06:39 AM
Regarding my last question: I unpacked pylib.zip to bin directory, and deleted pylib.zip
Started to work.
So your theory about zipimport seems to be valid.
I can see two solutions:
1) Drop pylib.zip idea and simply unpack gazillions of small files to bin
2) Somehow move all imports to one central place. This would be helluva work and will be prone to errors.
3) [Some other solution I don't understand/know about because I'm not the python guy]
Reply 

#8  kovidgoyal 09-27-2020, 06:48 AM
The files are in a zip file for performance reasons. Python is much faster at importing from a single file on windows.

The solution is pretty simple, if the issue is only with this plugin, simply move the import causing the problem to the top level/__init__

If it turns out more metadata plugins are impacted by this, then I will have to re-architext calibre to run the metadata plugins in their own processes, shouldnt be very hard.
Reply 

#9  jindroush 09-27-2020, 07:18 AM
The plugin does this:
from calibre import browser
that function internally (I assume much later) does this:
from calibre.utils.browser import Browser
This creates conflict with Google Images - preventing it from loading Browser.
When I put both of these in plugin's __init__, it works. But how do I know which such functions I should put in _my_ init, not to break other plugins?
Reply 

#10  kovidgoyal 09-27-2020, 07:19 AM
There arent that many, most plugins will be importing the same things. So for the moment, just make the change, if it causes any more issues I will revisit running te plugins per process
Reply 

  Next »  Last »  (1/2)
Today's Posts | Search this Thread | Login | Register