Mobileread
Fixing python zipimport issues on Windows
#1  kovidgoyal 10-03-2020, 02:18 AM
There have been several bugs since 5 was released, of the form of imports failing. Either at calibre startup, or during metadata download or with some plugins. I have traced them all down to the zipimport module in python 3 being buggy when importing from multiple threads. On Windows all python code in calibre is placed in a single zip file for performance reasons.

So to fix it, I ended up writing my own replacement for zipimport, which doesn't use a zip file, but instead a single memory-mapped file containing all the python modules, from which individual modules can be imported using a single hash lookup via a perfect hash function generated at build time. This is pretty much the fastest possible implementation for python importing, faster than zipimport and faster than normal filesystem based importing. And it does not rely on the broken zipimport.

It was quite a journey into Python's internals to get this working, but as of now I cannot replicate any of the reported import bugs on Windows, or any other platform. This is quite a lot of new code, in a deep, dark place, so I would appreciate a bit more testing from you. If you are using CALIBRE_DEVELOP_FROM please try with it unset as well, as when it is set the importing happens from the source code rather than the new frozen importer.

The builds using the new importing framework are available as calibre 5.1.1 at:

https://download.calibre-ebook.com/betas/
Reply 

#2  chaley 10-03-2020, 04:08 AM
It works on Windows 10/64. Things I did, chosen to try to stress importing:And just for completeness:I am running in debug mode but not from source. There are no untoward messages in the debug log.

Are there other tests I can do that will stress it?
Reply 

#3  davidfor 10-03-2020, 04:16 AM
As a start, have tried the portable version. The Kobo/Kepub plugins are all loading with this version. Though I haven't loaded anything else yet.
Reply 

#4  kovidgoyal 10-03-2020, 04:39 AM
Quote chaley

Are there other tests I can do that will stress it?
Leave it running for a while, especially if you use the content server. I am a bit leery about the mmaped I/O stability.
Reply 

#5  davidfor 10-03-2020, 05:07 AM
Using a copy of my configuration directory, three plugins didn't load. They were Databazeknih, Overdrive Link and Embed Comic Metadata. When I tried to install Overdrive Link, I got this error:
Code
calibre, version 5.1.1
ERROR: Install plugin failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart calibre.
Traceback (most recent call last): File "calibre\gui2\dialogs\plugin_updater.py", line 731, in _install_clicked File "calibre\customize\ui.py", line 472, in add_plugin File "calibre\customize\ui.py", line 61, in load_plugin File "calibre\customize\zipplugin.py", line 279, in load File "importlib\__init__.py", line 127, in import_module File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "calibre\customize\zipplugin.py", line 171, in exec_module File "calibre\customize\zipplugin.py", line 167, in get_code File "calibre_plugins.overdrive_link.__init__", line 1 #!/usr/bin/env python ^
SyntaxError: invalid character in identifier
When I tried using calibre-debug, the console show a square in front of the "#" in the output. This was there for all three failing plugins. I pasted the above into the editor, and it tells me that the character is a zero-width no-break space. All three have that character at the start of the file that causes the error. That should be fixed by the plugin authors, but, it is a difference in how they are handled. But, is it actually a sign of a bug in the zipimport module?
Reply 

#6  kovidgoyal 10-03-2020, 05:45 AM
That's the UTF-8 BOM. I should have the plugin loader strip that. https://github.com/kovidgoyal/calibre/commit/0d3a80e8509e2d1b4555dc532c51d0aa2c8622b5

Quote davidfor
Using a copy of my configuration directory, three plugins didn't load. They were Databazeknih, Overdrive Link and Embed Comic Metadata. When I tried to install Overdrive Link, I got this error:
Code
calibre, version 5.1.1
ERROR: Install plugin failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart calibre.
Traceback (most recent call last): File "calibre\gui2\dialogs\plugin_updater.py", line 731, in _install_clicked File "calibre\customize\ui.py", line 472, in add_plugin File "calibre\customize\ui.py", line 61, in load_plugin File "calibre\customize\zipplugin.py", line 279, in load File "importlib\__init__.py", line 127, in import_module File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "calibre\customize\zipplugin.py", line 171, in exec_module File "calibre\customize\zipplugin.py", line 167, in get_code File "calibre_plugins.overdrive_link.__init__", line 1 #!/usr/bin/env python ^
SyntaxError: invalid character in identifier
When I tried using calibre-debug, the console show a square in front of the "#" in the output. This was there for all three failing plugins. I pasted the above into the editor, and it tells me that the character is a zero-width no-break space. All three have that character at the start of the file that causes the error. That should be fixed by the plugin authors, but, it is a difference in how they are handled. But, is it actually a sign of a bug in the zipimport module?
Reply 

#7  kovidgoyal 10-03-2020, 05:48 AM
In fact, this is better: https://github.com/kovidgoyal/calibre/commit/0e25387a9fea9cd861c6e8d9cae7fe6ea3bf7d42
Reply 

#8  kovidgoyal 10-03-2020, 07:34 AM
Released 5.1.2 with fixes for the issue with UTF-8 BOM and some plugins reported by davidfor and another issue I discovered where using CALIBRE_DEVELOP_FROM was not working with the content server
Reply 

#9  jackie_w 10-03-2020, 04:07 PM
Quote kovidgoyal
Released 5.1.2 with fixes for the issue with UTF-8 BOM and some plugins reported by davidfor and another issue I discovered where using CALIBRE_DEVELOP_FROM was not working with the content server
I'm a little confused. When you say "Released 5.1.2", do you mean "Released a beta 5.1.2" or "Released a full 5.1.2"?

If it's meant to be the latter then my currently installed calibre v5.0.1 64-bit is only offering "Update found:5.1.0" in the statusbar.
Reply 

#10  Terisa de morgan 10-03-2020, 04:19 PM
Quote jackie_w
I'm a little confused. When you say "Released 5.1.2", do you mean "Released a beta 5.1.2" or "Released a full 5.1.2"?

If it's meant to be the latter then my currently installed calibre v5.0.1 64-bit is only offering "Update found:5.1.0" in the statusbar.
Deltas never appear in the statusbar.
Reply 

  Next »  Last »  (1/2)
Today's Posts | Search this Thread | Login | Register