What it is.

LibrarianSync is a Kindle collection builder, originally meant as a complement to Librarian, but which can be used completely independently.

It has been tested on a Kindle PW2 (EU, Wi-Fi) FW 5.4.5.1, and on a Touch FW 5.3. It probably works on other K5 devices. It apparently works on the Voyage as well.

It can use as a source:

• a json file in native format (such as generated by Librarian when syncing, see below)
• the json file automatically generated by the Calibre Kindle Collections plugin)
• the folder structure inside the documents/ folder.

Using the json file, the entries can either remplace all existing collections or update existing collections. Using folder structure, all previously existing collections are dropped.

The native json format looks as follows:

As shown in the last line, you can add (python) regular expressions instead of filenames (with a re: prefix); all matches will be added to the associated collection.

What is needed.

LibrarianSync comes as a KUAL menu entry. As such, it requires: a jailbroken PW2, MKK, KUAL, and python (snapshot > 0.10N-r10867).

For more information, see the README file on github.

Where it is.

The code resides on github, but since I do not use this personally anymore, the better place to look for changes is NiLuJe's fork (thanks!).
See attachments for an up to date Kindle package.

Note: the code has moved to a separate repository as LibrarianSync has become something useful in its own right. Please update your git remotes to the new repository.

Where it comes from.

LibrarianSync is inspired by this thread.

Thanks to NiLuJe for all the contributions and continuous support, and kaznelson for his initial courageous sqlite investigations.

Feedback is very much welcome.

LibrarianSync Changelog

v20201109 - Same as below . -- NiLuJe
v20190103 - Refreshed package from latest sources, so as not to leave four year old packages around... -- NiLuJe
v1.10 - fixed when getting a null cdetype, added Voyage support.
v1.9 - fixed support of non-cloud aware devices
v1.8 - fixed collection appearing twice in FW 5.6
v1.7 - added support for regular expressions in librarian json when building collections with LibrarianSync.
v1.6 - fixed exporting collections, added downloading ebooks hosted over http by librarian.
v1.5 - package available for all k5 models.
v1.4 - updating collections is now faster.
v1.3 - calibre plugin json support & ability to export current collections (librarian and calibre plugin versions). While running the script, current operation now logged and displayed.
v1.2 - experimental calibre plugin json support & ability to export current collections to extensions/exported_collections.json.
v1.1 - experimental calibre plugin json support
v1.0 - original version
Thanks for this.

It comes at a good time, since the interactive Collection Manager is not (yet) ported to all of the PW2 firmware / hardware combinations.

Note:
I think this will also work on other models of Kindles - that DB probably doesn't (didn't) change that often.
Yes, I guess it should work with other models, after all this thread was originally about the Kindle Touch. I only have a pw2 (EU), so I can't verify that. If anyone is willing to try it on a different model, feedback would be welcome.

Also, I updated the script and KUAL menu to allow for incremental collection updates, that do not rebuild all collections but merely add the json entries into existing or new collections.
@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.

EDIT: Done .
The Amazon Lua install includes the sqlite and json add-on libraries as I recall (at least on the PW1, I think on the KT also, PW2 is unknown).

If someone wanted to get ambitious, they could re-write this in Lua and avoid any add-in support packages.

The same code (and libraries) shipped by Amazon can also be run by the ARMhf Luajit if the user had those installed.

- - - -

Just a thought -
Not something I have time to do.
And twobob is still off, dealing with "life".
Great! I hadn't seen your update. I updated my script to use the new modules.
I didn't know the requests module, it's pretty nifty.
@barsanuphe: That was fast ^^.

Yep, requests is great . Did the switch affect performance in any measurable way?
barsanuphe
Whether it's possible to make "lite" version?
To create collections on folders names (without metainfo), and worked ONLY on Kindle? (without computer)
@NiLuJe:

I have not looked at the performance aspect yet, and am testing with only a few ebooks on my Kindle. I expect it to be faster though, but couldn't tell you by how much. Also, all operations are now sent in a single POST request, which should help too.

@kaznelson:

I updated the script and added a new entry in the KUAL menu to that effect. You should now be able to create collections from your folder structure, without json file. The collection name is the path of each ebook (without filename), relative to /mnt/us/documents.

ie if you have "/mnt/us/documents/sf/franck herbert/dune/dune.epub", the collection name will be "sf/franck herbert/dune".

I tried earlier to make nested collections, but I found only top collections are displayed on my Kindle. A subcollection can be added to a collection but does not show up on the interface. Reading through the Collection Manager thread, it seems it's a firmware issue...
@barsanuphe: Yup, nesting was broken in FW 5.4.2, and it hasn't resurfaced since.

I was thinking of using this with the Calibre Kindle Collections plugin, most likely by making it understand the json schema used by the plugin, since the plugin already takes care of a number of sanity checks (namely the uuid/ASIN handling of each book).

FWIW, the schema is as follows:

So, as an example with a few different types of hash, that would look something like:

(The indentation is all mine, for formatting purposes only).