Synchronisation kobo

Krakinou · October 26, 2022, 8:44am

For the record, I plan in a next release to

enhance the authorization mechanism so that the api_endpoint is available to visitors without exposing the whole app
integrate in the package the kepubify package so that Kobo integration works better
But I’m a little short on time right now, so I don’t have any release date in mind

wanabo · October 27, 2022, 4:04pm

Thanks for all. Everythings seems to be ok in my settings. But it does not work.
I tried 443 or 80 “portexterne du serveur” but alays the same : synchro failed.

I’ll wait for the next release if it solves my problem.

Mamie · November 4, 2022, 8:56pm

Thanks for this topic, I did not know it was possible to do this !

Krakinou · November 11, 2022, 11:19pm

Hi,
I’m currently testing the new version of the package with kepubify and dedicated authorization.
See documentation here

You can test yourself by using

sudo yunohost app install https://github.com/YunoHost-Apps/calibreweb_ynh/tree/testing --debug
or
sudo yunohost app upgrade calibreweb -u https://github.com/YunoHost-Apps/calibreweb_ynh/tree/testing

I did not notice any issues so far, but any feedback would be welcome before I push this release to master.

I noticed that once the epub are converted to kepub, the whole sync seems to work a lot better : all the books are synched at once, the reading states is updated correctly (no more back to the beginning of the chapter), the only drawback is that you have to wait for the conversion to take place when you first create/display the token.

wanabo · November 12, 2022, 9:04am

Thanks @Krakinou for all this hardwork.

The upgrade went well, no problem detected. I have juste a few books in calibre-web. The conversion was instantly done.

What I did :
I change the authtification token and replace the link in my kobo. I restart the kobo. I always have the synchonisation failure.

Krakinou · November 12, 2022, 12:44pm

Oups!
The version I pushed in testing is not the one I use
Sorry for the mixup, I’ll try to fix things, and if it still doesnot work for you, then we will dive into your problem.

Krakinou · November 12, 2022, 4:17pm

So, you can now update again with

sudo yunohost app upgrade calibreweb -u https://github.com/YunoHost-Apps/calibreweb_ynh/tree/testing

It should bring the last version with proper permission.
Then, before trying to sync anything, firstly reset the Calibreweb database by clicking on “Force full Kobo sync”. This will reset the sync database on calibre side.
NB : do not try to access directly to the endpoint via a web browser, as calibre-web will consider this as a sync, and your books will not sync correctly afterward unless you reset it with this option

You can then try to sync your kobo again.

Now, if it’s still not working can you make the following checks :

check that port 80 and 443 are correctly redirected to your server on your internet box
check that permission “Kobo sync” is assigned to visitor group in Yunohost
check that “Activate Kobo synchronisation” is set in the application
check that the external port is set as 443 in the application
check that the api_endpoint is correctly set in the kobo by copying exactly the line provided by calibreweb without adding anything

If this is still not working, can you give me the following info:

How is your system/network set up? (VPS, raspberry, other, any specific network stuff, etc.)
Do you use the sync with the option “ync only books in selected shelves with Kobo” set in the user settings ?
What is the exact error message that you get on the Kobo (if any)
can you give me the output of following command while trying to sync:

tail -f /var/log/nginx/yourdomain-access.log
tail -f /var/log/nginx/yourdomain-error.log
tail -f /var/log/calibreweb/calibreweb.log
tail -f /var/log/calibreweb/calibreweb-access.log

Can you give me the content of /etc/ssowat/conf.json regarding Calibreweb (calibreweb.main and calibreweb.Kobo sync)

Krakinou · November 12, 2022, 4:21pm

Also, on a side note, I think that the limitation I wrote in the doc “Kobo Sync doesn’t work when Calibreweb is installed on a subdomain” is not true anymore with this new testing version.
Any chance someone could test? I don’t have enough kobos and servers at home to test this without breaking all the books I’m reading

wanabo · November 12, 2022, 6:12pm

Okay thanks again for your work.

This is not working even with this last version.
I did the checks you asked for. I think that everythings is good for me.

How is your system/network set up? (VPS, raspberry, other, any specific network stuff, etc.)

I use a VM (proxmox). I have several VM (including yunohost). I use a reverse proxy with a synologynas. Every requests are send to synology nas (80, 443).

Do you use the sync with the option “ync only books in selected shelves with Kobo” set in the user settings ?

Yes the option is checked

After a few seconds : “Echec de la synchronisation. Merci de réessayer”

192.168.0.2 - - [12/Nov/2022:19:04:16 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/initialization HTTP/1.1" 200 84                98 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Tou                ch 0376/4.34.20097)"
192.168.0.2 - - [12/Nov/2022:19:04:16 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/profile HTTP/1.1" 307 305                 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch                 0376/4.34.20097)"
192.168.0.2 - thiboux [12/Nov/2022:19:04:16 +0100] "PROPFIND /nextcloud/remote.php/dav/files/thiboux/Documents HTTP/1.1" 207 401 "-                " "Mozilla/5.0 (Windows) mirall/3.6.1stable-Win64 (build 20221018) (Nextcloud, windows-10.0.22000 ClientArchitecture: x86_64 OsArch                itecture: x86_64)"
192.168.0.2 - thiboux [12/Nov/2022:19:04:16 +0100] "PROPFIND /nextcloud/remote.php/dav/files/thiboux/Photos HTTP/1.1" 207 398 "-" "                Mozilla/5.0 (Windows) mirall/3.6.1stable-Win64 (build 20221018) (Nextcloud, windows-10.0.22000 ClientArchitecture: x86_64 OsArchite                cture: x86_64)"
192.168.0.2 - - [12/Nov/2022:19:04:17 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/loyalty/benefits HTTP/1.1"                 307 323 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Ko                bo Touch 0376/4.34.20097)"
192.168.0.2 - thiboux [12/Nov/2022:19:04:17 +0100] "PROPFIND /nextcloud/remote.php/dav/files/thiboux/Photos%20t%C3%A9l%C3%A9phones                 HTTP/1.1" 207 421 "-" "Mozilla/5.0 (Windows) mirall/3.6.1stable-Win64 (build 20221018) (Nextcloud, windows-10.0.22000 ClientArchite                cture: x86_64 OsArchitecture: x86_64)"
192.168.0.2 - - [12/Nov/2022:19:04:17 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/deals HTTP/1.1" 307 291 "-" "Mo                zilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch 0376/4.                34.20097)"
192.168.0.2 - - [12/Nov/2022:19:04:17 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/assets?DiffRequests=[%7BKey:EPD                -KoboPlus-ReadOnly-NeverSubscribed,ETag:W/NjM3NzI1ODc2ODkwMDAwMDAwLTQ3Njc%3D%7D] HTTP/1.1" 404 1520 "-" "Mozilla/5.0 (Linux; U; And                roid 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch 0376/4.34.20097)"
192.168.0.2 - - [12/Nov/2022:19:04:17 +0100] "POST /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/analytics/gettests HTTP/1.1" 2                00 80 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo                 Touch 0376/4.34.20097)"
192.168.0.2 - - [12/Nov/2022:19:04:18 +0100] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/library/sync?Filter=ALL&Downloa                dUrlFilter=Generic,Android&PrioritizeRecentReads=true HTTP/1.1" 200 1795 "-" "Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebK                it/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch 0376/4.34.20097)"
192.168.0.2 - wanabo [12/Nov/2022:19:04:37 +0100] "GET /nextcloud/ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows) mirall/3.6.1stable-Win64 (build 20221018) (Nextcloud, windows-10.0.22621 ClientArchitecture: x86_64 OsArchitecture: x86_64)"

Nothing usefull here

[2022-11-10 04:10:00,002]  INFO {apscheduler.executors.default:123} Running job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-11 04:10:00 CET)" (scheduled at 2022-11-10 04:10:00+01:00)
[2022-11-10 04:10:00,006]  INFO {apscheduler.executors.default:144} Job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-11 04:10:00 CET)" executed successfully
[2022-11-11 04:10:00,004]  INFO {apscheduler.executors.default:123} Running job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-12 04:10:00 CET)" (scheduled at 2022-11-11 04:10:00+01:00)
[2022-11-11 04:10:00,004]  INFO {apscheduler.executors.default:144} Job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-12 04:10:00 CET)" executed successfully
[2022-11-12 04:10:00,004]  INFO {apscheduler.executors.default:123} Running job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-13 04:10:00 CET)" (scheduled at 2022-11-12 04:10:00+01:00)
[2022-11-12 04:10:00,005]  INFO {apscheduler.executors.default:144} Job "end scheduled task (trigger: cron[hour='4', minute='10'], next run at: 2022-11-13 04:10:00 CET)" executed successfully
[2022-11-12 09:47:16,631]  INFO {cps.server:275} webserver stop (restart=False)
[2022-11-12 09:47:16,637]  INFO {cps.server:275} webserver stop (restart=False)
[2022-11-12 09:47:22,321]  INFO {apscheduler.scheduler:202} Scheduler has been shut down

::1 - - [2022-11-12 19:12:05] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/initialization HTTP/1.0" 200 9191 0.171141
::ffff:127.0.0.1 - - [2022-11-12 19:12:05] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/profile HTTP/1.0" 307 1034 0.005685
::1 - - [2022-11-12 19:12:05] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/loyalty/benefits HTTP/1.0" 307 1061 0.005396
::ffff:127.0.0.1 - - [2022-11-12 19:12:05] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/deals HTTP/1.0" 307 1013 0.004941
::1 - - [2022-11-12 19:12:06] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/assets?DiffRequests=[%7BKey:EPD-KoboPlus-ReadOnly-NeverSubscribed,ETag:W/NjM3NzI1ODc2ODkwMDAwMDAwLTQ3Njc%3D%7D] HTTP/1.0" 404 1921 0.004777
::ffff:127.0.0.1 - - [2022-11-12 19:12:06] "POST /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/analytics/gettests HTTP/1.0" 200 1985 0.204421
::1 - - [2022-11-12 19:12:06] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/library/sync?Filter=ALL&DownloadUrlFilter=Generic,Android&PrioritizeRecentReads=true HTTP/1.0" 200 6132 0.262544

Krakinou · November 12, 2022, 7:34pm

Hmmm
So it seems that it connects at least a little…
There is only one 404 but I don’t know to what it’s related to, I’ll check tomorrow from my test system.
Can you try changing the “proxy request to kobo store”? I noticed that I had this error at the beginning and that fiddling with the setting made it go away.

Also, just to be in the safe side: you do have books on a shelf that is activated for kobo sync?

wanabo · November 12, 2022, 8:01pm

Can you be more precise ? Where is this option ? Change to what ?

Yes, 3 books.

Krakinou · November 12, 2022, 10:48pm

This one:

Krakinou · November 12, 2022, 11:00pm

Also, can you set the log file on debug (somewhere in the main configuration of the app), retry to sync and check again /var/log/calibreweb/calibreweb.log

wanabo · November 13, 2022, 9:22am

::ffff:127.0.0.1 - - [2022-11-13 10:19:49] "POST /calibre/admin/ajaxconfig HTTP/1.0" 200 964 0.030954
::ffff:127.0.0.1 - - [2022-11-13 10:20:01] "GET /calibre/admin/alive HTTP/1.0" 200 723 0.025946
::ffff:127.0.0.1 - - [2022-11-13 10:21:04] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/initialization HTTP/1.0" 200 9191 0.020380
::1 - - [2022-11-13 10:21:04] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/profile HTTP/1.0" 200 670 0.006149
::ffff:127.0.0.1 - - [2022-11-13 10:21:04] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/loyalty/benefits HTTP/1.0" 200 684 0.004953
::1 - - [2022-11-13 10:21:04] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/deals HTTP/1.0" 200 670 0.004447
::ffff:127.0.0.1 - - [2022-11-13 10:21:04] "POST /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/analytics/gettests HTTP/1.0" 200 749 0.007153
::1 - - [2022-11-13 10:21:05] "GET /calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/library/sync?Filter=ALL&DownloadUrlFilter=Generic,Android&PrioritizeRecentReads=true HTTP/1.0" 200 6065 0.095888

Krakinou · November 13, 2022, 9:16pm

/var/log/calibreweb/calibreweb.log, not /var/log/calibreweb/calibreweb-access.log

But I notice that now you have 200 return code instead of 307 : is it somewhat better?

wanabo · November 14, 2022, 11:01am

Oups sorry !!

root@yunohost:~# tail -f /var/log/calibreweb/calibreweb.log
[2022-11-14 11:58:38,337] DEBUG {cps.kobo:277} Remaining books to Sync: 0
[2022-11-14 11:58:44,199] DEBUG {cps.ub:83} Found stored session: b45bfacba3759024c065aa9e90d24d204862279164495927ba7660e682545087ae3ba95255e8da4307426527548b81cd8ec82ee7722886783465a5ae1d60780f
[2022-11-14 11:58:44,200]  INFO {cps.kobo:1052} Init
[2022-11-14 11:58:44,239] DEBUG {cps.kobo:973} Unimplemented User Request received: https://yunohost.stemaga.ovh/calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/user/profile
[2022-11-14 11:58:45,098] DEBUG {cps.kobo:1013} Unimplemented Products Request received: https://yunohost.stemaga.ovh/calibre/kobo/b1ec4bcb0a3ea176ac959901937a5c9e/v1/deals
[2022-11-14 11:58:45,277] DEBUG {cps.ub:83} Found stored session: b45bfacba3759024c065aa9e90d24d204862279164495927ba7660e682545087ae3ba95255e8da4307426527548b81cd8ec82ee7722886783465a5ae1d60780f
[2022-11-14 11:58:45,277]  INFO {cps.kobo:140} Kobo library sync request received.
[2022-11-14 11:58:45,277] DEBUG {cps.kobo:141} SyncToken: 0001-01-01 00:00:00,0001-01-01 00:00:00,0001-01-01 00:00:00,0001-01-01 00:00:00,0001-01-01 00:00:00,eyJ0eXAiOjEsInZlciI6bnVsbCwicHR5cCI6IlN5bmNUb2tlbiJ9.eyJJbnRlcm5hbFN5bmNUb2tlbiI6ImV5SjBlWEFpT2pFc0luWmxjaUk2SW5ZeUlpd2ljSFI1Y0NJNklsTjVibU5VYjJ0bGJpSjkuZXlKVGRXSnpZM0pwY0hScGIyNUZiblJwZEd4bGJXVnVkSE1pT25zaVNYTkpibWwwYVdGc0lqcHVkV3hzTENKSFpXNWxjbUYwYVc5dVZHbHRaU0k2Ym5Wc2JDd2lWR2x0WlhOMFlXMXdJam9pTVRrd01DMHdNUzB3TVZRd01Eb3dNRG93TUZvaUxDSkRhR1ZqYTFOMWJTSTZiblZzYkN3aVNXUWlPaUl3TURBd01EQXdNQzB3TURBd0xUQXdNREF0TURBd01DMHdNREF3TURBd01EQXdNREFpZlN3aVJuVjBkWEpsVTNWaWMyTnlhWEIwYVc5dVJXNTBhWFJzWlcxbGJuUnpJanB1ZFd4c0xDSkZiblJwZEd4bGJXVnVkSE1pT25zaVNYTkpibWwwYVdGc0lqcHVkV3hzTENKSFpXNWxjbUYwYVc5dVZHbHRaU0k2Ym5Wc2JDd2lWR2x0WlhOMFlXMXdJam9pTWpBeU1TMHdOUzB4TlZReE56bzBNRG96TmxvaUxDSkRhR1ZqYTFOMWJTSTZJbnRjSWxOcmFYQkRjbVZoZEdWa1hDSTZabUZzYzJVc1hDSkRjbVZoZEdWa1EyOTFiblJjSWpveExGd2lRM0psWVhSbFpGd2lPbHRjSW1NeU5XUTBNak13TFRFeFpqSXROR1psTmkwNU1HWXlMVEU0TXpjM1kyTmpaVEV5WkZ3aVhTeGNJbE5yYVhCTmIyUnBabWxsWkZ3aU9tWmhiSE5sTEZ3aVRXOWthV1pwWldSY0lqcGJYWDBpTENKSlpDSTZJbU15TldRME1qTXdMVEV4WmpJdE5HWmxOaTA1TUdZeUxURTRNemMzWTJOalpURXlaQ0o5TENKRVpXeGxkR1ZrUlc1MGFYUnNaVzFsYm5SeklqcDdJa2x6U1c1cGRHbGhiQ0k2Ym5Wc2JDd2lSMlZ1WlhKaGRHbHZibFJwYldVaU9tNTFiR3dzSWxScGJXVnpkR0Z0Y0NJNklqSXdNakV0TURRdE1EbFVNRGs2TURrNk5EbGFJaXdpUTJobFkydFRkVzBpT201MWJHd3NJa2xrSWpvaU1EQXdNREF3TURBdE1EQXdNQzB3TURBd0xUQXdNREF0TURBd01EQXdNREF3TURBd0luMHNJbEpsWVdScGJtZFRkR0YwWlhNaU9uc2lTWE5KYm1sMGFXRnNJanB1ZFd4c0xDSkhaVzVsY21GMGFXOXVWR2x0WlNJNmJuVnNiQ3dpVkdsdFpYTjBZVzF3SWpvaU1qQXlNUzB3TlMweU1sUXlNVG93TlRvd05Wb2lMQ0pEYUdWamExTjFiU0k2Ym5Wc2JDd2lTV1FpT2lJMVpERTJPR0UyWWkwd01XRmpMVFJoTlRNdFlXSXpZUzAzT0ROak16YzRNRE13WVRNaWZTd2lWR0ZuY3lJNmV5SkpjMGx1YVhScFlXd2lPbTUxYkd3c0lrZGxibVZ5WVhScGIyNVVhVzFsSWpwdWRXeHNMQ0pVYVcxbGMzUmhiWEFpT2lJeU1ESXhMVEV4TFRJd1ZESXdPalV4T2pVNVdpSXNJa05vWldOclUzVnRJanB1ZFd4c0xDSkpaQ0k2SWpRNFpqTm1OalkxTFdZd1lXRXRORFF3WmkwNVpEZzRMV1V3WldRM016WTJZamsxT0NKOUxDSkVaV3hsZEdWa1ZHRm5jeUk2ZXlKSmMwbHVhWFJwWVd3aU9tNTFiR3dzSWtkbGJtVnlZWFJwYjI1VWFXMWxJanB1ZFd4c0xDSlVhVzFsYzNSaGJYQWlPaUl4T1RBd0xUQXhMVEF4VkRBd09qQXdPakF3V2lJc0lrTm9aV05yVTNWdElqcHVkV3hzTENKSlpDSTZJakF3TURBd01EQXdMVEF3TURBdE1EQXdNQzB3TURBd0xUQXdNREF3TURBd01EQXdNQ0o5TENKUWNtOWtkV04wVFdWMFlXUmhkR0VpT25zaVNYTkpibWwwYVdGc0lqcHVkV3hzTENKSFpXNWxjbUYwYVc5dVZHbHRaU0k2Ym5Wc2JDd2lWR2x0WlhOMFlXMXdJam9pTWpBeU1pMHdPUzB4TTFReE5EbzFNVG8xT0ZvaUxDSkRhR1ZqYTFOMWJTSTZiblZzYkN3aVNXUWlPaUpqTWpWa05ESXpNQzB4TVdZeUxUUm1aVFl0T1RCbU1pMHhPRE0zTjJOalkyVXhNbVFpZlgwIiwiSXNDb250aW51YXRpb25Ub2tlbiI6ZmFsc2V9
[2022-11-14 11:58:45,288] DEBUG {cps.kobo:212} Books to Sync: 0
[2022-11-14 11:58:45,302] DEBUG {cps.kobo:277} Remaining books to Sync: 0

The is no difference for me either the 200 return code or the 307

Krakinou · November 16, 2022, 10:08pm

Ok, so I would say it is syncing : it just think there is nothing left to sync.

Can you try :
First adding a new book to a kobo synched shelf and then sync again.
If this does not work: Try reseting the sync with the “Force full Kobo sync” option in the settings.
In both case, if this does not work, share the log in case something can be seen.

Anyway, if it does not work I would say it is not an issue with the package anymore but more with the upstream app…

wanabo · November 20, 2022, 9:27am

[2022-11-20 10:23:33,443] DEBUG {cps.kobo:212} Books to Sync: 5
[2022-11-20 10:23:33,620] DEBUG {cps.kobo:277} Remaining books to Sync: 0

and few seconds later

[2022-11-20 10:24:22,194] DEBUG {cps.kobo:212} Books to Sync: 0
[2022-11-20 10:24:22,208] DEBUG {cps.kobo:277} Remaining books to Sync: 0

Something happens, but I can’t the books in the kobo. I always have the syncing error.

anyway, thanks a lot for your help !

system · December 5, 2022, 9:27am

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.