Repairing installation after fsck.ext4 gives internal server error

Hi dear Yunohosters,

I made another problem. The system is up-to-date, see details at
https://paste.yunohost.org/qoxeweviqe

In short:

  • “host”: “Debian 9.11”,
  • “kernel”: “4.19.62-sunxi”
  • “yunohost”: “3.6.5.3”
  • “yunohost-admin”: “3.6.5.1”
  • “moulinette”: “3.6.4.1”
  • “ssowat”: "3.6.4

The symptoms:

  • Any non-admin page returns an error 500 from nginX.
  • The admin pages work
  • Serial and SSH also work.
    From the command line I can use the system as if nothing is wrong.
2019/12/14 17:36:29 [error] 3178#3178: *135 lua entry thread aborted: runtime error: /usr/share/lua/5.1/json/decode/util.lua:35: unexpected character @ character: 1 0:1 [Z] line:
Z
stack traceback:
coroutine 0:
        [C]: in function 'error'
        /usr/share/lua/5.1/json/decode/util.lua:35: in function </usr/share/lua/5.1/json/decode/util.lua:32>
        [C]: in function 'match'
        /usr/share/lua/5.1/json/decode.lua:74: in function 'decode'
        /usr/share/ssowat/config.lua:13: in function 'get_config'
        /usr/share/ssowat/access.lua:20: in function </usr/share/ssowat/access.lua:1>, client: 2001:982:1768:1:21c:bfff:fed3:a878, server: domain.tld, request: "GET /nextcloud/status.php HTTP/2.0", host: "domain.tld"

The problem
Yesterday the system had crashed. Before booting the system, I fsck’d the SD-card (relatively fast Kingston 32GB, but not A1 labeled) and it had an error that was restored.

Troubleshooting / workaround
I did make a dd image of the SD-card, but only after fsck’ing. (That image kernel panics very early in boot, but the data is ‘safe’.) (I am, in another thread, busy getting Borg Backup running, that is on another system.)

Now if I check the card again, the same error is still there, I think some flash cells got bad so I intend to create a badblock-list until I can replace the SD-card.

Even before that, I hope to get this installation running again.

I opened the files in the traceback, and compared /usr/share/lua/5.1/json/decode/util.lua to one on another system to see if anything is corrupted.
That is not the case, so it must be the input that is fed to lua’s json-decoder.

If I access any location, there is only a single entry in the access- and error.log for the domain, not in the general nginx-access- and error.log.

From that I conclude that the first rewrite for any URL hangs the lua-json-decoder. I don’t know how to check which character is unexpected, but it can hardly be in one of the nginx-configs because then there would be more logging (correct?).

Now my questions are along two lines:

  1. Is there something that I should NOT do to save me a lot of trouble (such as the time that I started reinstalling seperate packages instead of touching /etc/yunohost/installed…)
  2. How can I troubleshoot this further? Which module makes that there is an error / creates the inpet with the unexpected character? Why json-decoding before anything else?
    2a Am I correct to think that it is a problem with a character of the input?

Update on symptoms:
I realize that I have been accessing the admin-interface via IP, so not via a ynh-configured domain.
Accessing admin via one of the domains gives the same error.

Careful conclusion from that: it falls back to the default server, and no rewriting is done --> no error.

Next:
I am not sure how sso/nginx defaults to the admin interface on IP-based access, so I’ll add a new domain and see if that has the same error. If yes, it would mean that any rewrite returns an error, if not, only existing config is broken.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.