Time-out error / fastcgi

rockyIII · February 25, 2020, 10:32am

Hi
we try to run a 3.6.5.3 yunohost server with

hubzilla
Synapse
Riot
OpenSondage
SearX
rss-bridge
TinyTiny Rss
Wordpress

Right now the server does not run stable and we can´t figure out why…

every few hours we get a nginx

“504 Gateway Time-out”
error

In the nginx error logs I find something like this:

[error] 27404#27404: *995669 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.345.67.89, server: xyz.tld, request: "GET /channel/xxx HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "xyz.tld""

Has someone an idea what I could try to fix this problems? THANKS

ljf · February 25, 2020, 11:12am

If you check your services in Web admin > Services Do you see some services down ?

rockyIII · February 25, 2020, 11:17am

NO - all services are UP

ljf · February 25, 2020, 11:18am

Even if you try to check it with

systemctl

Type “q” to go out of systemctl

You have a 504 error for each apps ?

rockyIII · February 25, 2020, 2:36pm

right now the system is up

systemctl

gives me right now:

root@admin-tools:~# systemctl                                                                             
  UNIT                                             LOAD   ACTIVE SUB       DESCRIPTION                    
  proc-sys-fs-binfmt_misc.automount                loaded active running   Arbitrary Executable File Forma
  sys-devices-platform-serial8250-tty-ttyS0.device loaded active plugged   /sys/devices/platform/serial825
  sys-devices-platform-serial8250-tty-ttyS1.device loaded active plugged   /sys/devices/platform/serial825
  sys-devices-platform-serial8250-tty-ttyS2.device loaded active plugged   /sys/devices/platform/serial825
  sys-devices-platform-serial8250-tty-ttyS3.device loaded active plugged   /sys/devices/platform/serial825
  sys-devices-vbd\x2d51713-block-xvda1.device      loaded active plugged   /sys/devices/vbd-51713/block/xv
  sys-devices-vbd\x2d51714-block-xvda2.device      loaded active plugged   /sys/devices/vbd-51714/block/xv
  sys-devices-vif\x2d0-net-eth0.device             loaded active plugged   /sys/devices/vif-0/net/eth0    
  sys-devices-virtual-tty-hvc0.device              loaded active plugged   /sys/devices/virtual/tty/hvc0  
  sys-devices-virtual-tty-hvc1.device              loaded active plugged   /sys/devices/virtual/tty/hvc1  
  sys-devices-virtual-tty-hvc2.device              loaded active plugged   /sys/devices/virtual/tty/hvc2  
  sys-devices-virtual-tty-hvc3.device              loaded active plugged   /sys/devices/virtual/tty/hvc3  
  sys-devices-virtual-tty-hvc4.device              loaded active plugged   /sys/devices/virtual/tty/hvc4  
  sys-devices-virtual-tty-hvc5.device              loaded active plugged   /sys/devices/virtual/tty/hvc5  
  sys-devices-virtual-tty-hvc6.device              loaded active plugged   /sys/devices/virtual/tty/hvc6  
  sys-devices-virtual-tty-hvc7.device              loaded active plugged   /sys/devices/virtual/tty/hvc7  
  sys-subsystem-net-devices-eth0.device            loaded active plugged   /sys/subsystem/net/devices/eth0
  -.mount                                          loaded active mounted   Root Mount                     
  dev-mqueue.mount                                 loaded active mounted   POSIX Message Queue File System
  proc-sys-fs-binfmt_misc.mount                    loaded active mounted   Arbitrary Executable File Forma
  run-user-0.mount                                 loaded active mounted   /run/user/0                    
  sys-kernel-debug-tracing.mount                   loaded active mounted   /sys/kernel/debug/tracing      
  sys-kernel-debug.mount                           loaded active mounted   Debug File System              
  systemd-ask-password-console.path                loaded active waiting   Dispatch Password Requests to C
  systemd-ask-password-wall.path                   loaded active waiting   Forward Password Requests to Wa
  init.scope                                       loaded active running   System and Service Manager     
  session-17917.scope                              loaded active running   Session 17917 of user root     
  avahi-daemon.service                             loaded active running   Avahi mDNS/DNS-SD Stack        
  coturn-synapse.service                           loaded active running   coturn                         
  coturn.service                                   loaded active exited    LSB: coturn TURN Server        
  cron.service                                     loaded active running   Regular background program proc
  dbus.service                                     loaded active running   D-Bus System Message Bus       
  dnsmasq.service                                  loaded active running   dnsmasq - A lightweight DHCP an
  dovecot.service                                  loaded active running   Dovecot IMAP/POP3 email server 
  fail2ban.service                                 loaded active running   Fail2Ban Service               
  fake-hwclock.service                             loaded active exited    Restore / save the current cloc
  getty@tty1.service                               loaded active running   Getty on tty1                  
  glances.service                                  loaded active running   LSB: Starts and daemonize Glanc
  haveged.service                                  loaded active running   Entropy daemon using the HAVEGE
  hddtemp.service                                  loaded active exited    LSB: disk temperature monitorin
root@admin-tools:~#

So your are saying i should try systemctl when the system is showing this 504 timeout error?

ljf · February 25, 2020, 2:42pm

With systemctl you need to use the down arrow to see all services before to type q to quit…

ljf · February 25, 2020, 2:44pm

If 504 errors are disapearing itself, it could be an overload issue. You could check resources used (ram), synapse use lot of ram

use “free -m” to check ram

rockyIII · February 25, 2020, 2:49pm

root@admin-tools:~# free -m                                                                                 
              total        used        free      shared  buff/cache   available                             
Mem:            980         458         149          27         372         476                             
Swap:          1023         766         257                                                                 
root@admin-tools:~#

ljf · February 25, 2020, 2:51pm

i think 980M is a bit too small for synapse but i dont host this service myself

rockyIII · February 25, 2020, 2:56pm

root@admin-tools:~# systemctl                                                                               
  UNIT                                             LOAD   ACTIVE SUB       DESCRIPTION                      
  proc-sys-fs-binfmt_misc.automount                loaded active running   Arbitrary Executable File Formats
  sys-devices-platform-serial8250-tty-ttyS0.device loaded active plugged   /sys/devices/platform/serial8250/
  sys-devices-platform-serial8250-tty-ttyS1.device loaded active plugged   /sys/devices/platform/serial8250/
  sys-devices-platform-serial8250-tty-ttyS2.device loaded active plugged   /sys/devices/platform/serial8250/
  sys-devices-platform-serial8250-tty-ttyS3.device loaded active plugged   /sys/devices/platform/serial8250/
  sys-devices-vbd\x2d51713-block-xvda1.device      loaded active plugged   /sys/devices/vbd-51713/block/xvda
  sys-devices-vbd\x2d51714-block-xvda2.device      loaded active plugged   /sys/devices/vbd-51714/block/xvda
  sys-devices-vif\x2d0-net-eth0.device             loaded active plugged   /sys/devices/vif-0/net/eth0      
  sys-devices-virtual-tty-hvc0.device              loaded active plugged   /sys/devices/virtual/tty/hvc0    
  sys-devices-virtual-tty-hvc1.device              loaded active plugged   /sys/devices/virtual/tty/hvc1    
  sys-devices-virtual-tty-hvc2.device              loaded active plugged   /sys/devices/virtual/tty/hvc2    
  sys-devices-virtual-tty-hvc3.device              loaded active plugged   /sys/devices/virtual/tty/hvc3    
  sys-devices-virtual-tty-hvc4.device              loaded active plugged   /sys/devices/virtual/tty/hvc4    
  sys-devices-virtual-tty-hvc5.device              loaded active plugged   /sys/devices/virtual/tty/hvc5    
  sys-devices-virtual-tty-hvc6.device              loaded active plugged   /sys/devices/virtual/tty/hvc6    
  sys-devices-virtual-tty-hvc7.device              loaded active plugged   /sys/devices/virtual/tty/hvc7    
  sys-subsystem-net-devices-eth0.device            loaded active plugged   /sys/subsystem/net/devices/eth0  
  -.mount                                          loaded active mounted   Root Mount                       
  dev-mqueue.mount                                 loaded active mounted   POSIX Message Queue File System  
  proc-sys-fs-binfmt_misc.mount                    loaded active mounted   Arbitrary Executable File Formats
  run-user-0.mount                                 loaded active mounted   /run/user/0                      
  sys-kernel-debug-tracing.mount                   loaded active mounted   /sys/kernel/debug/tracing        
  sys-kernel-debug.mount                           loaded active mounted   Debug File System                
  systemd-ask-password-console.path                loaded active waiting   Dispatch Password Requests to Con
  systemd-ask-password-wall.path                   loaded active waiting   Forward Password Requests to Wall
  init.scope                                       loaded active running   System and Service Manager       
  session-17924.scope                              loaded active running   Session 17924 of user root       
  avahi-daemon.service                             loaded active running   Avahi mDNS/DNS-SD Stack          
  coturn-synapse.service                           loaded active running   coturn                           
  coturn.service                                   loaded active exited    LSB: coturn TURN Server          
  cron.service                                     loaded active running   Regular background program proces
  dbus.service                                     loaded active running   D-Bus System Message Bus         
  dnsmasq.service                                  loaded active running   dnsmasq - A lightweight DHCP and 
  dovecot.service                                  loaded active running   Dovecot IMAP/POP3 email server   
  fail2ban.service                                 loaded active running   Fail2Ban Service                 
  fake-hwclock.service                             loaded active exited    Restore / save the current clock 
  getty@tty1.service                               loaded active running   Getty on tty1                    
  glances.service                                  loaded active running   LSB: Starts and daemonize Glances
  haveged.service                                  loaded active running   Entropy daemon using the HAVEGE a
  hddtemp.service                                  loaded active exited    LSB: disk temperature monitoring 
  kmod-static-nodes.service                        loaded active exited    Create list of required static de
  lm-sensors.service                               loaded active exited    Initialize hardware monitoring se
  mariadb.service                                  loaded active running   MariaDB 10.1.44 database server  
  matrix-synapse.service                           loaded active running   Synapse Matrix homeserver        
  metronome.service                                loaded active running   LSB: Metronome XMPP Server       
  networking.service                               loaded active exited    Raise network interfaces         
  nginx.service                                    loaded active running   A high performance web server and
  nslcd.service                                    loaded active running   LSB: LDAP connection daemon      
  ntp.service                                      loaded active running   LSB: Start NTP daemon            
  php7.0-fpm.service                               loaded active running   The PHP 7.0 FastCGI Process Manag
  php7.4-fpm.service                               loaded active running   The PHP 7.4 FastCGI Process Manag
  postfix.service                                  loaded active exited    Postfix Mail Transport Agent     
  postfix@-.service                                loaded active running   Postfix Mail Transport Agent (ins
  postgresql.service                               loaded active exited    PostgreSQL RDBMS                 
  postgresql@9.6-main.service                      loaded active running   PostgreSQL Cluster 9.6-main      
  postsrsd.service                                 loaded active running   PostSRSd Daemon                  
  quota.service                                    loaded active exited    Initial Check File System Quotas 
  redis-server.service                             loaded active running   Advanced key-value store         
  resolvconf.service                               loaded active exited    Nameserver information manager   
  rspamd.service                                   loaded active running   rapid spam filtering system      
  rsyslog.service                                  loaded active running   System Logging Service           
  serial-getty@hvc0.service                        loaded active running   Serial Getty on hvc0             
  shellinabox.service                              loaded active running   LSB: Shell In A Box Daemon       
  slapd.service                                    loaded active running   LSB: OpenLDAP standalone server (
● snmpd.service                                    loaded failed failed    Simple Network Management Protoco
  ssh.service                                      loaded active running   OpenBSD Secure Shell server      
  sysstat.service                                  loaded active exited    LSB: Start/stop sysstat's sadc   
  systemd-journald.service                         loaded active running   Journal Service                  
  systemd-logind.service                           loaded active running   Login Service                    
  systemd-modules-load.service                     loaded active exited    Load Kernel Modules              
  systemd-random-seed.service                      loaded active exited    Load/Save Random Seed            
  systemd-remount-fs.service                       loaded active exited    Remount Root and Kernel File Syst
  systemd-resolved.service                         loaded active running   Network Name Resolution          
  systemd-sysctl.service                           loaded active exited    Apply Kernel Variables           
  systemd-tmpfiles-setup-dev.service               loaded active exited    Create Static Device Nodes in /de
  systemd-tmpfiles-setup.service                   loaded active exited    Create Volatile Files and Directo
  systemd-udev-trigger.service                     loaded active exited    udev Coldplug all Devices        
  systemd-udevd.service                            loaded active running   udev Kernel Device Manager       
  systemd-update-utmp.service                      loaded active exited    Update UTMP about System Boot/Shu
  systemd-user-sessions.service                    loaded active exited    Permit User Sessions             
  ttrss.service                                    loaded active running   ttrss_backend                    
  unattended-upgrades.service                      loaded active exited    Unattended Upgrades Shutdown     
  unscd.service                                    loaded active running   Name Service Cache Daemon        
  user@0.service                                   loaded active running   User Manager for UID 0           
  uwsgi.service                                    loaded active running   LSB: Start/stop uWSGI server inst
  yunohost-api.service                             loaded active running   YunoHost API Server              
  yunohost-firewall.service                        loaded active exited    YunoHost Firewall                
  -.slice                                          loaded active active    Root Slice                       
  system-getty.slice                               loaded active active    system-getty.slice               
  system-postfix.slice                             loaded active active    system-postfix.slice             
  system-postgresql.slice                          loaded active active    system-postgresql.slice          
  system-serial\x2dgetty.slice                     loaded active active    system-serial\x2dgetty.slice     
  system.slice                                     loaded active active    System Slice                     
  user-0.slice                                     loaded active active    User Slice of root               
  user-0.slice                                     loaded active active    User Slice of root               
  user.slice                                       loaded active active    User and Session Slice           
  avahi-daemon.socket                              loaded active running   Avahi mDNS/DNS-SD Stack Activatio
  dbus.socket                                      loaded active running   D-Bus System Message Bus Socket  
  syslog.socket                                    loaded active running   Syslog Socket                    
  systemd-fsckd.socket                             loaded active listening fsck to fsckd communication Socke
  systemd-initctl.socket                           loaded active listening /dev/initctl Compatibility Named 
  systemd-journald-audit.socket                    loaded active running   Journal Audit Socket             
  systemd-journald-dev-log.socket                  loaded active running   Journal Socket (/dev/log)        
  systemd-journald.socket                          loaded active running   Journal Socket                   
  systemd-udevd-control.socket                     loaded active running   udev Control Socket              
  systemd-udevd-kernel.socket                      loaded active running   udev Kernel Socket               
  dev-xvda2.swap                                   loaded active active    /dev/xvda2                       
  basic.target                                     loaded active active    Basic System                     
  cryptsetup.target                                loaded active active    Encrypted Volumes                
  getty.target                                     loaded active active    Login Prompts                    
  graphical.target                                 loaded active active    Graphical Interface              
  local-fs-pre.target                              loaded active active    Local File Systems (Pre)         
  local-fs.target                                  loaded active active    Local File Systems               
  multi-user.target                                loaded active active    Multi-User System                
  network-online.target                            loaded active active    Network is Online                
  network.target                                   loaded active active    Network                          
  nss-lookup.target                                loaded active active    Host and Network Name Lookups    
  paths.target                                     loaded active active    Paths                            
  remote-fs.target                                 loaded active active    Remote File Systems              
  slices.target                                    loaded active active    Slices                           
  sockets.target                                   loaded active active    Sockets                          
  swap.target                                      loaded active active    Swap                             
  sysinit.target                                   loaded active active    System Initialization            
  time-sync.target                                 loaded active active    System Time Synchronized         
  timers.target                                    loaded active active    Timers                           
  apt-daily-upgrade.timer                          loaded active waiting   Daily apt upgrade and clean activ
  apt-daily.timer                                  loaded active waiting   Daily apt download activities    
  phpsessionclean.timer                            loaded active waiting   Clean PHP session files every 30 
  systemd-tmpfiles-clean.timer                     loaded active waiting   Daily Cleanup of Temporary Direct


LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.


128 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
root@admin-tools:~#

ljf · February 25, 2020, 3:14pm

I really think you rproblem is a RAM issue. You need more ram or less apps. If the computer use swap too often it could slow your system and trigger timeout error.

It’s the same if you have bad input/output (io) with stockage. But here i think it’s just a ram issue.

In more if your swap is full, you could have some service to be shutdown automatically.

I really believe synapse is a bad idea on a small computer like this.

rockyIII · February 25, 2020, 3:16pm

Thanks for your help… i will see if I can run the v-server with more ram…

thanks so much so fare

rockyIII · February 26, 2020, 12:50pm

I deactivated the services
matrix-synapse
coturn-synapse

but even so this does not free any RAM but quid the opposite…

root@admin-tools:~# free -m                                                                               
              total        used        free      shared  buff/cache   available                           
Mem:            980         788          13          48         178         125                           
Swap:          1023           7        1016                                                               
root@admin-tools:~#

do you have an idea way ? what can I do to free RAM?

ljf · February 26, 2020, 12:56pm

Have you deactivated and stopped those services ?

In your second free -m i see the total of ram + swap decrease. So it could have worked.

If you don’t use xmpp you can stop the service metronome

rockyIII · February 26, 2020, 1:09pm

I deactivated and stoped the services

matrix-synapse
coturn-synapse
metronome

and restarted the server

now I get this

root@admin-tools:~# free -m                                                                               
              total        used        free      shared  buff/cache   available                           
Mem:            980         644          23          47         312         265                           
Swap:          1023           1        1022                                                               
root@admin-tools:~#

so you think it freed ram even if it says just 23 free ram ?

ljf · February 26, 2020, 3:06pm

The available info and buff/cache info are more important.

rockyIII · March 7, 2020, 11:37am

after we updated the RAM to 2GB we have solved the problem so fare. THANKS

system · March 22, 2020, 11:37am

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.