Baikal since upgrade to yunohost 2.4

Hello,

I have upgraded yunohost from 2.2 to 2.4.

The upgrade was successfull (thanks to dev and contributors) except one thing with baikal.

The symptoms are :

  • From agendav I can not see event recorded before the upgrade
  • From agendav I can record an event but I can not see it
  • From agendav the event recorded is recorded in the mysql database
  • From roundcube I can see my contacts recorded before the upgrade but I can not record a new one
  • From roundcube it seems that the contact is not recorded in the mysql database

I can not find in the system logs some messages that helps. The only message I found is :

  • For an event record from agendav

XXX.XXX.XXX.XXX - username [02/Sep/2016:14:23:28 +0200] "REPORT /baikal/cal.php/calendars/username/default/ HTTP/1.1" 500 286 "-" "AgenDAV v1.2.6.2"

  • For a contact record from roundcube

XXX.XXX.XXX.XXX - username [02/Sep/2016:14:25:36 +0200] "POST /webmail/?_task=addressbook&_framed=1&_orig_source=carddav_1 HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=addressbook&_framed=1&_gid=1&_action=add&_source=carddav_1&_cid=0" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"[/details][/details]

Within these two messages we can see an error 500.

I got messages from logs before the upgrade and there is no error 500 :

  • For an event record from agendav

XXX.XXX.XXX.XXX - username [20/Aug/2016:19:29:14 +0200] "REPORT /baikal/cal.php/calendars/username/default/ HTTP/1.1" 207 198 "-" "AgenDAV v1.2.6.2"

  • For a contact record from roundcube

XXX.XXX.XXX.XXX - usernamme [04/Sep/2015:11:22:30 +0200] "POST /webmail/?_task=addressbook&_orig_source=carddav_1 HTTP/1.1" 200 937 "https://host.example.com/webmail/?_task=addressbook&_framed=1&_gid=1&_action=add&_source=carddav_1&_cid=0" "Mozilla/5.0 (X11; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0"

Is somebody had that issue ? Do you know where I can look for any clue ?

Thanks for your help

Regards

–
bidroik

Hi,

Since an upgrade of the Baikal app, AgenDAV (in version 1.2) became buggy with it, with the kind of issue you reported.
The latest version of AgenDAV solves it.
I just released it today so to install it you need :

$ yunohost app fetchlist -u https://app.yunohost.org/official.json -n yunohost
$ yunohost app upgrade agendav 

I am still in the process of analysing what went wrong with the baikal update. For example I noticed that my DAVDroid client does not sync anymore…
I did not look into the CardDAV part, but according to what you said, there might be an issue too with the roundcube carddav plugin.

Thanks for investigating and giving these details. Please keep us posted.

Hi Juju,

Thanks for quick answer.

I do not have good news :slight_smile:

I made the upgrade as you asked an agendav does not work anymore.

The first message I saw when I tried to open agendav is an error message telling me : Internal server error

I tried to force my destiny by typing F5 on the agendav page and messages in nginx were a bit differents

    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:42 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:42 +0200] "GET /agendav/css/agendav-built-2.0.0-beta2.min.css HTTP/1.1" 304 0 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:42 +0200] "GET /agendav/css/agendav-built-print-2.0.0-beta2.min.css HTTP/1.1" 304 0 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - - [03/Sep/2016:12:12:42 +0200] "GET /ynhpanel.js HTTP/1.1" 200 10934 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:42 +0200] "GET /ynhpanel.json HTTP/1.1" 200 2449 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - - [03/Sep/2016:12:12:42 +0200] "GET /ynhpanel.css HTTP/1.1" 200 113127 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:43 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:43 +0200] "GET /agendav/css/agendav-built-2.0.0-beta2.min.css HTTP/1.1" 304 0 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:43 +0200] "GET /agendav/css/agendav-built-print-2.0.0-beta2.min.css HTTP/1.1" 304 0 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - - [03/Sep/2016:12:12:43 +0200] "GET /ynhpanel.js HTTP/1.1" 200 10934 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - username [03/Sep/2016:12:12:43 +0200] "GET /ynhpanel.json HTTP/1.1" 200 2449 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
    xxx.xxx.xxx.xxx - - [03/Sep/2016:12:12:43 +0200] "GET /ynhpanel.css HTTP/1.1" 200 113127 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0
  • Messages in /var/log/agendav/2016-09-03.log are

    [2016-09-03 12:11:31] myapp.CRITICAL: AgenDAV\Exception\ConnectionProblem: cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) (uncaught exception) at /var/www/agendav/web/src/Http/Client.php line 184 {“exception”:"[object] (AgenDAV\Exception\ConnectionProblem(code: 0): cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) at /var/www/agendav/web/src/Http/Client.php:184)"} []
    [2016-09-03 12:12:42] myapp.CRITICAL: AgenDAV\Exception\ConnectionProblem: cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) (uncaught exception) at /var/www/agendav/web/src/Http/Client.php line 184 {“exception”:"[object] (AgenDAV\Exception\ConnectionProblem(code: 0): cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) at /var/www/agendav/web/src/Http/Client.php:184)"} []
    [2016-09-03 12:12:43] myapp.CRITICAL: AgenDAV\Exception\ConnectionProblem: cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) (uncaught exception) at /var/www/agendav/web/src/Http/Client.php line 184 {“exception”:"[object] (AgenDAV\Exception\ConnectionProblem(code: 0): cURL error 35: error:0200100D:system library:fopen:Permission denied (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) at /var/www/agendav/web/src/Http/Client.php:184)"} []

Because I saw messages about permissions I took a look to /var/www/agendav and saw that the owner is root. The directory agendav is 755 but I replace the owner by www-data.

After modifying owner the message Internal server error disapeared but I now have a nice white page and only a white page. The icon “yunohost” which permits to go back to the personal panel is not present.

I’am not familiar with web apps logs but It seems that when I try to log on agendav, the http request send an url with “/webmail”. It should be “/agendav” ? no ?

One more thing … It is not directly related to our issue but I saw that the file /var/www/agendav/web/config/settings.php is world-readable. Maybe the scripts upgrade should modify perms ? Or is it a technical issue ?

Do not hesistate to ask me for more details or helps in debugging. I’am not a guru in web apps and web languages but I have few skills in system administration. At this time I do not know where to look.

Thanks for your help.

–
Bidroik

Hi, I suspect something went wrong during your upgrade, like a php-fpm error.

Please always send the installation/upgrade logs when describing an installation/upgrade issue.
You may be able to upgrade again :

yunohost app upgrade agendav --debug

or if this does not work

yunohost app upgrade agendav --debug --url https://github.com/YunoHost-apps/agendav_ynh

Be sure nginx/php-fpm services are restarted after your upgrade.

Also you can look for errors in /var/log/php5-fpm.log and in /var/log/nginx/yourdomain.com-error.log

I had also the same issue, following a yunohost upgrade from v2.2 to v2.4.
I have just upgraded Agendav and now it is working fine on my side. :smile:
Thank you the upgrade juju!

@bidroik

Thanks for the report about the file permissions, this has now been locked down properly.
Tha latest upgrade introduces a specific agendav user to run the software, and it has write access to the least possible amount of files. Also the config, which holds the mysql credentials, is not world-readable anymore.

I also confirm DAVDroid is working fine with this baikal update. My issue was due to an experimentation with nginx ssl parameters changes, seems like I used a too restricted cipher list for my Android version. The default cipher list is fine.

Hi,

Thanks for your help.

I took the logs during the upgrade but because there was no error I did not post it.

I made a new upgrade as you said without any success. I get a page with the message “Internal server error”. In the /var/log/nginx/mydomain-acces.log :

05/Sep/2016:18:46:00 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"

There is no error in /var/log/php5-fpm.log, only lines like :

[05-Sep-2016 18:44:45] NOTICE: fpm is running, pid 2636
[05-Sep-2016 18:44:45] NOTICE: ready to handle connections
[05-Sep-2016 18:44:45] NOTICE: systemd monitor interval set to 10000ms
[05-Sep-2016 18:45:54] NOTICE: Terminating ...
[05-Sep-2016 18:45:54] NOTICE: exiting, bye-bye!
[05-Sep-2016 18:45:54] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful
[05-Sep-2016 18:45:55] NOTICE: fpm is running, pid 2731
[05-Sep-2016 18:45:55] NOTICE: ready to handle connections
[05-Sep-2016 18:45:55] NOTICE: systemd monitor interval set to 10000ms

By making a grep -w 500 in my nginx log files I saw that, since the upgrade, I have many occurence of that kind of http response. Not only on agendav. Maybe it is a clue. Maybe I have something specific on my system. These “error 500” are absent from my old log file (before the yunohost upgrade). For example I now have :

xxx.xxx.xxx.xxx - username [05/Sep/2016:18:06:05 +0200] "GET /webmail/ HTTP/1.1" 500 5 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:14:54 +0200] "GET /webmail/ HTTP/1.1" 500 5 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:15:19 +0200] "GET /agendav/login HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:15:22 +0200] "GET /agendav/login HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:15:23 +0200] "GET /agendav/login HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:20:56 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:21:34 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:23:59 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:25:11 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:25:34 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX.gmail" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:26:32 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:29:12 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:32:24 +0200] "GET /agendav/index.php/login HTTP/1.1" 500 947 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:06 +0200] "POST /webmail/?_task=login HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:18 +0200] "GET /webmail/ HTTP/1.1" 500 5 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:23 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:25 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:26 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:26 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:26 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:26 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:26 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:27 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:27 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:27 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/webmail/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:45:34 +0200] "GET /agendav/ HTTP/1.1" 500 5 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:46:00 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:51:29 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:51:38 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/agendav/login" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
xxx.xxx.xxx.xxx - username [05/Sep/2016:18:51:49 +0200] "GET /agendav/login HTTP/1.1" 500 947 "https://host.example.com/yunohost/sso/" "Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"

I have no time this afternoon to investigate further. I will post my agendav upgrade log on yunopaste tomorrow afternoon.

Thanks again for your help.

–
Bidroik

Hi all,

I have solved my problem with agendav.

That was a certificate problem.

To understand what was wrong I :

  • put agendav in the “dev” enviromnet instead of prod
  • used strace following the php-fpm processes (agendav pool)

I saw that when opening agendav there an error 35 from the lib curl. This error is happening when there is a problem with ssl connection.
In the strace output I saw that the php-fpm try to read the ca certificate by reading the file /etc/ssl/certs/af6cfc12.0.
This is a symbolic link to /etc/ssl/certs/ssl-cert-snakeoil.pem'. I removed that link and re-create it to point on/etc/ssl/certs/ca-yunohost_crt.pem`

Now agendav is working well, including davdroid sync.

I think what I’ve done is ugly (maybe an update will break my workaround). I do not know how cert management is working on debian.
If someone has an explanation, I’am listening.

PS : problem with my contacts in roundcube and errors 500 (probably the same root cause) are not solved

–
Bidroik

Hi once again,

Contact problem is also solved … Thanks to strace :slight_smile:

This is a migration problem with roundcube.

It seems that during the roundcube update there was a problem with the database.

If in the logs /var/www/roundcube/logs/carddav.warn you see messages like : BACKEND: (update_addressbook) [1054] Unknown column 'authentication_scheme' in 'field list' you can refer to the page https://github.com/blind-coder/rcmcarddav/issues/156

The idea is to drop all the tables named carddav_* from the roundcube database then logout and login into rouncube.

On my side I lost some group assignement but all my contacts are there.

The process I follow is :

  • First backup the tables you will drop :

mysqldump --user=<user> --password=<pasword> --databases roundcube --tables carddav_addressbooks carddav_contacts carddav_group_user carddav_groups carddav_migrations carddav_xsubtypes > /<my_dir>/tables-carddav.sql

  • Log into mysql and drop the tables in this order (there are foreign keys)
    $ mysql -u root -p
    mysql> use roundcube;
    mysql> drop table carddav_xsubtypes;
    mysql> drop table carddav_migrations;
    mysql> drop table carddav_group_user;
    mysql> drop table carddav_groups;
    mysql> drop table carddav_contacts;
    mysql> drop table carddav_addressbooks;

  • Now you can log out the log in. The roundcound will be a little bit slow like everyday because of the database upgrade.

This operation also solved the error 500 and white page in roundcube

Hope this work for you and do not forget to make backup before !!!

bye

–
Bidroik

1 Like

Thank you. I had the same contact issue and your post helped me to fix it.