Everything about nothing: August 2015

I had a problem that my old mail system is Zimbra 7.2.7 running on CentOS 5. I can not upgrade to the newest version of Zimbra since it isn't supported on CentOS 5 any more. In addition, there is a problem that Zimbra 7.2.7 isn't available on CentOS 7. So, I'm in some kind of a deadlock situation here. To break from this deadlock I decided to first install Zimbra 7.2.7 on CentOS 7 and then to upgrade to Zimbra 8.6. Since Zimbra 7.2.7 isn't supported on CentOS 7, it is a bit of a hacky process. This post describes in detail what I had to do in order to install Zimbra 7 on CentOS 7.

Few things to note before I start with description of the installation. First, my new server has a different IP address from the old one, but I have given both servers the same name (the same FQDN). Also, I installed DNS server on the new server with almost the same configuration as the old one. The difference is in IP addresses pointing to the server itself (MX records primarily). Finally, both servers are 64-bit.

The whole post is structured into two parts. The first part is more in a diary form. The second part is a condensed, cookbook style text on how to upgrade without any text or discussions about problems.

Installation process

Start by downloading Zimbra 7.2.7 for RHEL6 and Zimbra 8.6.0 for RHEL7. Unpack both archives.

Go to the Zimbra 7.2.7 directory and before starting installation process open the file util/utilfunc.sh in the editor of you choice. In line 2287 you'll see the following statement:

PACKAGEINST='rpm -iv'

You should change it to:

PACKAGEINST='rpm --force -iv'

Now, start the installation process with:

# ./install.sh -s --platform-override

Note that I used the option -s so that configuration process isn't started before necessary changes are made. The problem is caused by newer version of Perl on CentOS 7 while installation version of Zimbra 7 relies on an older version from CentOS 5. Also be careful that you select packages for installation that are also installed on the old server!

The idea is to copy Perl from Zimbra 8.6. In order to do so, go to the directory /opt/zimbra/zimbramon and rename directory lib/ to lib.orig/, or something similar:

# cd /opt/zimbra/zimbramon
# mv lib lib.orig

Then, go to the directory where you unpacked Zimbra 8.6.0. You should now unpack zimbra-core RPM package using the following commands:

# mkdir tmp
# cd tmp
# rpm2cpio ../packages/zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm | cpio -id

and then go to directory opt/zimbra/zimbramon (note missing leading slash!) and copy lib directory to Zimbra7 tree:

# rsync --delete -av lib /opt/zimbra/zimbramon/

Additionally, make the following two symlinks in directory /usr/lib64:

# ln -s libssl.so.1.0.1e libssl.so.1.0.0
# ln -s libcrypto.so.1.0.1e libcrypto.so.1.0.0

Finally, run the setup command:

# /opt/zimbra/libexec/zmsetup.pl

It can happen that zmsetup.pl complains about missing Perl modules. You should install any module missing. In my case the missing modules were:

perl-LDAP
perl-Net-DNS

Each time zmsetup.pl stops because some module is missing, just install the missing module and start zmsetup.pl again.

Finally, when the installation was finished, I saved freshly installed Zimbra in case I need it later, e.g. when I want to start from this point on. I did this with the following command:

rsync -av /opt/zimbra /opt/zimbra.orig

Note that in the following text I'll reference from time to time zimbra.orig directory because we'll need some files from there.

Experimenting with a new installation

This section is some form of a diary on how I tried to migrate Zimbra from the old server to the new one, and then how I tried to upgrade Zimbra 7 to Zimbra 8. I think that it is good to write about this trial-and-error process for at least two reasons:

You can learn from my experience, as (almost) nothing goes perfectly without trying.
If you google for some error messages, you might find it helpful to see what is the solution. There are no error messages in the condensed version of the diary section.

In case you just want a cookbook on how to do it, skip to the Cookbook section in this blog post.

Migrating existing Zimbra installation to a new server

I suggest that before you do this, you do some preparation steps:

Block outbound connections to port 25, i.e. prevent new instance of Zimbra to send emails until you migration is complete. And then, before going into production, check mail queue for orphaned/duplicate mails.
Synchronizations are time consuming on any, even moderately used host. So, do first live migration, and then stop old server and repeat migration.
In any case, don't do migration when the old server is running because you might leave your new server in inconsistent state.

Now, go to the old installation of Zimbra and copy the whole Zimbra tree over the Zimbra installation on the new server. The command should look something like this:

$ rsync -av --delete --exclude zimbramon/lib /opt/zimbra root@newserver:/opt/

Probably this will take some time. Also, don't forget to exclude zimbramon/lib directory or otherwise you'll end up with an old version of Perl and when you try to start some tool from Zimbra you'll get the following error message:

$ zmcontrol stop
Perl lib version (v5.8.8) doesn't match executable version (v5.16.3) at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/Config.pm line 46.
Compilation failed in require at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
BEGIN failed--compilation aborted at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
Compilation failed in require at /opt/zimbra/bin/zmcontrol line 25.
BEGIN failed--compilation aborted at /opt/zimbra/bin/zmcontrol line 25.

Anyway, in case you got the previous error just copy again zimbramon/lib directory from Zimbra 8 as explained earlier in the text.

At this point zmcontrol stop/start wan't work almost certainly. So, I suggest that you try to start service by service and check each one if it works or not. Checks can be performed looking at the process list or using netstat/ss command to see listening ports. In case some service doesn't work, check log files in /opt/zimbra/log directory and the file /var/log/zimbra.log. Actually, it would be good to check log files no matter if you think a service works or not.

The first service you should start is ldap. That one is easy, just run the following command as a zimbra user:

$ ldap start

then, look if there is slapd process running:

$ ps -ef | grep sldapd
zimbra 30224 1 0 06:56 ? 00:00:00 /opt/zimbra/openldap/sbin/slapd -l LOCAL0 -u zimbra -h ldap://mail.zemris.fer.hr:389 ldapi:/// -F /opt/zimbra/data/ldap/config

and you can also check if ldap port (389) is opened:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 3.1.2.1:389 0.0.0.0:* LISTEN
tcp 0 0 3.1.2.1:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN
tcp6 0 0 ::1:53 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:953 :::* LISTEN

Logging, by the way, goes to syslog/journalctl because of the option -l, and that means we should look into /var/log/zimbra.log for slapd messages:

Next, I tried to start mailbox service:

$ zmmailboxdctl start

and that didn't work. Namely, no java process:

$ ps -ef | grep java
zimbra 4717 1470 0 07:19 pts/0 00:00:00 grep --color=auto java

and no listening sockets for imap, pop, etc.:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 3.1.2.1:389 0.0.0.0:* LISTEN
tcp 0 0 3.1.2.1:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN
tcp6 0 0 ::1:53 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:953 :::* LISTEN

Looking into log file (/opt/zimbra/log/zmmailboxd.out) reveals the following error messages:

Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
... 18 more
1227 WARN [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Cannot assign requested address

After few more minutes of scratching my head of why (while constantly resisting temptation to use strace tool) I finally realized that the problem is related to my customization on the old server. Namely, I changed jetty configuration so that Zimbra binds to a specific IP address which isn't available on the new server. The solution for this is easy, I saved a copy of Zimbra 7 immediately after installation (zimbra.orig directory, remember) so the following two commands solved this particular problem:

$ cd /opt/zimbra.orig/zimbra/$ rsync -av --delete jetty-6.1.22.z6 /opt/zimbra/

So, again I tried to start mailbox service on Zimbra:

$ zmmailboxdctl startStarting mailboxd...done.

This time, it seems to be good. But checking ports and processes again showed that it doesn't work! Looking at log file showed the following cryptic error:

Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
at javax.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:80)
at com.zimbra.common.util.NetUtil.newBoundServerSocket(NetUtil.java:95)
at com.zimbra.common.util.NetUtil.bindServerSocket(NetUtil.java:163)
at com.zimbra.common.util.NetUtil.bindSslTcpServerSocket(NetUtil.java:54)
...

That was very cryptic error message. Luckily, Googling for it immediately gave the following thread. Namely, the problem is with SSL/TLS, and the following two commands from the given thread fixed that part of the problem:

$ mv /opt/zimbra/mailboxd/etc/keystore /opt/zimbra/mailboxd/etc/keystore.borked
$ sudo /opt/zimbra/bin/zmcertmgr deploycrt self

This time, it almost worked as now I had two java processes and listening sockets. But, there were a log of messages in log file like the following one:

1445 WARN [main] log - failed org.mortbay.jetty.NCSARequestLog@64d55986: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN [main] log - failed RequestLogHandler@72e8a021: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN [main] log - failed HandlerCollection@6691177: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN [main] log - failed RewriteHandler@5bf99eea: java.io.IOException: Cannot write log directory /opt/zimbra/log
1446 WARN [main] log - failed DebugHandler@613043d2: java.io.IOException: Cannot write log directory /opt/zimbra/log

At first sight, that shouldn't happened. Zimbra is owner of those files and SELinux isn't controlling Zimbra. After some scratching I found out that there is a configuration file /opt/zimbra/jetty/etc/jetty.properties in which UID and GID values for Zimbra user/group are not correct, i.e. they are same as on the old server while Zimbra on the new server has different UID/GID values:

$ id zimbra
uid=997(zimbra) gid=996(zimbra) groups=996(zimbra),4(adm),5(tty),89(postfix)

So, I edited it and changed their values to 997 and 996. Note that you should check the correct values for your specific case. Trying again the error messages didn't go away. This time I checked process list and I found out that the process ID of Java is 500. So, I missed some configuration setting. Turns out that jetty.properties file is automatically generated from jetty.properties.in and UID and GID values are taken from Zimbra configuration. So, I had to change them permanently using the following commands (as a zimbra user):

$ zmlocalconfig -e zimbra_uid=997
$ zmlocalconfig -e zimbra_gid=996

And this time it worked! But, I realized now that ldap isn't working any more:

$ ldap start
Failed to start slapd. Attempting debug start to determine error.
TLS: error:0200100D:system library:fopen:Permission denied bss_file.c:398
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:400
55d5688d main: TLS init def ctx failed: -1

After some time I realized that when I redeployed certificate using the link given above, it turns out that UID and GID used to set ownership of different certificate files were the ones specified in Zimbra configuration and not the actual UID and GID, i.e.:

$ ls -l /opt/zimbra/conf/slapd.*
-rw-r-----. 1 500 500 1107 Aug 20 07:36 /opt/zimbra/conf/slapd.crt
-rw-r-----. 1 500 500 1704 Aug 20 07:36 /opt/zimbra/conf/slapd.key

I also had to find all the files whose owner was set to previous value (500) and change it to a new UID/GID values. I did this using find command as a root user:

find /opt/zimbra -uid 500 -exec chown zimbra.zimbra {} \;

Now, both LDAP and mailbox services worked! Time to try zmcontrol start but first I stopped running services. Still didn't work. This time I received two error messages from Web and mail processes:

Starting apache.../opt/zimbra/httpd-2.2.22/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
failed.
/opt/zimbra/postfix/sbin/postalias: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory
/opt/zimbra/postfix/sbin/postfix: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory

But these are easy to solve, just copy missing libraries from CentOS 5 installation. And note that there is a library and symlink to the library! I tried starting Zimbra again using zmcontrol. Again, no luck. The problem this time is a missing Perl subroutine while starting stat module:

Undefined subroutine &main::getZimbraHome called at /opt/zimbra/bin/zmstatctl line 156.

This subroutine was removed in Perl packages distributed with Zimbra 8.6, which we are using here. I found this subroutine to be defined in Zmstat.pl file and first I tried to fix it by just copying this old file, but it turns out that more than one file was changed so I tried, and succeeded, with by copying all non-binary files:

# cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
# rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Trying again zmcontrol we are receiving warnings about deprecated use of certain Perl features:

Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 25.
Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 26.

But these can be safely ignored. Finally, Zimbra 7 works! I did some more checking but not thorough one since my intention now was to upgrade Zimbra to version 8.6.

Upgrading Zimbra 7 to Zimbra 8.6

Finally, I got to the step where I tried to upgrade Zimbra to a newer version. Go to the unpacked Zimbra 8 directory and start upgrade process as usual:

./install.sh --platform-override

After starting upgrade process, I received the following error message:

Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
Fix cert configuration prior to upgrading.

Quick GoogleFu gave the following thread with the solution to the problem.

$ zmlocalconfig -e ldap_master_url=ldaps://mymaster.somewhere.com:636
$ zmlocalconfig -e ldap_url=ldaps://myreplica.somewhere.com:636
$ zmlocalconfig -e ldap_starttls_supported=0
$ zmlocalconfig -e ldap_port=636

Be careful to change host names to names that match your configuration. This wasn't enough because it turned out to be that my CA expired, i.e.:

# openssl x509 -in /opt/zimbra/conf/ca/ca.pem -noout -text | grep 'Not After'
Not After : Apr 11 11:22:21 2015 GMT

Which meant that I had to recreate CA:

# /opt/zimbra/bin/zmcertmgr createca -new
# /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
# /opt/zimbra/bin/zmcertmgr deploycrt self
# /opt/zimbra/bin/zmcertmgr deployca

Be sure to restart slapd after this, i.e. in my case slapd was already running so it didn't pick up new CA/CERT! Actually, it is much better to stop Zimbra and then to start upgrade process. Anyway, I restarted upgrade process! There were two warnings, but it seems they can be ignored:

ERROR 1133 (28000) at line 2: Can't find any matching row in the user table
ERROR 1396 (HY000) at line 1: Operation DROP USER failed for ''@'mail.somedomain.com'

Also, at one point installation stopped because some errors with LDAP's password. The configuration menu was presented with lot of marks that things have to be fixed. I selected LDAP password, and when the configuration program assigned new password everything after that went well.

And that's it! Unfortunately, my old mail has some additional services I had to migrate too, but I won't write about them in this post.

I have to note several things. First, this process heavily depends on configurations on both new and old server being as close as possible. This means that if you did customizations that are outside of /opt/zimbra tree, then you'll have to transfer those too. In my case, I integrated mailman with Zimbra, and mailman lives in other directories, so I had problems. Second,

Quick how to migrate Zimbra7/CentOS5 to Zimbra8/CentOS7

So, here is the recipe on how to move old Zimbra installation to the new server. This is condensed version of the previous section.

Preparation

Download ZCS 7.2.7 (for RHEL6) and ZCS 8.6.0 (you can download newer version if you wish, but I didn't test it so expect there might be some issues). I'll assume you downloaded them into /tmp directory.
Unpack both archives.
Enter into ZCS 8.6 directory and then into packages/ subdirectory. Make there temporary directory (e.g. tmp/) and enter into it. Unpack core package (zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm) using rpm2cpio and cpio commands.
Enter into ZCS 7.2.7 directory and modify file util/utilfunc.sh file. Line 2287 should be PACKAGEINST='rpm --force -iv'.
From the old server copy the following libraries into /lib64 directory on a new server: libssl.so.*, libcrypto.so.*, libpcre.so.*, and libexpat.so.*.

Installation

Enter direcotry ZCS 7.2.7 and start installation with the following command:
./install.sh -s --platform-override
When script finishes, go to /opt/zimbra/zimbramon directory rename lib directory to lib.orig.
Go to the directory where you unpacked zimbra-core package (should be something like /tmp/zcs-8.6.0-something/packages/tmp/) and execute the following command:
rsync -av opt/zimbra/zimbramon/lib /opt/zimbra/zimbramon/.
Run the command /opt/zimbra/libexec/zmsetup.pl. In case the script complains about missing Perl module and stops, install the missing module and run the script again.
Stop Zimbra and disable starting during the boot process:

# /etc/init.d/zimbra stop
# chkconfig zimbra off
Make a copy of vanilla Zimbra directory tree, i.e. execute the following commands:

# cd /opt
# rsync -av zimbra zimbra.orig

Syncing with old server

Go to the old server and run the following command:

rsync -av --delete --exclude zimbramon/lib --exclude jetty\* /opt/zimbra newserver:/opt/
Again on the new server run the following commands (as a zimbra user):

$ zmlocalconfig -e zimbra_uid=997
$ zmlocalconfig -e zimbra_gid=996

Change UID and GID to match your new installation!
Turn on LDAP server, i.e. switch to the Zimbra user and execute the following command:

$ ldap start
Run the following commands to generate new CA.

# /opt/zimbra/bin/zmcertmgr createca -new
# /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
# /opt/zimbra/bin/zmcertmgr deploycrt self
# /opt/zimbra/bin/zmcertmgr deployca

Might not be necessary, but it doesn't hurt. And yes, if you have commercial CERT, you'll have to import it and these steps are not necessary.
Turn off LDAP server, i.e. switch to the Zimbra user and execute the following command:

$ ldap stop
Run the following commands to "fix" Perl version mismatch on new and old servers:

# cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
# rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Upgrade

Go to the directory where you unpacked ZCS 8.6 and run upgrade process as usual:

./install.sh --platform-override

Some additional notes

In this section I'll add some additional notes.

I noticed after some time that while starting Zimbra, zmconfigd service reports error during startup. After some debugging I found out that the service is working, but that the problem is in how check if it is working is implemented. Namely, one step in the process is to connect to localhost port 7171 using nc tool. But since localhost first resolves to IPv6 address, and the service doesn't listen on IPv6, so the connection is refused, nc doesn't try IPv4 then, and script wrongfully thinks that the service isn't running. The solution was to edit /etc/hosts file and change it so that localhost resolves exclusively to 127.0.0.1 while localhost6 resolves to ::1.

Everything about nothing

Monday, August 24, 2015

Installing Zimbra 7.2.7 on CentOS 7 and upgrading to 8.6