How to solve SELinux and MySQL log rotation issue

Written by Ivan Groenewold | Aug 14, 2019 4:00:00 AM

Disable SELinux? Think again!

SELinux is always a complicated topic. If you search this on the web, most people will advise just disabling it, but that may not be acceptable from a security point of view in your organization. In this case, we are going to see how to solve an issue with SELinux and MySQL log rotation We had configured log rotation as per this post. The scripts seemed to work perfectly when running manually. However, when running under cron they would fail to run. Since there were no other errors in the logs, eventually I tracked that down to SELinux. What I found is that SELinux default policies prevent logrotate daemon from making the changes to files outside of /var/log. In this case, MySQL logs were living on /var/lib/mysql so that was clearly the problem.

Figuring out SELinux

The first thing to do when debugging a SELinux issue, is to set permissive mode on a test server.

# setenforce 0

This mode will have the effect of letting processes make any changes they need, and SELinux will record any operations that would be blocked if we were running in enforcing mode. Now, we set up our logrotate cron with the script, and wait for the next run to happen. After the next run, we can check /var/log/audit/audit.log. After filtering the info we need, we can display the error in a human-readable format by running the following command:


# grep logrotate /var/log/audit/audit.log | audit2allow -w -a

This will generate one or more messages like the following:

type=AVC msg=audit(1557805022.012:2213614): avc: denied { setattr } for pid=4645 comm="logrotate" name="mysql-error.log" dev="dm-1" ino=4891395 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:mysqld_db_t:s0 tclass=file
 Was caused by:
 Missing type enforcement (TE) allow rule.
 
You can use audit2allow to generate a loadable module to allow this access.

Now we have some more details about the problem. If the above command doesn't work for you, it's likely you will need to install some helper packages e.g.


# yum -y install policycoreutils-python

Generating custom policy files

Once we have the information, we can go ahead and build a custom SELinux policy that will (hopefully) solve the issue. The first step is to generate a .te policy file, which is in human-readable format:


# grep logrotate /var/log/audit/audit.log | audit2allow -m mypol > mypol.te
 # cat mypol.te
 
 module mypol 1.0;
 
 require {
 type mysqld_db_t;
 type logrotate_t;
 class file setattr;
 } 

#============= logrotate_t ==============
 allow logrotate_t mysqld_db_t:file setattr;

Now we have to convert the .te file to a policy module:


# checkmodule -M -m -o mypol.mod mypol.te

and also compile the policy by running:


# semodule_package -o mypol.pp -m mypol.mod

Finally, we can install the generated .pp file by doing:

semodule -i mypol.pp

Now we can check the list of installed policies to validate it is present:


# semodule -l
 ...
 mrtg 1.9.0
 mta 2.7.3
 munin 1.9.1
 mypol 1.0
 ...

Now everything should work, right? Think again! In my testing, I found that sometimes I needed to repeat the above process several times to gather all the necessary rules, as some messages appeared in audit log only after I fixed the previous error. The good thing is you can combine all rules that are displayed by audit2allow in a single policy file.

Even more fun!

Another thing that can complicate debugging is that some policies have a way to bypass auditing which means that even if there is a denied operation, you won't see it in audit.log. You can rebuild existing policies without the "don't audit" rules to get every single message to display as follows:

semodule -DB

After you do your debugging, you can build it back with "don't audit" rules included again:

semodule -B

Closing thoughts

I am sure by now you love SELinux as much as I do... To sum up, the policy that worked for this case is:

module pol 1.0;
 
 require {
 type mysqld_db_t;
 type logrotate_t;
 class dir { add_name read remove_name write };
 class file { create ioctl open read rename getattr setattr unlink write };
 }
 
 #============= logrotate_t ==============
 allow logrotate_t mysqld_db_t:dir { add_name read remove_name write };
 allow logrotate_t mysqld_db_t:file { create ioctl open read rename getattr setattr unlink write };

Note that with this policy, logrotate can also make changes to any mysqld_db_t file, which means datafiles and/or redo log files for example. Ideally, one would create a new file type just for log files, but this will be subject of another blog post.

Happy SELinuxing.

MySQL Database Consulting

Looking to optimize your MySQL use?

View full post