Autodeployment using GitLab Webhooks

Gitlab webhooks

GitLab Webhooks

GitLab provides various Webhooks to perform automated tasks after a certain event. For e.g. Push events Gitlab webhooks will work exactly similar to post-receive hook in GIT. Only thing is it needs a webpage (mostly on your server) which gets executed after a certain commit is made to the GitLab repository. This webpage can perform tasks like updating the local mirror repository and making the corresponding change in the files in your public_html folder. Read more about webhooks in GitLab

Assuming you have a GitLab repository at:

Steps to follow on your production/live server

  1. SSH to your production/live server
  2. Generate an SSH key for auto deployment to be used by the GitLab server

    Name of the key: gitlab_rsa_deploy, keep the passphrase blank
  3. Add the contents of the key to Deyloy Keys section of your Project Settings and Enable it
  4. Create a config file in .ssh folder and add below contents to the file. Add your user key added to GitLab user key settings.
  5. Create a folder name gitlab preferably outside your public_html folder
  6. Create a mirror repository by cloning your gitlab project
  7. Set Work Directory
  8. Perform git fetch once on the mirror repository so that the SSH keys get added to the server
  9. Create the webhook file on your server as webhook.php. Add below contents to the file and save this file inside your public_html folder.
  10. Add the URL of the webhook.php file to the list of webhooks on GitLab for Push events as shown below
    GitLab Webhook
  11. Now just commit some change either through GitLab or through your local repository and check if autodeployment is working

Points to consider before installing a new wordpress plugin

Installing a new plugin is very easy in wordpress. All you need to do is to search for your plugin, select it, install and activate. If you do not find it suitable to your needs, just deactivate, delete it and move on.

However not many people realise what a plugin does in the background once it gets activated and assume that once a plugin is deleted its all gone which is really not true in most cases. A deleted plugin mostly leaves quite a few traces in your system. Depending upon the plugin these traces can severely affect the performance of your system if you try too many plugins without checking what it does in the background

Many plugin developers do not follow WordPress coding standards. They do not provide an unintall function for the plugin. This means you need to manually clean up all the traces of the plugin after it is deactivated and deleted.

I am currently not experiencing any performance issue with my site

Although your site may be performing well currently, if you do not consider cleaning your system then soon a badly developed plugin can cause wordpress queries to slow down. Once the system starts slowing down it will be difficult to clean the system as by then you won’t be sure which tables are currently in use and which records in the database are unnecessary as it is quite easy to forget which plugins you tried in the past.

e.g. Some plugins developers are not quite efficient in their coding. They save each setting as a separate row in the wp_options table. Ideally all settings could be stored in a single row if setting are stored in an array. I remember a plugin which created 300 rows in the wp_options table to store settings with autoload set to yes which means the rows get selected on every Dashboard page load.

WordPress gets all the rows on each Dashboard page. If the table size grows bigger it will definately slow down the sytem.

Points to consider before installing a new wordpress plugin

First of all Install, activate and configure the plugin on your test environment or localhost

Check the source code both on Dashboard and your website: Check how many CSS and JS calls the plugin is making, whether it adds any inline CSS and other extra code.

Some plugins developers do this quite efficiently. They provide a settings screen and ask you on which Posts Types you want the plugin to run which means plugin related CSS or JS files only appear on those Post Types.

Are the queries optimised: Check if your new plugin is executing any queries which can slow down your system. To monitor such slow queries make use of Query Monitor plugin. Here are some examples how to make use of Query Monitor plugin to increase performance of your WordPress site and to detect slow queries hampering your system.

Does it create new tables: Check if the plugin creates any new tables. If yes make a note of those tables somewhere. Check what data it adds to those tables and if the tables are really necessary or the plugin developer could have used WordPress inbuit tables. This will give you an idea if the plugin developer has thought about optimising the code. Additional tables are ok but to use wordpress inbuilt tables would be ideal as wordpress has already optimised queries for those tables.

If you have a multisite, most probably the plugin would create extra tables for all your sites.

What does it add to wp_options table: Some plugin authors add _tansient records to this table. e.g. a plugin used for related links would add related links for each post in this table. So for 20k posts there would be 20k records in this table. These records get selected on every dashboard page load. This would slow down the dashboard to a great extent.

wp_options table is generally used to store plugin settings. Some plugin authors store each setting in a separate row. Ideally all settings should be stored in a single row in an arrow form.

Does the plugin provide an uninstall function: A good plugin developer provides an uninstall function which deletes the tables, capabilities, entries in wp_option table etc once the plugin is deactivated.


How to tackle WordPress slow queries

How to tackle WordPress slow queries

Photo credit – elisfanclub859241997

Here are some wordpress slow queries i.e. queries which take more than 0.05s. It really depends on your wordpress site i.e. how big is the database, plugins and your site configuration. However if you are facing performance issues related to the Dashboard then it is more likely to be due to the slow wordpress dashboard queries.

Query Monitor is good plugin to check/analyse your slow queries.

Some WordPress Slow queries

Below query auto populates the custom fields drop down box.

For large tables this query can take lot of time like 2secs or so. If you do not need custom fields it is very easy to turn them off using below function.

For more information read this interesting post on CSS Tricks

Below query runs on every Dashboard page so it is important that your wp_options table is optimised.

Depending upon the plugins you have installed, the wp_options table size can grow rapidly. Some plugins use this table to store _transient options. These _transient options are objects stored in cache. For e.g. a plugin called as Manual Related Posts stores related links per post in separate rows as _transient options. So if you have 50K posts there would be 50K rows in this table. The size of the table can also grow rapidly because each row would atleast be 1M in size.

The table structure is also not optimised properly for e.g. option_id column is defined as bigint type, autoload is set to var. It should have been set as boolean or enum. Depending upon the table size and other configurations it can even take 8secs to run above query which is quite alarming.

Few tips to optimize wp_options table

  1. Check for _transient entries and if possible replace the plugins which create lot of _transient entries. Please note that although the purpose of these entries is for caching, since this table runs crucial queries on all Dashboard pages, it defeats the purpose for large wordpress installations. The table structure also does not help the cause.
  2. If it is not possible to replace the plugins creating lot of _transient entries then use the Transient Cleaner plugin which will delete the expired transient entries and will automatically do the housekeeping for you.
  3. Change the table structure a bit. Add autoload column to the list of indexes, change the option_id to int (12)

Types of GIT repositories

GIT Repositories
GIT Repositories

Three types of GIT repositories

  1. GIT working repository
  2. GIT bare repository
  3. GIT mirror repository

GIT working repository

This repository is the one which is on your local machine. This is the respositoy which you work on i.e. add, edit or delete your application files and commit your changes.

This repository can be created using below 2 ways

  1. git init: Initialises an empty working repository. Necessary files are then added or created later. This command is mostly used when you are starting with a new application in which case the repository is empty when you begin.
  2. git clone: This command is used if you already have a working repository on your server, GitHub, GitLab, etc. This will create a Git repository on your local maching with working/application files already inside the repository.

This repository has a .git folder with all the version related files inside this folder while your application files reside outside this folder.

GIT bare repository

This repository is generally created on your server mostly for sharing or collaboration purposes. This repository does not include your application files. It only includes the version control related files. All the commits from the local repositories are made to this repository

This repository is created using below command

GIT mirror repsitory

This repository is the exact replica of the source repository. It is created for mainly 2 purposes

  1. Autodeployment
  2. Back up your GIT repository to a remote server

As shown in the above diagram, a mirror repository is created on your production server to autodeploy the commits to the working directory. In the above diagram, local commits are pushed to GitHub or GitLab repository. A webhook on these servers then updates the mirror repository on your production server and updates the files in your working directory.

Prevent your emails going to spam/junk

Photo credit 11183232994 suicidexking

Emails going to spam folder can be a sticky issue and generally there is no simple way to fix the problem. If you are already facing this issue then either your domain is blacklisted somewhere or you may just be missing a few simple tricks.

In any case a preliminary check should be to check if your sending domain or your IP is blacklisted. This would be a preliminary check because if you are already blacklisted then no matter what efforts you make to fix other things your emails will still end up going to spam / junk folder.

Check if your site is blacklisted using below tools by adding your site domain or IP address. If you are using cloudflare then try both i.e. your site domain and then your site IP address. This is because if you are using cloudflare, a domain check may not resolve to the true IP of your site.

If your site is blacklisted you will need to request to have the mail server’s IP address delisted from any blacklists that it’s currently on.

Here’s a guide on how to get your IP delisted from a particular blacklist

If your site is not blacklisted then its half the good news for you. Below steps would fix most of the issues.

Check your email headers

All your domain emails should have SPF and DKIM records and they should be properly configured through your DNS settings. Both these records should show up as pass in your email headers.

One way to check your email headers is to send an email through your domain to one of the gmail addresses, open the email, click on the down arrow next to the “Reply” button, and select show original which will open a new tab with the email headers printed on top. On this page, if you see “signed-by: your domain name” then your DKIM signature is configured properly.

If you are on shared host, mostly these records are set up correctly and you won’t have to do anything. If they are not set up properly i.e. if they show up as fail or with some other status message you can request your hosting provider to set them up correctly. However please note that if you are on a shared host you are sharing the IP address with a few other sites. If any of the other sites/domains are blacklisted or are involved in some form of spamming it directly affects the IP and thus it affects the emails sent through your domain.

Double opt-in

Do no add any email addresses to your mailing list without verifying them i.e. always try to have a doubte opt-in method. The double opt-in confirms that the person who entered their email address actually wants to subscribe to the list. This can be easily achived by sending a verification email to the email address entered by the user.

Easy opt-out

Each email sent by you should have an unsubscribe link easily visible/prominent within the email. If users do not find this link they are more likely to get frustrated and mark your email as Spam or even Report Abuse. Since such users are never going to read your emails anyway it is better to provide them with an easily accessible Unsubscribe option.

Clear communication

It is a good policy to tell your users about the frequency of emails they will be getting from you. If for some reason you change the frequency, it is better to inform your users beforhand and to review their subscriptions. Such communciation only helps in building trust. If someone opts out due to such communication it will only help to filter your database to keep quality contacts.

Bulk mail sender

Sometimes either Gmail, Yahoo or Hotmail may block or mark all your emails as spam. It is very easy to identify this issue. If most of your Gmail users are registering but they are not verifying their accounts that means it is likely that Gmail is marking your emails as spam. This could be for number of reasons. One reason could be Gmail does not recognise you as a Bulk mail sender so when it receives lot of emails from your domain within a short period of time it thinks that you are spamming. In that case it is important to tell Gmail that you are a bulk mail sender by filling Bulk sender contact form

House keeping

Try to make some automation or regularly monitor your mailing list to keep it clean

  1. Automatically unsubscribe users whose addresses keep bouncing. Hard bounces can be removed straightaway while soft bounces can be monitored a few times.
  2. Remove users from your database who have marked youe emails as spam

Email Transaction system

It is good practice to use some form of email transactional system like Sendgrid, Mailgun, etc

Such systems are very easy to configure and they provide all the necessary information about how your emails are performing i.e.

  1. Who is reading your emails
  2. Which links are getting clicked
  3. Geographical data
  4. Who have marked your email as spam
  5. Hard and soft bounces, etc

Separate IP

If possible buy a separate IP for your site and accordingly choose a plan which allows you to host your site/sites on your own IP address (not shared with anyone else). This will allow you to keep the IP and your domain clean of blacklisting or spamming.