Sendgrid Contacts API  Examples

Sendgrid Contacts API Examples

screen-shot-2016-06-06-at-1-42-42-pm-768x294

If you are using Sendgrid to send your Marketing / Promotional emails then using the Sendgrid Contacts API can automate quite a few things for you. Here are some simple API examples to add/remove/edit recipients to the Sendgrid Contacts Database and synchronising the Sendgrid lists with the WordPress mailing lists.

There are 2 ways to synchronise your mailing lists in wordpress with the Sendgrid lists. One is using the daily/weekly cron job and the other is through real time i.e. an on demand system.

Obviously we all would prefer the real time system. However it largely depends on how you handle your subscription process in WordPress.

  • If you have created your own subscription system then you are either storing the contacts in your own table or you are using the WordPress usermeta table. API can be called by making the necessary changes in your system. Synchronisation of the mailing lists in that case can be done in real time.
  • If you are using some plugin which offers hooks on various actions like
    • user subscription,
    • user confirmation/verification,
    • user unsubscription,

    simply use this hooks to provide real time synchronization by adding necessary code in functions.php.

  • However if you are using some wordpress plugin which does not offer hooks as mentioned above then the only option is using cron job.

Here are some Sendgrid Contacts API PHP examples. They show how to synchronize receipients on your existing mailing list with Sendgrid.

Here is the official documentation on Sendgrid API

Add New Recipient – Sendgrid Contacts API

Above code adds a new receipient to Sendgrid contacts Database. It does not add if the receipient already exits but in any case does return the Recipient ID of the recipient. Also the above code does not assign any lists to the contact. To assign some Sengrid list to the contact we need to know the recipient ID of the contact which Sendgrid returns after adding a new contact to the Database. Through the above code we get the receipient ID in the $receipient_id variable. You can store this variable within your wordpress table as you would need it  to perform the Sendgrid tasks related to the recipient whenever the contact gets edited within WordPress. If you do not store this variable in your wordpress table then you would need to execute above code each time for the contact you need to get the recipient ID.

Add Receipient to a particular list – Sendgrid Contacts API

Above function requires the ID of the Sendgrid List (SENDGRID_LIST_ID) to which the recipient needs to be added.

You can get the Sendgrid List ID from the Sendgrid interface. Just click on the list and you will notice the List ID in the URL

If you have added the receipient ID received from Sendgrid to your own table in WordPress then you will get it from there else you may need to run the Add new Recipient function again to get the receipient ID.

Remove/Delete recipient from a list – Sendgrid Contacts API

This function requires Sendgrid List ID and the Recipient ID as input. It removes the receipient from a particular list. It does not delete the recipient from the Sendgrid Contacts Database.

Offsite data storage for Disaster Recovery

Offsite data storage for Disaster Recovery

2631871046_e876569317_z
Photo credit – williamhook2631871046

Offsite data storage for your website means copying your database, code, media and other files to a remote server so that in case of any disaster the server can be rebuilt using the data available on the remote server.

Copying your database and code backups and other files like images, etc to a remote/offsite server can be done for various reasons. It is mostly done for 2 reasons.

  1. Preparing for Disaster Recover where the server can be rebuild using the data from offsite data storage
  2. Redundancy (when the live server stops due to some reason, the remote server can take over)

Although below steps can work in both the cases, this particular article is written with a viewpoint of understanding how to setup an offsite data storage for Distaster Recovery.

For preparing disaster recovery the remote server should ideally follow below criteria

Backup server requirements

  1. The hosting provider should be different to your live/production server
  2. The hosting server should be in a different location to your live/production server
  3. Bandwidth may not be a crucial factor but Disk space is important so you need to plan a proper size server after studying your data and storage requirements

How to handle different file types for offsite data storage

1. Database backups

There are number of ways database can be copied/replicated on the remote server

  1. Real time: Techniques like mysql replication helps to achive real time data synchronization on live and remote server. Any updates performed on the live server will be replicated to the remote/slave server. Here is a documentation on how to set up mysql replication. For this method it is important that your database is accessible over the network.
  2. mysqldump: This is very simple way of copying the live server database to the remote server. Read how to create a script to copy database from the live server to the remote server automatically. For this method it is not required to have your database accessible over the network however the changes would not be realtime. It would depend on the cron job you set to achive this.
  3. Backup Utility: There are various 3rd party utilities. Read how Sypex Dumper (SXD) can create automatic backups for you which you can transfer to the remote server automatically on daily basis. The database can then be imported through the SXD installation on the remote server.

2. Application/code files and images

The best way to trasfer code files and images is using rsync (remote sync) command. Below rsync command runs on the origin/live server to copy the incremental changes to the remote server. This script can be set as cron job to run on daily basis.

3. Housekeeping on the remote server

If you are transferring database to the remote server on daily basis, soon your remote server disk might get full. It is important to set up some housekeeping script on the remote server.

Above script keeps last 10 days of database backup files and delates the rest. Again a cron job can be set up on the remote server to perform this automatically on daily basis.

Create staging environment using GitLab

Gitlab Webhooks

Prequisite: Please go through the steps to set up GitLab on the Production/live server before going through this tutorial.

Steps to create a staging environment using GitLab branch

1. Create a new branch for your GitLab project as shown below

1 A. Go to your project page in GitLab and click the option to add New branch

Create a Gitlab branch
Add new Gitlab branch

1 B. Create a new branch named as dev

Gitlab new branch

2. Create your staging site by copying the files and the necessary databases

3. Follow all the steps mentioned on the article Autodeployment using GitLab Webhooks on your staging sever with only the exception of Step 7 which needs to be modified slightly on the staging server. For the staging server that command would change to

Please note we are checking out only the dev branch on the staging server. So only your changes on the dev branch will be seen on this server.

4. Add the staging webhook.php file also to the list of webhooks on GitLab for Push events as shown below

GitLab Webhook

This completes setting up your staging server to preview your changes on the dev branch before they get merged to the master branch.

Now the question is how do we commit our changes?

Steps to commit your changes

    1. On your local (working) git repository first of all go on to the dev branch before you make any change
    2. Now make changes to the required files and commit your changes. Since you are on dev branch the changes will automatically get committed to the dev branch.
    3. Now push your changes to the dev server
    4. Observe the changes on the dev server
    5. Assuming the changes are ok, move on to the master branch
    6. Merge the dev branch with master
    7. Push the changes to the production/live server

Autodeployment using GitLab Webhooks

Gitlab webhooks

GitLab Webhooks

GitLab provides various Webhooks to perform automated tasks after a certain event. For e.g. Push events Gitlab webhooks will work exactly similar to post-receive hook in GIT. Only thing is it needs a webpage (mostly on your server) which gets executed after a certain commit is made to the GitLab repository. This webpage can perform tasks like updating the local mirror repository and making the corresponding change in the files in your public_html folder. Read more about webhooks in GitLab

Assuming you have a GitLab repository at: git@gitlab.com:username/some_project.git

Steps to follow on your production/live server

  1. SSH to your production/live server
  2. Generate an SSH key for auto deployment to be used by the GitLab server

    Name of the key: gitlab_rsa_deploy, keep the passphrase blank
  3. Add the contents of the gitlab_rsa_deploy.pub key to Deyloy Keys section of your Project Settings and Enable it
  4. Create a config file in .ssh folder and add below contents to the file. Add your user key added to GitLab user key settings.
  5. Create a folder name gitlab preferably outside your public_html folder
  6. Create a mirror repository by cloning your gitlab project
  7. Set Work Directory
  8. Perform git fetch once on the mirror repository so that the SSH keys get added to the server
  9. Create the webhook file on your server as webhook.php. Add below contents to the file and save this file inside your public_html folder.
  10. Add the URL of the webhook.php file to the list of webhooks on GitLab for Push events as shown below
    GitLab Webhook
  11. Now just commit some change either through GitLab or through your local repository and check if autodeployment is working

Points to consider before installing a new wordpress plugin

Installing a new plugin is very easy in wordpress. All you need to do is to search for your plugin, select it, install and activate. If you do not find it suitable to your needs, just deactivate, delete it and move on.

However not many people realise what a plugin does in the background once it gets activated and assume that once a plugin is deleted its all gone which is really not true in most cases. A deleted plugin mostly leaves quite a few traces in your system. Depending upon the plugin these traces can severely affect the performance of your system if you try too many plugins without checking what it does in the background

Many plugin developers do not follow WordPress coding standards. They do not provide an unintall function for the plugin. This means you need to manually clean up all the traces of the plugin after it is deactivated and deleted.

I am currently not experiencing any performance issue with my site

Although your site may be performing well currently, if you do not consider cleaning your system then soon a badly developed plugin can cause wordpress queries to slow down. Once the system starts slowing down it will be difficult to clean the system as by then you won’t be sure which tables are currently in use and which records in the database are unnecessary as it is quite easy to forget which plugins you tried in the past.

e.g. Some plugins developers are not quite efficient in their coding. They save each setting as a separate row in the wp_options table. Ideally all settings could be stored in a single row if setting are stored in an array. I remember a plugin which created 300 rows in the wp_options table to store settings with autoload set to yes which means the rows get selected on every Dashboard page load.

WordPress gets all the rows on each Dashboard page. If the table size grows bigger it will definately slow down the sytem.

Points to consider before installing a new wordpress plugin

First of all Install, activate and configure the plugin on your test environment or localhost

Check the source code both on Dashboard and your website: Check how many CSS and JS calls the plugin is making, whether it adds any inline CSS and other extra code.

Some plugins developers do this quite efficiently. They provide a settings screen and ask you on which Posts Types you want the plugin to run which means plugin related CSS or JS files only appear on those Post Types.

Are the queries optimised: Check if your new plugin is executing any queries which can slow down your system. To monitor such slow queries make use of Query Monitor plugin. Here are some examples how to make use of Query Monitor plugin to increase performance of your WordPress site and to detect slow queries hampering your system.

Does it create new tables: Check if the plugin creates any new tables. If yes make a note of those tables somewhere. Check what data it adds to those tables and if the tables are really necessary or the plugin developer could have used WordPress inbuit tables. This will give you an idea if the plugin developer has thought about optimising the code. Additional tables are ok but to use wordpress inbuilt tables would be ideal as wordpress has already optimised queries for those tables.

If you have a multisite, most probably the plugin would create extra tables for all your sites.

What does it add to wp_options table: Some plugin authors add _tansient records to this table. e.g. a plugin used for related links would add related links for each post in this table. So for 20k posts there would be 20k records in this table. These records get selected on every dashboard page load. This would slow down the dashboard to a great extent.

wp_options table is generally used to store plugin settings. Some plugin authors store each setting in a separate row. Ideally all settings should be stored in a single row in an arrow form.

Does the plugin provide an uninstall function: A good plugin developer provides an uninstall function which deletes the tables, capabilities, entries in wp_option table etc once the plugin is deactivated.

 

How to tackle WordPress slow queries

How to tackle WordPress slow queries

859241997_aaa015e54c_z
Photo credit – elisfanclub859241997

Here are some wordpress slow queries i.e. queries which take more than 0.05s. It really depends on your wordpress site i.e. how big is the database, plugins and your site configuration. However if you are facing performance issues related to the Dashboard then it is more likely to be due to the slow wordpress dashboard queries.

Query Monitor is good plugin to check/analyse your slow queries.

Some WordPress Slow queries

Below query auto populates the custom fields drop down box.

For large tables this query can take lot of time like 2secs or so. If you do not need custom fields it is very easy to turn them off using below function.

For more information read this interesting post on CSS Tricks

Below query runs on every Dashboard page so it is important that your wp_options table is optimised.

Depending upon the plugins you have installed, the wp_options table size can grow rapidly. Some plugins use this table to store _transient options. These _transient options are objects stored in cache. For e.g. a plugin called as Manual Related Posts stores related links per post in separate rows as _transient options. So if you have 50K posts there would be 50K rows in this table. The size of the table can also grow rapidly because each row would atleast be 1M in size.

The table structure is also not optimised properly for e.g. option_id column is defined as bigint type, autoload is set to var. It should have been set as boolean or enum. Depending upon the table size and other configurations it can even take 8secs to run above query which is quite alarming.

Few tips to optimize wp_options table

  1. Check for _transient entries and if possible replace the plugins which create lot of _transient entries. Please note that although the purpose of these entries is for caching, since this table runs crucial queries on all Dashboard pages, it defeats the purpose for large wordpress installations. The table structure also does not help the cause.
  2. If it is not possible to replace the plugins creating lot of _transient entries then use the Transient Cleaner plugin which will delete the expired transient entries and will automatically do the housekeeping for you.
  3. Change the table structure a bit. Add autoload column to the list of indexes, change the option_id to int (12)

Types of GIT repositories

GIT Repositories
GIT Repositories

Three types of GIT repositories

  1. GIT working repository
  2. GIT bare repository
  3. GIT mirror repository

GIT working repository

This repository is the one which is on your local machine. This is the respositoy which you work on i.e. add, edit or delete your application files and commit your changes.

This repository can be created using below 2 ways

  1. git init: Initialises an empty working repository. Necessary files are then added or created later. This command is mostly used when you are starting with a new application in which case the repository is empty when you begin.
  2. git clone: This command is used if you already have a working repository on your server, GitHub, GitLab, etc. This will create a Git repository on your local maching with working/application files already inside the repository.

This repository has a .git folder with all the version related files inside this folder while your application files reside outside this folder.

GIT bare repository

This repository is generally created on your server mostly for sharing or collaboration purposes. This repository does not include your application files. It only includes the version control related files. All the commits from the local repositories are made to this repository

This repository is created using below command

GIT mirror repsitory

This repository is the exact replica of the source repository. It is created for mainly 2 purposes

  1. Autodeployment
  2. Back up your GIT repository to a remote server

As shown in the above diagram, a mirror repository is created on your production server to autodeploy the commits to the working directory. In the above diagram, local commits are pushed to GitHub or GitLab repository. A webhook on these servers then updates the mirror repository on your production server and updates the files in your working directory.

Prevent your emails going to spam/junk

Photo credit 11183232994 suicidexking

Emails going to spam folder can be a sticky issue and generally there is no simple way to fix the problem. If you are already facing this issue then either your domain is blacklisted somewhere or you may just be missing a few simple tricks.

In any case a preliminary check should be to check if your sending domain or your IP is blacklisted. This would be a preliminary check because if you are already blacklisted then no matter what efforts you make to fix other things your emails will still end up going to spam / junk folder.

Check if your site is blacklisted using below tools by adding your site domain or IP address. If you are using cloudflare then try both i.e. your site domain and then your site IP address. This is because if you are using cloudflare, a domain check may not resolve to the true IP of your site.

http://mxtoolbox.com/

https://www.senderscore.org/

If your site is blacklisted you will need to request to have the mail server’s IP address delisted from any blacklists that it’s currently on.

Here’s a guide on how to get your IP delisted from a particular blacklist

If your site is not blacklisted then its half the good news for you. Below steps would fix most of the issues.

Check your email headers

All your domain emails should have SPF and DKIM records and they should be properly configured through your DNS settings. Both these records should show up as pass in your email headers.

One way to check your email headers is to send an email through your domain to one of the gmail addresses, open the email, click on the down arrow next to the “Reply” button, and select show original which will open a new tab with the email headers printed on top. On this page, if you see “signed-by: your domain name” then your DKIM signature is configured properly.

If you are on shared host, mostly these records are set up correctly and you won’t have to do anything. If they are not set up properly i.e. if they show up as fail or with some other status message you can request your hosting provider to set them up correctly. However please note that if you are on a shared host you are sharing the IP address with a few other sites. If any of the other sites/domains are blacklisted or are involved in some form of spamming it directly affects the IP and thus it affects the emails sent through your domain.

Double opt-in

Do no add any email addresses to your mailing list without verifying them i.e. always try to have a doubte opt-in method. The double opt-in confirms that the person who entered their email address actually wants to subscribe to the list. This can be easily achived by sending a verification email to the email address entered by the user.

Easy opt-out

Each email sent by you should have an unsubscribe link easily visible/prominent within the email. If users do not find this link they are more likely to get frustrated and mark your email as Spam or even Report Abuse. Since such users are never going to read your emails anyway it is better to provide them with an easily accessible Unsubscribe option.

Clear communication

It is a good policy to tell your users about the frequency of emails they will be getting from you. If for some reason you change the frequency, it is better to inform your users beforhand and to review their subscriptions. Such communciation only helps in building trust. If someone opts out due to such communication it will only help to filter your database to keep quality contacts.

Bulk mail sender

Sometimes either Gmail, Yahoo or Hotmail may block or mark all your emails as spam. It is very easy to identify this issue. If most of your Gmail users are registering but they are not verifying their accounts that means it is likely that Gmail is marking your emails as spam. This could be for number of reasons. One reason could be Gmail does not recognise you as a Bulk mail sender so when it receives lot of emails from your domain within a short period of time it thinks that you are spamming. In that case it is important to tell Gmail that you are a bulk mail sender by filling Bulk sender contact form

House keeping

Try to make some automation or regularly monitor your mailing list to keep it clean

  1. Automatically unsubscribe users whose addresses keep bouncing. Hard bounces can be removed straightaway while soft bounces can be monitored a few times.
  2. Remove users from your database who have marked youe emails as spam

Email Transaction system

It is good practice to use some form of email transactional system like Sendgrid, Mailgun, etc

Such systems are very easy to configure and they provide all the necessary information about how your emails are performing i.e.

  1. Who is reading your emails
  2. Which links are getting clicked
  3. Geographical data
  4. Who have marked your email as spam
  5. Hard and soft bounces, etc

Separate IP

If possible buy a separate IP for your site and accordingly choose a plan which allows you to host your site/sites on your own IP address (not shared with anyone else). This will allow you to keep the IP and your domain clean of blacklisting or spamming.

 

 

How to apply Sendgrid categories to WordPress emails

Sendgrid Categories

What are Sendgrid Email Categories

Creating categories for different kinds of emails sent through sendgrid is quite beneficial. It allows you to track emails based on each category i.e. it allows you to tag your emails by topics.

e.g. it would be nice to know

  • how many user registration emails actually were delivered,
  • how many people requested password resets, etc

For instance emails sent through wordpress can be categorised as

  1. User Registration
  2. Password Reset
  3. Newsletter, etc

How to add Sendgrid categories

As per the sendgrid documentation

You can add categories to the X-SMTPAPI header of the emails you send via SendGrid. This will allow you to track emails based on your own categorization system.

In case of wordpress emails here are the steps to follow

  1. Install the WordPress Sengrid plugin: Since the version 1.6.9 this plugin allows to add categories in the email headers.
  2. add the category headers to the wp_mail function as shown below

How to add category headers to the wp_mail function

The above headers can be added to your custom plugins where you define your own wp_mail function.

However in case of wordpress internal emails e.g. User Registration Confirmation email you would need to first customise that email to be able to add the required headers. For instance here is a nice link which shows how to customise User Registration Confirmation Emails

Sendgrid Category statistics

Once the required headers are added sengrid can shows statistics based on each sendgrid categories as shown below

 

Sendgrid Category statistics

 

User Role Editor plugin – Critical Security vulnerability

User role editor

If you are running User Role Editor plugin version 4.24 or older, immediately upgrade to the latest version 4.25

In version 4.24 and older the vulnerability allows any registered user to gain administrator access.

Please see more details about the vulnerability which was exposed by wordfence, a popular security plugin for wordpress.

The plugin used a function to check if a certain user has access to edit another user. But this function was not being used properly which created the vulnerability.

The author was checking if users have access to edit another user using the ‘current_user_can’ function and checking for the ‘edit_user’ (without an ‘s’ on the end) capability on a specific user ID.