Whether you've just taken up coding or you're already an expert, you've probably realized that in order for your coding efforts to go smoothly your hardware needs to be up to par. Coding can become a pretty rigorous workout for a computer quickly and larger programs can downright crash weaker computers. Laptops are especially known for having more stability issues than traditional desktop computers due to their power supply restrictions and limited cooling, but not all laptops suffer from these problems.
Sometimes when running a website you might need to run scheduled tasks at certain times. Certain software may require you to set up these tasks so they can work correctly. For example, you may need to update or backup your database on a daily or weekly basis, or send a notification email.
For these types of scheduled task, you can set up a cron job to handle them.
What is a cron job?
There's a bit of jargon to cron jobs, so let's go through exactly what they are and how they work:
The software utility cron is a time-based job scheduler in Unix-like computer operating systems. People who set up and maintain software environments use cron to schedule jobs to run periodically at fixed times, dates, or intervals.
Put simply, a program on the web server (known as a cron daemon) runs in the background and is responsible for launching your cron jobs when they are scheduled to be run.
This schedule is stored in a configuration file named
crontab where all of your tasks and their times are listed. This step-by-step guide will help you to schedule your tasks using this crontab program.
What you need to schedule a job
There are a couple of things you'll need to be able to create a cronjob for your website:
- A unix/Linux based operating system. Cron and crontab are command line functions which only work with Unix-like operating systems such as Linux. If you're using a Windows server, you won't be able to do cron jobs. Instead, you'll need to use the Windows Task Scheduler.
- Secure shell access or control panel interface for Cron. You need to be able to access your site or server via the command line with shell (SSH) access or alternatively, your web host needs to have a Cron scheduling app in their control panel dashboard.
This tutorial assumes that you have shell access to create your cron jobs. Even if your hosting company offers a tool to set these tasks for you, it's useful to lean how to do it yourself. It's very easy once you get started.
If your web hosting provider doesn't offer SSH, my cheap web hosting guide might help you find one that does.
Setting up your cron job on the server
- Choosing your schedule for the task. Before you start you'll need to work out how often your cron job task should run. Some shared web hosts have set limits on this and might only allow you to run a task once every 15 minutes or more. If you have a VPS or your own server, usually you have no limits on how frequent your tasks can be.
- Learning how to write your script for crontab. The syntax for crontab is a little bit confusing at first, but not too difficult once you understand it. The basic format of a crontab schedule has 6 fields which are separated by spaces. Here's what each field means:
minute hour day month day-of-week command-line-to-execute
Each field has an accepted value range, and they must be in that order with no empty or missing fields. The accepted values for each are:
Field Accepted value minute 0-59 hour 0-23 day 1-31 month 1-12 day-of-week 0-6 command-to-execute the command to run, and any parameters it needs
The asterisk is often used in the cron job and crontab syntax as a wildcard. It means all possible numbers for that position. For example, setting the minute as * would mean the cron job is set to run every minute.
Cron job examples
This cron job will run every minute of every hour of every day (and might not work if you run shared hosting that restrict cron jobs):
* * * * * your-command
This cron job will run at minute 0 of every hour and is a common hourly task:
0 * * * * your-command
This job will also run hourly, but at 30 minutes past the hour instead:
30 * * * * your-command
This cron job will run once per day at 5.40pm (note the use of 24 hour time):
40 17 * * * your-command
This job will run every Friday, every hour (but not on any other day of the week):
0 * * * 5 your-command
You can use multiple numbers separated with a comma. This will run at 15, 30, and 45 minutes of every hour, every day:
15,30,45 * * * * your-command
You can also the division operator. This will run 10 times every hour (every 6 minutes):
*/6 * * * * your-command
Specify a number range using the dash symbol. This will run once per hour between 01:15am and 04:15pm:
15 1-4 * * * your-command
You can also run a cron job every time the server reboots:
Editing the crontab
Once you have your script and your cron job ready, you'll need to add it to the crontab so that it can be scheduled and run on the server. To view the crontab you can use the follow command when connected to your server using SSH:
This will open up the crontab file in the vi text editor used with Unix based operating systems. It can be difficult for beginners to understand and use though. If you decide to edit the crontab this way, you may need to learn some vi commands.
To see the crontab without actually editing it you can use:
If you want to remove the contents of your crontab, you can use this command:
Creating your crontab file
The easiest way for beginners to create a cron job and add it to crontab is to use the examples above, along with your script, and create a simple text file with all of your jobs – each on a separate line. You save this file using whatever name you like and then you'll need to upload it to your server using an FTP client such as FileZilla.
To use this text file as your crontab, you can use the following command (you may need to “cd” or change directory to find where you uploaded your text file):
Doing this will overwrite all existing cron jobs without warning, so make sure you've either backed them up or included them in your text file if you need them.
Email notifications for your cron jobs
By default, cron will send any output from the script to your email address if you've specified one. If you have multiple cron jobs running and you don't change the default output, you'll receive everything the script outputs – which can be overwhelming for your email inbox.
To add email notifications to your crontab schedule, you'll need to modify it by adding
MAILTO="" to a new line. For example:
MAILTO="firstname.lastname@example.org" 0 * * * * your/cron/job.php
This will send email notifications each time this script is run, complete with outputs. For this job, that means an email every hour for each time the script is run.
Receiving error notifications
If you don't want to receive the entire output of your tasks, but you do want to be notified of errors in the script then you can redirect normal output messages into a “black hole” by using the command
> /dev/null. Anything sent here is ignored by the system, but error messages will send as normal. For example:
MAILTO="email@example.com" 0 * * * * your/cron/job.php > /dev/null
Common cron job errors
You you received any error messages when running the command
crontab my-cron-jobs.txt then you may have made one of the following errors:
- You split your time schedule and script command onto more than one line. All cron job tasks should be a single line. Make sure you turn off the Word Wrap function of any text editor you use and avoid Microsoft Word or other word processors that add formatting.
- Your crontab file didn't end with a new line. At the end of your last crontab command line, you need to press enter or return to end your file with a new line.
- Your text file was not in ASCII format. If you used Word, or added formatting this may have caused the error. Resave your file as .txt using a simple editor like Notepad then reupload the .txt using your FTP client.
Cron jobs can seem overwhelming at first, but once you've run a few of them with various scripts you'll see that they are a valuable tool for system admins and performing many functions on web sites and web applications.
Every day, hundreds of web crawlers (also known as spiders, bots or robots) will search the web and will most like crawl your website as well. It could be Google trying to index your website to show in their search results, or a spam bot trying to find email addresses to send junk mail to.
For most webmasters, it is a very good idea to control which parts of your site they are allowed to crawl and visit – and which they are not. To do this, we use a simple text file called
robots.txt in the main directory of your website (for example:
https://sitebeginner.com/robots.txt) you can advise the crawlers which pages and directories they should access or ignore. Robots.txt is purely advisory though – the search engine spiders can ignore it and still index your site. To avoid this, you can block spiders using .htaccess instead.
Why you should have a robots.txt file for your website
There are a number of very good reasons why you might want to restrict access to your site for crawlers by using the robots.txt file.
- To prevented waster server resources. Each time a crawler finds your site it will attempt to call all of your scripts in the same way a browser would. This means that your images, videos and other media will be loaded, along with search forms and other scripts like contact forms.
This can be a big drain on your server resources and it can sometimes cause your server to crash, resulting in down time for your website. It can be useful to monitor your server logs to find crawlers that slow your website down or that spider your site in large volume to block them with robots.txt.
- To save your server bandwidth. While many commercial web hosts now offer unlimited (within reason) bandwidth, you might find that allowing spiders to crawl a very large website unrestricted will cause a spike in bandwidth user that you may have to pay for with your web host. This happens when the spiders access videos and images and other large files which have to be downloaded and indexed.
Having these in folders which are blocked with the robots.txt file can reduce the bandwidth and save you an expensive hosting bill.
- To block a rogue or spam bot you don't want to access your site. It could be that a spider than has a very high crawl rate that is slowing your site down or a spam bot is trying to hit your contact form with unsolicited email spam. In these cases you can block them from accessing your site using robots.txt or .htaccess (or preferably, both).
To do this you'll need to find the name of the bot as it appears in your server log so we can use it to target and blog with robots.txt.
Setting up a robots.txt file
It's really easy to get started with a robots.txt file. It's just a plain text file that you keep in the main directory of your site. However, you'll need to be careful to make sure you don't inadvertently block something important. You can easily block your entire site from Google and see your traffic from their search disappear!
To start with, create a plain text document and save it as robots.txt. It needs to be uploaded to the root folder (
mysite.com/robots.txt and not
mysite.com/folder/robots.txt). By placing it in the root, search engines and other crawlers know exactly where to look before they begin to index your site.
Each entry in the file will have a User-agent line to identify the spider you want to instruct followed by one or more Disallow: lines to tell that crawler what to avoid.
So with that in mind let's start with a basic entry into your robots.txt:
Robots.txt example 1
User-agent: * Disallow: /
In this sample robots.txt the asterisks (*) is a wildcard which targets all User-agents. This is instructing them to not index any of your pages. Don't use this unless you want to completely block Google and all other search engines and spiders.
It's useful to use this if you are targeting a specific agent, such as a spam bot or other crawler you don't want on your site.
Robots.txt example 2
Next, you can block specific directories from being indexed by any bots. This is useful if you have admin areas, password protected areas or testing sections of your site which you don't want the search engines to see:
User-agent: * Disallow: /cgi-bin/ Disallow: /admin-area/ Disallow: /testing/test.htm
Robots.txt example 3
Being specific in your robots.txt will override a previous rule. So if, for example, you have blocked your pages from being indexed by all spiders but you do want to give Google access, you can do this:
User-agent: * Disallow: / User-agent: Googlebot Disallow: /cgi-bin/ Disallow: /testing/test.htm
Robots.txt example 4
Instead of disallowing all of your pages from being crawled, you are also able to allow all of them to be accessed by not putting anything after
Disallow:. For example:
User-agent: * Disallow: / User-agent: Googlebot Disallow:
Robots.txt example 5
Some web crawlers (like Google) also accept
Allow: as a wait to explicitly tell them which pages and directories to crawl. Here, we are disallowing all crawlers except Google:
User-agent: * Disallow: / User-agent: Googlebot Allow: /
This is the recommended way for you to set this up according to Google's FAQ page, but the robots.txt protocol doesn't officially support it – so only use it for Google at this time.
Common mistakes when using robots.txt
When you first start creating your robots.txt file, there are a few common mistakes you might make:
- Disallowing your site by mistake. It's very easy to leave the backslash on
Disallow: /which would block the specified user agent from accessing any of your content. This is particularly dangerous with search engine spiders and they may remove your site from their index. Double check to make sure you are blocking the right files for the right bots.
- It doesn't always block the bots. Robots.txt is advisory and some bots – particularly spam bots – will ignore the file and continue to crawl your site. In cases like these, use .htaccess to block those crawlers instead.
- Listing your hidden directories. Your file is available for all to see at
yoursite.com/robots.txtso don't list any files or folders you don't want people or robots to find. Listing the folders in robots.txt will not block them from being accessed directly in the browser or by a bot that ignores the file. If you need to restrict access, use .htaccess to password protect the directory.
- Placing multiple directories on one line. If you place more than one directory to disallow on a line, robots.txt will not function properly and some folders will still be crawled. Always list each directory on a new
If you run a website which gets traffic then a very good way to build on that audience to create a long-term relationship is to collect their email address. This gives you a one-to-one way of communicating with them to share new content or to promote services or products they might be interested in.
What are autoresponders?
Autoresponders are emails that are sent automatically send emails to subscribers when a certain event or action is triggered. If you want to email folks the week after they sign up to see how they're enjoying your content you can easily do that with one of the many email autoresponder services that are available.
Note: A few years ago, the term autoresponder was used to only mean those automatic replies you sometimes get when somebody is on holiday. For example:
I'm on vacation until the 5th of January. Any urgent messages can be forwarded to John Smith at firstname.lastname@example.org.
These are still technically an autoresponder, however they are more often referred to as an auto-reply since email marketing software has become commonplace.
What can an email autoresponder be used for?
There are dozens of ways that you can use triggers and events to set up an autoresponder depending on what you want to achieve with your website. A few examples of this include:
- Sending a new subscriber a link to your archived content when they subscribe to your site.
- Promote a product, service or new article that you're publishing soon based on a specific date.
- Let a customer know when a free trial they signed up for has expired.
- Remind a customer if their annual membership is due to expire or be renewed.
- Send a free report or ebook to subscribers as a thank you for joining your newsletter.
- Wish your subscriber a happy birthday each year.
- Contact customers that haven't been active for a while and invite them back with a special offer.
Chances are you may have seen some of these types of email in your own inbox. With email autoresponders you can do this yourself as well.
Free email autoresponder services
The autoresponder feature is just one of the many features included in popular email marketing software today. There are a number of free or inexpensive options available as well, which means that autoresponders and email marketing newsletters are easy for any webmaster to create themselves.
Here are a number of email marketing solutions that will allow you to set up autoresponders for your newsletter, sorted from free at the top, to cheap, and to more costly solutions at the bottom:
- MailChimp. This is an excellent place to start your email autoresponder and newsletter building. For the first 2,000 subscribers, your account is completely free and you're allowed to send up to 12,000 emails a month to your list.
It's very easy to use and set up an autoresponder campaign right away. For more advanced autoresponder options, you would need to upgrade to a paid account however if this is your first email newsletter then the free options will be more than enough. Plus, prices start from as little as $10 when you do want to get more serious.
- ReachMail. I haven't tried this piece of software myself, but it is another free option that allows you the full email marketing package, including autoresponders. Their free account allows you up to 5,000 subscribers and 15,000 emails per month – which is very good.
My personal recommendation would be MailChimp, as I've used them and enjoyed the product. However this is a good alternative if you've used MailChimp previously and want to try something different.
- Aweber. This is like the gold standard of email and autoresponder software, but there is a small cost involved. However, if you want to deliver a professional email newsletter with advanced autoresponder options, Aweber is one of the best.
Prices start at $1 for the first month and then $19 for your first 500 subscribers. It comes with a very easy to use interface and plenty of templates you can use to customize your autoresponder design. This would be my second pick after MailChimp.
- Campaign Monitor. This is another paid offering, and one I haven't used but it's very well priced at $9 per month for up to 500 subscribers. The focus of the Campaign Monitor autoresponders is a high quality design and they have some really good looking templates to choose from.
I have heard that these templates are a little more difficult to modify and design than MailChimp, but it may be a great choice if you're a designer.
A common error message that you might see when creating your website is a 500 Internal Server error. The error is particularly frustrating because it gives you no information about what specifically caused the server error. The steps in this article should help you to diagnose and find what is causing your 500 Internal Server error.
How the 500 Internal Server error appears
The internal server error message can be customized by each website that you visit, which means that the error might appear in various different forms when you experience it. A few of the common examples of how this error appears are:
"500 Internal Server Error" "HTTP 500 - Internal Server Error" "Internal Server Error" "HTTP 500 Internal Error" "500 Error" "HTTP Error 500"
If you're using Internet Explorer and get the dreaded The website cannot display the page message, this might be a 500 Internal Server Error as well. You can check this by looking for the 500 in your IE toolbar.
What causes a 500 Internal Server error
A server error can be caused by any number of things from uploading the incorrect file to as bug in a piece of code. The web server is telling you that something went wrong, but it's not sure exactly what that is.
It's a server-side error which means it might be something to do with the websites programming or configuration settings.
If the server is using Microsoft IIS software rather than Apache, there may be additional clues in the error to help you pinpoint the issue. For example, a 500.19 means the server configuration data is invalid. You can see the list of Microsoft server specific error codes here.
Fixing the 500 Internal Server error
The problem here is with the server, not your computer or browser, but it may be that there are a few issues with your connection to the server that are causing the problem. Here are few solutions that might help to fix the error:
- Reload the web page with a “hard refresh”. Sometimes the 500 error happens when multiple people visit the site at once and a simply refresh will resolve it. If you do a hard refresh you'll force the browser to connect to the server and download the latest version of the site. The keyboard shortcut for this is:
- Windows: Ctrl + F5
- Mac: Apple + R or Cmd + R
- Linux: F5
- Changing your file/folder permissions. You can login to your server using FTP and find the files or folders that are returning the error and change the file permissions to see if that fixes the problem. In general the file permissions should be set to 644 for files and 755 for folders but this may depend on any software you are using. You can make these changes in FileZilla by right-clicking the file or folder and choosing the File permissions… option.
- Check your .htaccess file. If you are running an Apache server and are using the URL rewrite engine, check to make sure it's set up correctly. Often this means the mod_rewrite needs to be enabled in your
- Increase the timeouts for the server and scripts. Sometimes the 500 Internal Server error is because the server timed out, or a script you're running timed out. You can increase the Apache server timeout in
If you're running a script, like PHP for example, you can increase the time out of this as well. For PHP this would mean changing the
max_execution_timefrom the default of 30 (seconds) to something longer:
- Clear your browser cookies and cache. This is the one of the first things and tech support will advise, and while the server errors are not caused by either cookies or the cache, it might be that the error has been fixed and you are viewing the wrong version of the site because of them. It's worth doing as it only takes a few minutes in your browser's settings.
If you still can't fix the internal server error
If you've tried to troubleshoot your 500 server error and still can't fix it, then you may need to speak to your web host and see if they are able to find what process or script is causing the error.
If you are using a commercial web hosting company their customer support should be able to provide information on how to fix the error and in many cases they will apply the fix for you at no cost.