Part 2: SaaS Tools vs CMS tools in Website Optimization

In a previous post we established that "web optimization is the process of systematically improving your website’s performance by experimenting with content variations and target audiences, while monitoring and measuring the results to determine which changes will lead to an increase in engagement, clicks, views, downloads, purchases, or other type of action". We talked about two web optimization tool types:

1. Software as Service (SaaS) Solutions: Centrally hosted software, licensed on a subscription basis and accessed via web browser. Examples: Optimizely, Monetate, Qubit, Visual Website Optimizer, ion, Marketizator, Personyze, Adobe Target.
2. CMS Optimization Tools: Built in features or add-ons integrated to the CMS. Examples: Sitecore, Episerver, Adobe Experience Manager.

At the end, we left an open question: how do you determine which is the right tool type for satisfying your business needs?

First, you must determine what are your business requirements. Identify goals, document use cases, prioritize your demands. Then, consider the options while comparing and contrasting pros and cons. Here are some questions you should ask yourself when selecting one approach or the other:

What features do you require?

The most important website optimization features found in both SaaS and CMS tools are those for testing (A/B, multivariate, split URL, funnel), personalization and reporting. When evaluating your options, you should prioritize these capabilities first, and then you might compare others such as: visual editor, heatmaps, surveys, and reviews. There are no limits for the capabilities you can have for a CMS if you have the resources to implement them. Here are some hints of what to look at.

A/B Testing

A tool that supports A/B testing will allow you to compare two or more versions of a web page against one another to determine which one performs best according to data and statistics. In order to carry out these comparison tests, the tool will split traffic between the different versions. Some SaaS solutions began as solely A/B testing tools, and then evolved to include personalization. Others began the other way around, starting as personalization tools and then adding the testing functionality later on. This means that some SaaS products have features that are less developed than others. Therefore, you should consider the maturity of each feature compared to the relative priority of your needs.

Multivariate Testing

You implement multivariate testing to determine which combination of variations performs the best out of all of the possible combinations. For CMSs to perform multivariate testing you need to manually create every possible combination of changes. SaaS tools where this functionality is available combine all the changes you make and generate resulting designs automatically.

Split URL Testing

Split URL testing allows you to distribute traffic to different URLs and is recommended for testing major design changes. You have two landing pages for your website www.example.com, hosted on example.com/a and example.com/b. The test consist on dividing your traffic among the two pages and determining which of them converts better. This capability is available only on some SaaS tools (e.g. VWO) and CMSs (e.g. Sitecore), if you think this will be require then you must make sure it is supported.

Funnel Testing

Multi-page (also known as "funnel") testing is similar to A/B testing except that rather than making variations to a single page, you make changes that are implemented consistently over several pages. Like A/B testing, site visitors of a multi-page test are bucketed into one version of a funnel or the other. By tracking the way these visitors interact with the different pages they are shown, you can determine which sequence of pages is most effective. Like split URL testing, this feature is not found in all CMS tools or SaaS tools. If it is a requirement for your business to be able to define conversion paths and track the funnel performance then look for vendors for this specific capability.

Personalization

When implementing personalization you must first segment visitors, and second deliver individual messages to the defined audiences. The criteria for defining audiences vary from tool to tool. On SaaS tools some criteria might not be available to all subscription plans. For example, the "geographic location" criterion is available in some SaaS tools only for customers who enroll on a premium plan (the most expensive one). On CMSs the criterion for creating audiences doesn't vary based on subscription type, and it is possible to implement additional criteria if required.

If you need to use authenticated users' profile data as criteria for segmenting audiences, this is more of a challenge for SaaS tools. Typically, that information will be stored in a CMS along with the rest of the content. With a SaaS tool, the use of profile attributes (if available) requires uploading a file with your users' profile information. That means your privacy policy must allow you to share that data with a third party. This is not a one time concern since your site continuously adds and removes users, so you probably need to implement a batch job to upload your user data to the SaaS cloud. The data will also not likely be as real-time as it would be if you got it directly from your web CMS.

An interesting feature for audiences targeting - available in SaaS tools but not in CMS tools - is the integration with external data management tools like Oracle BlueKai. This external data management system will enable you to perform behavioral marketing. What this means is that the content that users see on your website is based on their recent browsing history.

Analytics and reporting

The reports generated from the collected analytics will tell you how your content versions are performing against the tracked goals. This is an important capability to look at since these insights are critical for making business decisions. In general, there is a larger number of metrics used for measuring personalization and testing performance in SaaS than for CMS. Additionally, the capability of alerting the user when a test version is statistically better than the original is available on some SaaS tools but not in CMS tools.

Visual Editor

Using the SaaS tool visual editor, you can modify any HTML tag without needing to code. However, more advanced users can still edit the source code if they wish. This visual editor is exclusive to SaaS tools and something that will empower marketers to easily and quickly test their hypotheses.

Heatmaps

Heatmaps are extremely useful for evaluating the usability of your website, because they show where visitors are scrolling and clicking and what areas of a site they are ignoring. This feature is also exclusive to some website optimization SaaS tools. There are even some SaaS tools on the market that focus only on heatmaps.

Surveys and Website Reviews

Want to ask your customers for their opinions? On-page surveys helps you uncover the intentions, motivations, and apprehensions of your visitors through pop-up surveys on your website. Asking for reviews will allow you to collect feedback and improvement ideas for your website.

Can you manage content in multiple systems?

When you use a SaaS tool you will need to create all the content variations (copy, images) for your experiments on an external system. If you use a CMS, all the content is stored in the same place. In other words, the only way to have a single content repository is to use a CMS with personalization capabilities to optimize your website.

Are you concerned about the data ownership?

Using a SaaS tool necessarily entails third party involvement. The content and the resulting analytics will be stored on the provider'd servers. If the data ownership is a concern for you, then a SaaS tool won't be the right choice.

Do you have clear personalization goals?

Implementing personalization on a CMS requires development effort, so you must know upfront what sections/components of the website can be personalized. SaaS tools are easy to set up and allow you to quickly start running experiments. If you don't have clear personalization goals at the time of site design/development and need to first experiment to get a clearer vision, SaaS tools can help with that. 

What website sections must be optimized?

Because SaaS solutions use the website published HTML as the content source, you will be able to edit any tag on your website pages without previous planning or development effort. However, that also means an inability to test website sections generated dynamically. For example a div with an id value - which is the concatenation of a scope variable - and a string cannot be personalized if the SaaS solution can't rely on its being the same for every request. It won't break, but your experiment won't work and your results will be unreliable.

In a CMS you will only be able to personalize content within components developed to support personalization features. You need to know upfront which components of your website will be objects of personalization when building it, or you will need a developer to modify them later. Without proper planning and development you will not be able to test or personalize all aspects of the page.

How much time do you have?

Usually you will need things done "yesterday." Using a SaaS tool you can be up and running quickly. You just need to sign up for the service and insert a code snippet in your website's head tag to start experimenting with content variations and tracking the results. It will be quicker to set up target audiences and content experiments with a SaaS tool than with a CMS tool.

Is content maintenance a concern for you?

A downside of a SaaS tool is that any changes to HTML tag names or an id will cause an experiment failure. If that is the case, you will need to redefine previously set experiments.

If you plan to migrate from one CMS to another, having your website personalization set in a SaaS solution means you don't lose anything, as long as your website's HTML isn’t modified.  Alternatively, if you build a ton of logic/experiments into your SaaS tool and then decide to move to another tool, all that content, logic, and analytics will be lost.

How is your website load time affected?

One of the first questions people ask when they understand how SaaS tools work is: "Won't that slow down pages rendering for my users?" When using a SaaS tool, the processing of the web optimization is offloaded to the SaaS service provider's server and the visitor's browser. This means less load on your own servers; however, if their servers or the network are clogged, this could slow down the end results for your users. Most SaaS solutions have addressed this by pushing things out to CDNs and by having some undetectable delays placed into the logic that allow the content to get down to the browser from the SaaS service prior and render on the visitor's browser. Some even allow you to host your own copies of their JS files, thereby reducing the requests to their servers and making you more in control of a user's final experience. However, some SaaS solutions have still struggled with flickering issues for the end user in the past, so it is something you should definitely investigate before signing a long-term contract with a SaaS provider.

With a CMS solution, flickering is not a problem since the markup is generated and sent from the server itself to the client without any redrawing on the client side. Of course the CMS architecture must make good use of caching and CDN to guarantee good performance and low page load times for the end user.

What’s the cost?

Since SaaS Solutions are licensed on subscription basis, it is cheaper to get access to these kinds of tools compared to the upfront cost of a CMS with an integrated marketing automation platform. CMSs with marketing automation capabilities are considered high-end, so they are at the top of the price spectrum for Web CMS's. Most SaaS solutions will offer you a subscription to a monthly or yearly plan. A plan often limits you to a certain number of page views, domains, account users, and experiments, and you must pay for additional access if your usage increases. Also, cheaper plans usually exclude features available in the premium plans.Some SaaS tools offer a free trial period if you want to test out different tools. Ultimately, when purchasing a tool, we recommend you always get real quotes from vendors to compare and decide.

Final Considerations

If you have a website that isn’t managed in a CMS, already have a CMS but it doesn’t provide you with website optimization features, or can’t afford a CMS with such capabilities, SaaS solutions can be a good fit for you.

If you are thinking about getting a CMS you could consider the platforms that do not provide optimization tools and instead use a SaaS optimization tool if you:

• Can manage content in multiple systems.
• Don’t have clear personalization goals or need to experiment to have a clearer vision.
• Require the ability to personalize any HTML tag.
• Must set quick and easy tests to get quick results.
• Need better reporting capabilities.

You would use CMS marketing automation if you:

• Want to have a single content repository.
• Are concerned about data ownership.
• Have clear personalization goals at the time of site design/development.
• Must define personas only in one system (possibly for access from other internal systems.)

Having said these things, there is nothing stopping you from using SaaS tools and a CMS’s marketing automation tools together to leverage both to fulfill your marketing needs. In our next post we will discuss how to approach that strategy.