Category Archives: CalendarScripts

A/B Testing Of Email Message Subjects in Arigato Gozaimasu

The latest version of the Gozaimasu module of our premium WordPress auto-responder plugin supports A/B testing of message subjects. (This is different than the subscribe form design A/B tests. Check them too, they are great. It’s also different than split testing whole autoresponder messages.)

What Does It Do

A/B testing is a very powerful method to figure out what subjects in your messages draw more attention, email reads and clicks. Sending the same email but with different subjects to different subscribers can help you figure out what is the best subject.

Normally you would run A/B test for some time, then add or remove different subjects and at the end you will stop the test and use only the subject that works best.

How Exactly Does It Work

The A/B subjects feature picks one random subject for each email the software sends from the subjects which you have defined for the test. This is completely random. To avoid unnecessary queries and server overload it will not guarantee that each subject is picked equal number of times.

Here’s a basic example. You can create a welcome message in an autoresponder campaign. You want a catchy subject but you are not sure whether this is a good idea. So you can decide to run an A/B test between these two (or more) subjects:

Welcome to Our Newsletter!

or

Want To Learn About The Best Fitness Practices? You Came To The Right Place!

Some of your subscribers will receive the first subject, some the second. By looking at the reports: read stats, unsubscribe stats (and if you have the Intelligence module you can add trackable links for click stats), you will see which of these subjects performs better. This is simple and efficient.

How To Set It Up

The feature is available both for autoresponder messages and newsletters so let’s just see it on newsletters. On the Add/Edit Newsletter page you will see a checkbox for A/B Test next to the newsletter subject.

Clicking on it shows a new field and transfers your current subject to it. Clicking on the + sign next to it lets you add more variants, as many as you want:

screenshot of adding A/B subjects

Save the newsletter and now when it’s sent, it will pick these subjects randomly instead of the default Newsletter Subject field (empty fields will be ignored).

Absolutely the same works for autoresponder email messages.

Then in the Newsletter or Email message Reports page you will see a new link “Reports per A/B Subject” and will be able to see how many emails are sent with each subject, how many are read, the % read rate, and for Intelligence module owners also the trackable link clicks:

It’s worth noting that “read” stats rely on a tracking image. Since some email programs disable messages by default, the read number is not 100% reliable and sometimes shows slightly less than the actual number of read emails. It’s still a great way to identify which subject performs better.

Trackable link clicks will count one click per email message regardless how many links you have placed inside and how many times each reader clicked them. You want to know which subject makes people actually read the message and now if they clicked 10 links inside – counting all the clicks would defeat the purpose of the report.

In your Advanced Stats page, the Unsubscribe stats tab will also show reports based on A/B subjects for these email messages and newsletters which currently have an A/B test enabled:

Unsubscribe statistics

You may wonder why the number of emails sent total for this newsletter is higher than the sum of emails per A/B subject. This is because we were sending this newsletter for a while without running A/B tests and started using tests later.

The unsubscrbe stats per A/B subject are very useful: they can show you what subjects irritate the subscribers – probably because of looking spammy or maybe over promising (and then under delivering in the content of the email).

How To Conduct Surveys With WatuPRO

WatuPRO is a quiz plugin but it’s also perfect for running surveys. A survey is essentially a quiz that doesn’t give specific result / outcome to the user but presents the admin with the data of the user’s answers.

The most basic concepts are shown in the video and explained in more details in the post below.

Here is what you need to do to run a survey with WatuPRO:

  •  Mark the questions as “survey” questions. This is done with a checkbox on the Add / Edit Question page saying “This is a survey question. This will ensure no green / red checkmark will be shown if you decide to show user’s answers on the “final screen”. You can also use the mass-edit functions in your Manage Questions page to turn multiple questions into survey questions at once.
  • Don’t create grades. Nothing stops you to create grades but typically surveys do not grade the user’s performance.
  • View the results. You can then view, export, etc all the answers of the respondents. More information here.

The above is enough to run functional survey. However there are several tools that can make surveys created with WatuPRO even better:

The Reporting Module

See a short demo survey that uses the Reporting Module.

If you include the Reporting module with your WatuPRO purchase you’ll be able to see a lot more structured data and run data analytics on the survey responses.

The cumulative stats per question will show you how many respondents have selected each option of each question. If the question is not labeled as a survey type and has correct & incorrect states, you’ll also see this information.

stats per question

Cumulative stats per question

Follow the link to full details to get to the page that gives you detailed list of everyone’s answers, which can also be exported to CSV.

The stats per category and tag is very useful for quantitative questions because it will show cumulative points collected in each category, and again correct/incorrect percentages if the question has such characteristics:

Stats per category and tag

Stats per category and tag

The super powerful cross tabulation analysis turns WatuPRO into a real scientific tool for running insightful data analysis on any survey:

Cross tabulation analysis

Cross tabulation analysis

Learn more about cross-tabulation analysis here.

The poll-like shortcodes enabled by the Reporting module let you also present structured %-based data for every question to the user. Example here. Obviously you can use them to also run simple one-question polls.

poll shortcode example

The Likert Scale Survey Maker

If you want to run a likert scale survey we have a free helper plugin that will make your work so much easier. It makes adding a lot of survey questions at least ten time faster than via the standard way and predefines the quiz settings to be most appropriate for a survey. Don’t worry, all the questions and settings are then editable just like in any quiz made with WatuPRO.

WatuPRO 6.5

The quiz plugin WatuPRO has been updated to version 6.5 with lot of new features and bug fixes as usual. Here’s the full changelog:

  • A new shortcode allows you to display cumulative stats on all user’s logged in attempts on multiple quizzes. You can also include conditional content based on the achieved results from all the quizzes.
  • Added option to pull X random question categories in the quiz. This will work together with other randomization and pull random settings.
  • Added option to not display the optional answer explanation if the question is left unanswered.
  • The checkbox for MathJax and LaTeX problems on import was not working for Aiken imports.
  • Improved handling of WP emojis entered by mobile phones.
  • Added optional parameters show_id=0 show_delete=0 show_details=0 to the watupro-takings shortcode to allow hiding these columns.
  • Added shortcode “watupro-leaderboard-position” to return the position of the currently logged in user in the global leaderboard by points.
  • Added new columns to the WatuPRO Simple import format: “is required?” and “answer explanation”.
  • Part of the question contents can be hidden on the final screen if you place them betwneen HTML comments: <!–watupro-hide-start–> and <!–watupro-hide-end–>. The rich text editor should be switched to Text mode when entering the comments.
  • When submitting a quiz (or going to a next page) with unanswered required questions the error message will show which of the questions exactly should be answered.
  • Added name of logged in users in the email sent to admin.
  • Added 5-columns style for questions showing their answers in columns.
  • Two versions of the question compact format are now available, see the link next to the selector for more information.
  • You can set individual quiz schedules for logged in users.
  • Timer allowance for timed quizzes is now configurable from WatuPRO Settings page.
  • Added variables %%CATEGORY-WRONG-X%% and %%CATEGORY-EMPTY-X%%  for the manual category based output.
  • Changed the JS alerts on the required contact fields at start/end of a quiz to inline messages.
  • Added option to use respondent’s email address as a reply-to address when sending quiz results to admin or managers.
  • Added option to exclude the default “Details of …” line when sending quiz email to admin. The option is in WatuPRO Settings page, section Defaults.
  • Added a counter on the Manage questions and Reuse specific question page to show how many questions are selected for mass updates or reusing.
  • Added variable %%UNIQUE-ID%% for the final screen – a unique identifier of the submitted quiz attempt.
  • Added option to hide the following columns from “View details” pop-up -> table format: ID, No., Points, Result. The option is in WatuPRO Settings page -> User Settings tab.
  • From WatuPRO Settings page -> Theme and Design tab you can switch off browser’s autocomplete feature on the quiz form.
  • Added option to store contact data for logged in users if the quiz requests contact data. In this case it will be pre-filled for other quizzes or new attempts on the same quiz.
  • [Intelligence module] Added option to restart limits for quiz attempts after payment. More information at https://blog.calendarscripts.info/watupro-intelligence-module-restart-quiz-attempts-when-payment-is-made/
  • [Intelligence module] When a manager’s role is permitted to access quizzes with “Apply user group / user role category restrictions” they will see only the results of users within the same user groups they have + the results of the non-logged users. This applies only when you are using WatuPRO user groups.
  • Added shortcode attribute require_login for overwriting quiz settings.
  • [Intelligence module] Added user email address in the list of payments for a paid test.
  • [Reporting module] The shortcode watupror-poll that produces a poll chart on individual quesitons can now accept paremeter orientation=”vertical” to create a vertical bar chart.
  • [Reporting module] Added parameter draw_unanswered to the watupror-pie-chart shortcode. It will generate a pie sector for the unanswered questions in each category.
  • [Reporting module] Added paremeter sum_subcategories to watupror-pie-chart and watupror-quiz-cat-chart shortcodes. The parameter will sum the subcategory performance into the parent category and will not draw the subcategories in the charts.
  • WooCommerce integration: you can now select a WooCommerce product directly in the Edit quiz -> Intelligence module tab and let the plugin automatically set the required attributes for you. Note that only Virtual AND Downloadable products will be shown in the drop-down.
  • Improvement to the WooCommerce integration: unlinking the product will remove the custom attribute
  • Fixed issue with counting multiple-choice questions answers in “Stats per question” page when those contained commas.
  • Fixed issue with scheduled timed tests when the scheduled end time is sooner than the timer end time.
  • Fixed bug: the watupro-result shortcode will now load the WatuPRO CSS and scripts so details are properly displayed.
  • Fixed bug [Intelligence module]: when role setting to “view/approve results” with “apply user group permissions” checked, the option “Do not allow sending email with the edited results” was getting ignored.
  • Fixed bug [Intelligence module]: sometimes deselecting an item in one box of match/matrix questions was making a value that’s reserved in another box available for selection.
  • Fixed bug in flashcard questions: a card stopped flipping after 3-4 flips.
  • Fixed database bug: when users entered emojis in answers to open-end questions their results were not stored. Note there may still be issues with using emojis across the plugin: this is a bug in the built-in WordPress database handler. We are applying a conversion function on all places manually.
  • Added attribute “pagination” to the watupro shortcode to overwrite the quiz pagination setting.
  • Added option to set header and footer for PDF results printing. This feature requires PDF bridge version 1.7.
  • Removed sessions usage from the core plugin. They are now replaced with cookies to avoid loopback issues on some installations.
  • Fixed bug [Intelligence module]: no new lines in “fill the gaps” questions.
  • Fixed bug [Reporting module]: Analytics integration did not work on Ajax quizzes.
  • Fixed bug: when drop-down fields were used for asking contact details at the beginning of a quiz, they were still shown on the next pages.
  • Fixed bug: captchas should not be required when auto-submitting timed quiz with expired time.
  • Fixed bug: restricting access to specific list of users should be case-insensitive for email addresses.
  • Fixed bug in the new watupro-leaderboard-position shortcode.
  • Because of problems with WP auto paragraphs function reverted to using our own function when displaying questions. Added configuration option to use wpautop().
  • Fixed bug [Reporting module]: the pie chart per category should always show green for correct answers and red for wrong (unless colors are specified by the user)
  • Fixed bug [Intelligence module]: when the new “View and edit/approve results” setting was set for a role the user group / category restrictions were not applying.
  • Fixed issue with category paginator highlighting when using next page / previous page buttons.
  • Fixed bug: sometimes the limit for number of attempts per email address was remaining active even when we are not asking for user email address.
  • Fixed bug in “Don’t display questions that were previously answered by the user” option.
  • Fixed issue with blank space showing on top of the user menu.
  • Fixed bug: if “ask for contact details” contained a required checkbox there was no visible error message when it’s missed.
  • Fixed column titles mismatch on “Export with details” function in View Results page.
  • [Intelligence module] Fixed display issues with horizontal sortable questions on the final screen.