Last month, the 14 news organizations participating in the Digital Subscriptions Accelerator gathered in Austin for their second in-person meeting. But before everyone arrived in Texas for two days of presentations and conversations, each publisher had homework to complete.

In between the Accelerator kick-off in March and the meeting in Austin, each publication had to conduct tests of their digital subscription offerings. Outlets tried everything from tweaking the payment flows to A/B testing the language on their offers.

This post will give an overview of the series of tests The Seattle Times undertook. By looking at the series of tests the Times ran, you’ll get a sense of some of the different ways publishers can work to optimize their digital subscription offers.

One-day sale

Sales are a tried and true way to generate new subscription starts. Consumers often respond to discounted prices. The Times regularly offers seasonal sales, but it had never offered a one-day flash sale.

These types of sales work well because time pressure and features such as countdown clocks or tickers with the number of people who are taking advantage of an offer can convince users to act quickly to take advantage of the offer.

Inspired by The Boston Globe sharing that it successfully offers a one-day sale each month, The Times revisited the concept that it had previously shelved. After hearing about the Globe’s approach, the Times said, “Let’s try this and see what happens,” Times VP for Marketing and Public Service Kati Erwert recalled.

The result? The sale was a “bang-up success,” she said.

It was the Times’ most successful single day of selling digital subscriptions by about 60 percent, she said.

It remains to be seen whether the new subscribers will ultimately stick around, but given the initial success, the Times is thinking about doing the sales judiciously, but with greater frequency now. Flash sales, however, often have diminishing returns so it’s important for publishers to get the timing right for their audience. Additionally, subscribers who come in through a discounted deal may be more likely to cancel their subscriptions once the prices goes up, so knowing the lifetime value of a subscriber is critical also.

The Times has seen very strong retention of those who have subscribed during sale periods, with well over 90 percent converting beyond the promotional period to full rate. And little difference in long-term retention of those acquired during sale periods when those that convert during non-sale periods.

Price boost

In an attempt to find another tool to use to boost digital subscription revenue, the Times experimented by raising the price for a digital subscription after one year. Currently, a digital subscription to the Times costs $3.99 per week. In the experiment, the price rose to $4.99 per week.

The first test the Times ran was to determine whether the price increase will affect conversions. The publisher added language to its offer page that showed its incremental price hike. Half of users received the typical offer and the other half received the test offer of $4.99 after 52 weeks.

The Times expected fewer sign ups as a result of the price hike, but it calculated that the move would be profitable if conversions fell by less than 6.25 percent.

After 40 days of testing, the test offer was actually winning by 3.6 percent, and there were possible ranges of results that ultimately showed the worst it could lose by was 6.1 percent, within the limit the Times had set.

The $4.99 price step up is now the default offer on the Times’ website, and the publisher plans to keep testing it — especially a year from now when the first price increases go into effect. The Times will need to see whether readers are willing to stick around despite the higher price.

“If we can keep the drop in conversion low, we have a lot of room to test in a year from now,” Senior Product Manager Jeff Holdridge said.

PayPal

The Times also tested whether adding PayPal as a checkout option increased subscription conversions. Its hypothesis was that PayPal functionality would help increase conversions by 5 percent due to the decreased checkout friction and additional payment option.

During the experiment, half of users saw the usual credit card-only checkout flow and half had the variation with PayPal included.

The Times isn’t quite sure why there was such a drop off with PayPal since it doesn’t have analytics or videos of how users move through the checkout flow once they move onto PayPal’s platform.

Its working hypothesis though is that the new price increase after 52 weeks is displayed more prominently on PayPal and causing users to reconsider.

The Times made some tweaks to how it mentions the price and it is now running additional tests with PayPal to see if that’s the case.

Ultimately though, there’s more to the test data. Even though the conversion rate might be lower, it still might be worthwhile to send a certain percentage of users to PayPal because the systems there make it easier to update lapsed credit cards and could be a boon to retention rates.

When analyzing experiments, it’s worth noting that conversion rate data doesn’t tell the whole story, instead, it’s worthwhile to look at other metrics that can determine whether an experiment is a success.

“It could be retention benefit,” said Curtis Huber, senior director of circulation and audience revenue. “If we look at the type of people who are converting via PayPal, overall conversion is depressed but it gives us access to subscribers who wouldn’t convert otherwise. [Conversion] Data doesn’t tell the whole story.”

The Facebook Journalism Project: Local News Subscriptions Accelerator is a pilot program designed to help news publishers build their digital subscription revenues. Funded and organized by The Facebook Journalism Project, the 3-month program includes hands-on workshops led by news industry veteran Tim Griggs, a grantmaking program organized by The Lenfest Institute for Journalism, and regular reports on best practices authored by both The Lenfest Institute and the Facebook Journalism Project.

Check out more posts about the Facebook Journalism Project program sessions here.

This article originally appeared on the Facebook Journalism Project blog.