I have been collecting Beta apps on Google Play for two years and this is what I learned.
Content:
- About TestingCatalog Research
- What I did?
- TestingCatalog Database Overview
- Best Android Beta Apps charts
- TestingCatalog Community Surveys
- Summary
I am running the biggest collection of Android apps for beta testing called TestingCatalog with a community of more than 20,000 beta testers behind. I collected some data during the last year and I want to share it with the community including some comments from my side. I hope that you will enjoy it.
What this Beta TestingCatalog Research is about
Google Play Store has the possibility to release apps in separate Alpha/Beta release channels. Access to those releases could be opened to everyone or closed (available to a limited amount of users). The app could then have a dedicated Google Plus community or a Google Group. However, the beta testing process is far from ideal. Even after the Google Play Store update with the "early access" category, there are a lot of blank spots.
- There is no place where the user can browse all apps available for beta testing. Early access tab has a limited amount of apps and doesn't have frequent updates.
- The beta testing process is not optimised in terms of feedback gathering and feedback analysis.
TestingCatalog aims to solve those problems with a set of tools and services for developers and beta testers.
What I did?
By having access to the large beta testers community I managed to gather some insights about Android app beta testing.
TestingCatalog has a database with data about applications, application updates, beta testers and different relations between them.
I started collecting Android apps two years ago and have been publishing one app per day since then. Some user actions, such as OPT-INs tracking, starts from May 2017 (when I managed to upgrade my database tier).
Application updates contain information from the What's New Play Store section and pinned posts from the related Google Plus communities. Updates parsing is automated but there were some days when they were not scraped due to different reasons.
In this post, I want to present some total numbers which represent the current state of my collection. I also want to share some statistics about TOP apps/communities in different categories.
You can find some good apps for beta testing here or take a look at some good examples of beta testing management.
Beta TestingCatalog Database Overview
These are total numbers of the analysed dataset:
- Total Amount of Beta Apps - 637 (Including Open beta apps, closed beta apps and standalone beta apps)
- Total Beta Communities in Google Plus - 271 (Including open communities, closed communities and Google Groups)
- Ended Programs - 48 (Not available for beta testing anymore)
- Total Scraped Updates - 4256 (Including What's New section from Google Play and pinned posts from Google Plus)
- Total Amount of Beta Testers - 21348 (Registered Users)
- Total Amount of Favorited Apps - 2238
- Total Amount of unique OPT-INs - 56909 (Redirects from TestingCatalog to Google Play since May 2017)
- Total Amount of Community Joins - 10146 (Redirects from TestingCatalog to Google Plus since May 2017)
It's not a huge amount of data but I think it is enough for the brief overview.
As a next step, I selected the Top 20 results from those categories:
- Top App OPT-INs
- Top Community Joins
- Top Updates on Google Play
- Top Updates on Google Plus
- Top Communities by the amount of members
- Top Rated Apps by TestingCatalog rating**
**TestingCatalog rating counts the number of users who marked the app as a favourite and amount of total updates.
Best Android Beta Apps charts.
Let's take a look at Top OPT-IN's
1398 | Google Play services beta - Google
1053 | Gboard - the Google Keyboard beta - Google
1043 | Google beta - Google
904 | Netflix Beta
790 | TestingCatalog beta - Featured App
750 | Files Go Beta: Free up space on your phone - Google
741 | Maps beta - Google
732 | WhatsApp Messenger beta - Facebook
721 | WebView Beta Channel - Google
670 | Instagram Beta & Alpha - Facebook
Google - 9, Facebook - 4, Microsoft - 0
There are no surprises here - Google Apps, Messengers, and social media apps are the most popular. It is interesting to see how different Google and Facebook apps are distributed.
Top Joins
399 | TestingCatalog beta - Featured App
362 | The Walking Dead: Road to Survival Beta
354 | WebView Beta Channel - Google
346 | Telegram Beta - Not a Community
246 | Twitter beta
228 | Google Cast Receiver Beta - Google
223 | Lawnchair Launcher beta
223 | Snapchat Beta Unofficial
218 | Smart Launcher 3 beta
205 | Nova Launcher Beta Official
Google - 2, Facebook - 1, Microsoft - 3
Launchers, tools and music apps. Microsoft becomes a big player here. Some communities are required for joining the beta and some are not. The Walking Dead game has a closed beta community where people have to wait for several months before joining. There are different factors influencing Join numbers such as SEO, internal rating and date of publishing but most of these apps are more than one year old.
Google Play What's New updates
49 | Signal Private Messenger beta
38 | Facer for Android Wear Beta Test
38 | WordPress for Android Beta
35 | Action Launcher
33 | Chrooma Keyboard
33 | Today Weather - Forecast
32 | Minecraft: Pocket Edition beta
32 | The New York Times Android Beta Program
30 | Slack for Android Beta Test
30 | Microsoft Launcher Beta (Arrow) - Microsoft
Google - 0, Facebook - 0, Microsoft - 1
This is a count of What's New section changes. It means that if app updates without changing What's New, it doesn't count. Changelogs are an important part of the beta testing process. Those apps have frequent updates and they maintain changelogs properly.
Google Plus pinned posts updates
55 | AppSheet User Community
47 | Nova Launcher Beta Official
45 | BlackPlayer
34 | TextNow Android Beta Testers
33 | The Walking Dead: Road to Survival Beta
32 | All-In-One Toolbox Beta Test Group
31 | Flick Launcher beta
28 | Podcast Player - Free beta
25 | Numbers Game - Numberama
23 | COL Reminder Beta Test
Google - 0, Facebook - 0, Microsoft - 2
Google Plus pinned post mostly used for community updates and changelogs communication. This number represents how often a pinned post was changed. Community rules, Announcements, Changelogs, Promotions - everything goes here. At some point, it shows how often the development team communicates with users.
Google Plus Members
353344 | CM Launcher
256502 | CM Locker Beta
162763 | Clean Master Beta
140218 | Office Lens Android Preview - Microsoft
136759 | Adblock Browser for Android Beta
127003 | Nova Launcher Beta Official
117619 | CCleaner for Android Beta
84782 | Office Insider for Android - Microsoft
84418 | Microsoft Launcher Beta, Arrow - Microsoft
61154 | OneDrive Preview for Android - Microsoft
Google - 0, Facebook - 0, Microsoft - 2
Not all communities are equal. Some apps are promoting their beta communities through their apps, like CM and Microsoft. Some don't, however, and those numbers don't mean that all members are active beta testers but it definitely affects the amount of received feedback (Including spam, unfortunately).
TestingCatalog rating
177 | Snapchat Beta
138 | Nova Launcher Beta Official
123 | The Walking Dead: Road to Survival Beta
102 | All-In-One Toolbox Beta Test Group
102 | Action Launcher
100 | Talon Beta
98 | Pulse SMS+Tablet Messenger beta
96 | BlackPlayer
96 | AppSheet User Community
86 | Google beta - Google
Google - 2, Facebook - 1, Microsoft - 2
As I mentioned - the TestingCatalog rating counts the number of users who marked the app as a favourite and amount of total updates. Google and Facebook apps have a high rating because they are popular and Microsoft apps have a high rating because they have better communication.
I decided to compare Google, Facebook and Microsoft apps to see the difference in beta testing approaches. There are also some Amazon beta apps there but they were published not so long time ago and have less amount of data. Other apps (Except maybe CM, which I don't want to count) cannot be combined in the same way.
Overall score:
Google - 13, Facebook - 6, Microsoft - 10
This shows how big companies do beta testing. Let me explain the difference.
Most of their apps are pre-installed and they are the most popular compared to other apps. With a huge user base, they are mostly using the Open Beta Testing approach. Apart from that, they have several beta communities in Google Plus (for Web View and for Cast Receiver). It is also known that some beta programs had a limited amount of spots for some time (Gboard beta program was full after the launch until they extended the limit). For the most of their apps, Google release updates in stages (by country or by device group or by any other factor). This is one of the reasons why they are not communicating changelogs to users very often. So, the What' New section doesn't have frequent updates because of this. I can imagine that the amount of feedback they receive in Google Play (and the amount of spam) is enough for debugging and analyzing possible issues. If the app has more than 1kk installs, open beta might be a good way to get some additional insights.
Here is A list of all Google apps in Beta.
They have very popular apps and their approach is very close to what Google does. Many of them are also pre-installed. Open beta programs, stages releases without frequent changelogs updates. No one of their beta programs uses a Google Plus community. Messenger app has a Facebook Group and Instagram Alpha uses Google Group (for the access management and to distinguish alpha testers from beta testers). Facebook has fewer apps but, in fact, those apps are super popular and their beta testing approach is based on that.
Getting Beta releases is easy and this is how you can join Instagram Alpha
Microsoft
This is more interesting. After abandoning the WP platform they switched their focus to the Android platform. They extended the number of their Android apps (for example they bought Arrow Launched some time ago). I cannot say that MS apps are extremely popular because they are mostly Windows users-oriented and they are trying to provide better MS products support on the Android platform. Some of their apps have an Open Beta program (like Edge Browser) while most of them have a dedicated beta testing Community in Google Plus. They provide clear Changelogs and updates in Google Plus communities. This approach is different and it brings more transparency to the beta testing process. How many of you are suffering from not knowing what are you testing in a new update?
Some other apps in those charts are popular and good, some are not so good, but you always can take a look at those which are good for beta testing. Usually a dedicated Google Plus community and frequent changelogs are factors of the successful beta testing program.
Beta TestingCatalog Community Surveys
Apart from this, I was also doing many different polls in beta communities. Here I chose the 7 most interesting surveys.
Which Android version do you have? 125 votes
- 8 - 34%
- 7 - 54%
- 6 - 10%
- 5 - 2%
- 4 or lower - 2%
This is just an average distribution of Android versions across the community. This data is from December so I would assume that this distribution will be approximately valid after 4 - 6 month from the public release of the new Android version. (Oreo was publicly released on 21 of August 2017).
How many beta apps do you have on your phone? 124 votes
- Zero - 6%
- 1-5 - 9%
- 6-10 - 32%
- 10-30 - 19%
- 30+ - 14%
This also depends on the amount of installed apps (because it varies a lot). Most testers have a beta version of popular apps such as Google and Facebook apps.
How do you report issues? 71 vote
- Via Google Play - 39%
- Via Google Plus - 17%
- Via TestingCatalog - 4%
- Via In-app Form - 31%
- Don't report Issues - 8%
This is related to the low amount of Google Plus communities. Google Play and In-app forms are the most used communication channels. I cannot say that it is a good part because Google Play has a char limit and no formatting, for example. This leads to the high amount of spam and useless reports.
Did you face any problems with beta testing? 99 votes
- Yes, as a tester - 52%
- Yes, as a developer - 15%
- No, as a tester - 26%
- No, as a developer - 0%
- Not sure - 7%
This is a very small amount of voters but it is not a good sign when all developers have problems with beta testing. The current state of beta testing requires a lot of changes and optimizations.
As an Android user, what are you interested in? 106 votes
- I want to test apps - 73%
- I want to develop apps - 9%
- I want to review apps - 2%
- I only want to use apps - 8%
- I just want to click - 8%
This is just an overview of how many users want to do testing. How many of them reported issues? I would bet on less than 5%. To make this happen, you should have a match between a good app and target user. It also means that the reporting channel should be easy to find and there should be some kind of reporting culture built in the app community. The lack of common standards, best practices, and tools, makes this a difficult achievement.
Do you know how to report bugs properly? 70 votes
- Yes, steps, exp/act results, etc.. - 43%
- Yes, with my own words - 21%
- No, I am using my own words - 14%
- No, I don't know - 13%
- Just "Nothing works" - 9%
Beta testing is very close to the Software testing in general but the difference between them is also huge. This is also a result of a non-optimised workflow.
Which kind of content is the most interesting for you? 119 votes
- The list of bugs and issues - 27%
- Different testing approaches - 15%
- Features review - 35%
- App quality rating - 18%
- Something else - 5%
This is very specific to the what TestingCatalog is doing (test reports and reviews) but it also shows what kind of content is interesting to beta testers. User engagement is an important part of beta testing. This is one of the reasons why beta communities might be a good tool for beta managers.
Beta Apps Research Summary
The goal of every beta testing program is collecting the data. There are no exceptions. You have to fight for the users attention to make him/her install your app but you will have to fight a lot more for their attention if you want to get a valuable feedback. Getting testers to install the app is usually a problem of new apps while feedback collection is problematic for everyone.
People love to try new features early and they like to receive a response to their feedback. The more detailed your changelog/update/survey is, the more detailed/focused feedback you will receive. At some point, beta testing is also a marketing activity.
From the beta testers perspective, I would try to take a look at the apps who pay attention to users feedback. There are many of them and they are looking hard for your attention.
Happy Testing!
P.S. If you found any mistakes in the text, please let me know. I spent a lot of time to make this post readable but I could miss something.
