I have been collecting Beta apps on Google Play for two years and this is what I learned.
- About TestingCatalog Research
- What I did?
- TestingCatalog Database Overview
- Best Android Beta Apps charts
- TestingCatalog Community Surveys
I am running the biggest collection of Android apps for beta testing called TestingCatalog with a community of more than 20,000 beta testers behind.
I collected some data during the last year and I want to share it with the community including some comments from my side. I hope that you will enjoy it.
What this Beta TestingCatalog Research is about.
Google Play Store has the possibility to release apps in separate Alpha/Beta release channels. Access to those releases could be opened to everyone or closed (available to a limited amount of users). The app could then have a dedicated Google Plus community or a Google Group. However, the beta testing process is far from ideal. Even after the Google Play Store update with the "early access" category, there are a lot of blank spots.
- There is no place where the user can browse all apps available for beta testing. Early access tab has a limited amount of apps and doesn't have frequent updates.
- The beta testing process is not optimised in terms of feedback gathering and feedback analysis.
TestingCatalog aims to solve those problems with a set of tools and services for developers and beta testers.
What I did?
By having access to the large beta testers community I managed to gather some insights about Android app beta testing.
TestingCatalog has a database with data about applications, application updates, beta testers and different relations between them.
I started collecting Android apps two years ago and have been publishing one app per day since then. Some user actions, such as OPT-INs tracking, starts from May 2017 (when I managed to upgrade my database tier).
Application updates contain information from the What's New Play Store section and pinned posts from the related Google Plus communities. Updates parsing is automated but there were some days when they were not scraped due to different reasons.
In this post, I want to present some total numbers which represent the current state of my collection. I also want to share some statistics about TOP apps/communities in different categories.
You can find some good apps for beta testing here or take a look at some good examples of beta testing management.
Beta TestingCatalog Database Overview.
These are total numbers of the analysed dataset:
- Total Amount of Beta Apps - 637 (Including Open beta apps, closed beta apps and standalone beta apps)
- Total Beta Communities in Google Plus - 271 (Including open communities, closed communities and Google Groups)
- Ended Programs - 48 (Not available for beta testing anymore)
- Total Scraped Updates - 4256 (Including What's New section from Google Play and pinned posts from Google Plus)
- Total Amount of Beta Testers - 21348 (Registered Users)
- Total Amount of Favorited Apps - 2238
- Total Amount of unique OPT-INs - 56909 (Redirects from TestingCatalog to Google Play since May 2017)
- Total Amount of Community Joins - 10146 (Redirects from TestingCatalog to Google Plus since May 2017)
It's not a huge amount of data but I think it is enough for the brief overview.
As a next step, I selected the Top 20 results from those categories:
- Top App OPT-INs
- Top Community Joins
- Top Updates on Google Play
- Top Updates on Google Plus
- Top Communities by amount of members
- Top Rated Apps by TestingCatalog rating**
**TestingCatalog rating counts the number of users who marked the app as a favourite and amount of total updates.
Best Android Beta Apps charts.
Let's take a look at Top OPT-IN's
- 1398 | Google Play services beta - Google
- 1053 | Gboard - the Google Keyboard beta - Google
- 1043 | Google beta - Google
- 904 | Netflix Beta
- 790 | TestingCatalog beta - Featured App
- 750 | Files Go Beta: Free up space on your phone - Google
- 741 | Maps beta - Google
- 732 | WhatsApp Messenger beta - Facebook
- 721 | WebView Beta Channel - Google
- 670 | Instagram Beta & Alpha - Facebook
- 611 | Facebook beta - Facebook
- 581 | YouTube Go beta - Google
- 516 | Messenger beta - Facebook
- 447 | Chrome Beta - Google
- 443 | Google Cast Receiver Beta - Google
- 426 | Nova Launcher Beta Official
- 421 | Twitter beta
- 417 | Telegram Beta
- 413 | Snapchat Beta
- 404 | Amazon Music Beta
Google - 9, Facebook - 4, Microsoft - 0
There are no surprises here - Google Apps, Messengers and social media apps are the most popular. It is interesting to see how different Google and Facebook apps are distributed.
- 399 | TestingCatalog beta - Featured App
- 362 | The Walking Dead: Road to Survival Beta
- 354 | WebView Beta Channel - Google
- 346 | Telegram Beta - Not a Community
- 246 | Twitter beta
- 228 | Google Cast Receiver Beta - Google
- 223 | Lawnchair Launcher beta
- 223 | Snapchat Beta Unofficial
- 218 | Smart Launcher 3 beta
- 205 | Nova Launcher Beta Official
- 195 | Microsoft Edge Preview - Microsoft
- 165 | Instagram Beta & Alpha - Facebook, A Google Group
- 136 | ZEDGE Beta
- 136 | SHAREit - Transfer & Share beta
- 134 | SoundCloud Android Public Beta Group
- 133 | Retro Music Player Beta
- 130 | Flamingo for Twitter beta
- 128 | OneDrive Preview for Android - Microsoft
- 124 | Solar system beta
- 121 | Office Insider for Android - Microsoft
Google - 2, Facebook - 1, Microsoft - 3
Launchers, tools and music apps. Microsoft becomes a big player here. Some communities are required for joining the beta and some are not. The Walking Dead game has a closed beta community where people have to wait for several months before joining. There are different factors influencing Join numbers such as SEO, internal rating and date of publishing but most of these apps are more than one year old.
Google Play What's New updates
- 49 | Signal Private Messenger beta
- 38 | Facer for Android Wear Beta Test
- 38 | WordPress for Android Beta
- 35 | Action Launcher
- 33 | Chrooma Keyboard
- 33 | Today Weather - Forecast
- 32 | Minecraft: Pocket Edition beta
- 32 | The New York Times Android Beta Program
- 30 | Slack for Android Beta Test
- 30 | Microsoft Launcher Beta (Arrow) - Microsoft
- 28 | CastBox - Free Audio beta
- 28 | Pulse SMS+Tablet Messenger beta
- 28 | Talon Beta
- 27 | Relay For Reddit Beta
- 26 | Anchor beta
- 25 | Typany Keyboard Beta
- 24 | Fella for Facebook: 1 MB only Beta
- 24 | Sync for reddit
- 23 | Fleksy Beta
- 22 | Audials Android Beta Testers
Google - 0, Facebook - 0, Microsoft - 1
This is a count of What's New section changes. It means that if app updates without changing What's New, it doesn't count. Changelogs are an important part of the beta testing process. Those apps have frequent updates and they maintain changelogs properly.
Google Plus pinned posts updates
- 55 | AppSheet User Community
- 47 | Nova Launcher Beta Official
- 45 | BlackPlayer
- 34 | TextNow Android Beta Testers
- 33 | The Walking Dead: Road to Survival Beta
- 32 | All-In-One Toolbox Beta Test Group
- 31 | Flick Launcher beta
- 28 | Podcast Player - Free beta
- 25 | Numbers Game - Numberama
- 23 | COL Reminder Beta Test
- 23 | Infinite Painter beta
- 21 | Autobeat Player
- 21 | SoundCloud Android Public Beta Group
- 20 | TuneIn Beta Testing
- 20 | drupers
- 19 | Shuttle Beta Testers
- 18 | Facer for Android Wear Beta Test
- 18 | Office Insider for Android - Microsoft
- 18 | Pocket Android Beta
- 17 | Microsoft Launcher Beta, Arrow - Microsoft
Google - 0, Facebook - 0, Microsoft - 2
Google Plus pinned post mostly used for community updates and changelogs communication. This number represents how often a pinned post was changed. Community rules, Announcements, Changelogs, Promotions - everything goes here. At some point, it shows how often development team communicates with users.
Google Plus Members
- 353344 | CM Launcher
- 256502 | CM Locker Beta
- 162763 | Clean Master Beta
- 140218 | Office Lens Android Preview - Microsoft
- 136759 | Adblock Browser for Android Beta
- 127003 | Nova Launcher Beta Official
- 117619 | CCleaner for Android Beta
- 84782 | Office Insider for Android - Microsoft
- 84418 | Microsoft Launcher Beta, Arrow - Microsoft
- 61154 | OneDrive Preview for Android - Microsoft
- 61009 | Fleksy Beta
- 57316 | Beautiful Widgets Beta
- 54568 | AutoMate for Android Beta
- 51574 | All-In-One Toolbox Beta Test Group
- 42964 | drupers
- 38865 | Snapchat Beta
- 37797 | n7player BETA
- 36097 | Remote Control Collection Beta Android
- 34441 | Dolphin Browser Beta Testers
- 34096 | Kingsoft Office Beta
Google - 0, Facebook - 0, Microsoft - 2
Not all communities are equal. Some apps are promoting their beta communities through their apps, like CM and Microsoft. Some don't, however, and those numbers don't mean that all members are active beta testers but it definitely affects the amount of received feedback (Including spam, unfortunately).
- 177 | Snapchat Beta
- 138 | Nova Launcher Beta Official
- 123 | The Walking Dead: Road to Survival Beta
- 102 | All-In-One Toolbox Beta Test Group
- 102 | Action Launcher
- 100 | Talon Beta
- 98 | Pulse SMS+Tablet Messenger beta
- 96 | BlackPlayer
- 96 | AppSheet User Community
- 86 | Google beta - Google
- 81 | Pocket Android Beta
- 79 | Facer for Android Wear Beta Test
- 74 | Microsoft Launcher Beta, Arrow - Microsoft
- 74 | hike-android-beta
- 72 | TestingCatalog beta
- 72 | COL Reminder Beta Test
- 71 | WebView Beta Channel - Google
- 71 | WhatsApp Messenger beta - Facebook
- 70 | Chrooma Keyboard
- 70 | Office Insider for Android - Microsoft
Google - 2, Facebook - 1, Microsoft - 2
As I mentioned - the TestingCatalog rating counts the number of users who marked the app as a favourite and amount of total updates. Google and Facebook apps have a high rating because they are popular and Microsoft apps have a high rating because they have better communication.
I decided to compare Google, Facebook and Microsoft apps to see the difference in beta testing approaches. There are also some Amazon beta apps there but they were published not so long time ago and have less amount of data. Other apps (Except maybe CM, which I don't want to count) cannot be combined in the same way.
Google - 13, Facebook - 6, Microsoft - 10
This shows how big companies do beta testing. Let me explain the difference.
Most of their apps are pre-installed and they are the most popular compared to other apps. With a huge user base, they are mostly using the Open Beta Testing approach. Apart from that, they have several beta communities in Google Plus (for Web View and for Cast Receiver). It is also known that some beta programs had a limited amount of spots for some time (Gboard beta program was full after the launch until they extended the limit). For the most of their apps, Google release updates in stages (by country or by device group or by any other factor). This is one of the reasons why they are not communicating changelogs to users very often. So, the What' New section doesn't have frequent updates because of this. I can imagine that the amount of feedback they receive in Google Play (and the amount of spam) is enough for debugging and analysing possible issues. If the app has more than 1kk installs, open beta might be a good way to get some additional insights.
Here is A list of all Google apps in Beta.
They have very popular apps and their approach is very close to what Google does. Many of them are also pre-installed. Open beta programs, stages releases without frequent changelogs updates. No one of their beta program uses a Google Plus community. Messenger app has a Facebook Group and Instagram Alpha uses Google Group (for the access management and to distinguish alpha testers from beta testers). Facebook has less apps but, in fact, those apps are super popular and their beta testing approach is based on that.
Getting Beta releases is easy and this is how you can join Instagram Alpha
This is more interesting. After abandoning the WP platform they switched their focus to the Android platform. They extended the number of their Android apps (for example they bought Arrow Launched some time ago). I cannot say that MS apps are extremely popular because they are mostly Windows users-oriented and they are trying to provide better MS products support on the Android platform. Some of their apps have an Open Beta program (like Edge Browser) while most of them have a dedicated beta testing Community in Google Plus. They provide clear Changelogs and updates in Google Plus communities. This approach is different and it brings more transparency to the beta testing process. How many of you are suffering from not knowing what are you testing in a new update?
Some other apps in those charts are popular and good, some are not so good, but you always can take a look at those which are good for beta testing. Usually a dedicated Google Plus community and frequent changelogs are factors of the successful beta testing program.
Beta TestingCatalog Community Surveys.
Apart from this, I was also doing many different polls in beta communities. Here I chose the 7 most interesting surveys.
Which Android version do you have? 125 votes
- 8 - 34%
- 7 - 54%
- 6 - 10%
- 5 - 2%
- 4 or lower - 2%
This is just an average distribution of Android versions across the community. This data is from December so I would assume that this distribution will be approximately valid after 4 - 6 month from the public release of the new Android version. (Oreo was publicly released on 21 of August 2017).
How many beta apps do you have on your phone? 124 votes
- Zero - 6%
- 1-5 - 9%
- 6-10 - 32%
- 10-30 - 19%
- 30+ - 14%
This also depends on the amount of installed apps (because it varies a lot). Most testers have a beta version of popular apps such as Google and Facebook apps.
How do you report issues? 71 vote
- Via Google Play - 39%
- Via Google Plus - 17%
- Via TestingCatalog - 4%
- Via In-app Form - 31%
- Don't report Issues - 8%
This is related to the low amount of Google Plus communities. Google Play and In-app forms are the most used communication channels. I cannot say that it is a good part because Google Play has a char limit and no formatting, for example. This leads to the high amount of spam and useless reports.
Did you face any problems with beta testing? 99 votes
- Yes, as a tester - 52%
- Yes, as a developer - 15%
- No, as a tester - 26%
- No, as a developer - 0%
- Not sure - 7%
This is a very small amount of voters but it is not a good sign when all developers have problems with beta testing. The current state of beta testing requires a lot of changes and optimisations.
As an Android user, what are you interested about? 106 votes
- I want to test apps - 73%
- I want to develop apps - 9%
- I want to review apps - 2%
- I only want to use apps - 8%
- I just want to click - 8%
This is just an overview of how many users want to do testing. How many of them reported issues? I would bet on less than 5%. To make this happen, you should have a match between a good app and target user. It also means that the reporting channel should be easy to find and there should be some kind of reporting culture built in the app community. The lack of common standards, best practices and tools, makes this a difficult achievement.
Do you know how to report bugs properly? 70 votes
- Yes, steps, exp/act results, etc.. - 43%
- Yes, with my own words - 21%
- No, I am using my own words - 14%
- No, I don't know - 13%
- Just "Nothing works" - 9%
Beta testing is very close to the Software testing in general but the difference between them is also huge. This is also a result of a non-optimised workflow.
Which kind of content is the most interesting for you? 119 votes
- The list of bugs and issues - 27%
- Different testing approaches - 15%
- Features review - 35%
- App quality rating - 18%
- Something else - 5%
This is very specific to the what TestingCatalog is doing (test reports and reviews) but it also shows what kind of content is interesting to beta testers. User engagement is an important part of beta testing. This is one of the reasons why beta communities might be a good tool for beta managers.
Beta Apps Research Summary.
The goal of every beta testing program is collecting the data. There are no exceptions. You have to fight for the users attention to make him/her install your app but you will have to fight a lot more for their attention if you want to get a valuable feedback. Getting testers to install the app is usually a problem of new apps while feedback collection is problematic for everyone.
People love to try new features early and they like to receive a response to their feedback. The more detailed your changelog/update/survey is, the more detailed/focused feedback you will receive. At some point, beta testing is also a marketing activity.
From the beta testers perspective, I would try to take a look at the apps who pay attention to users feedback. There are many of them and they are looking hard for your attention.
If you are interested in beta apps Changelogs and Updates, check this TestingCatalog Updates Tracking Solution
P.S. If you found any mistakes in the text, please let me know. I spent a lot of time to make this post readable but I could miss something.