Different types of iOS “crashes” and their implications for 3rd party crash reporting tools

Sometimes a “crash” can be something entirely different from what you think it is. This makes it quite hard for (3rd party) crash reporting tools to report them. Let’s travel down the rabbit hole of “crashes” without losing our minds, and report them all!

Talking about apps that “crash”.

To an iOS developer, the phrase “my app is crashing” usually means something like this:

“I am using the app which then suddenly disappears, and am taken to the home screen. This is annoying, I had almost finished writing a really long blog post and hadn’t saved it yet. Now all product of my labor is lost; all because the app developer had a bug in their code. ”.

Turns out, one of the interesting side-effects of working on crash reporting tools for (mobile) app developers[1][2] is that you get to talk a lot about crashing applications, get to see a lot of stack traces[3], and get a lot of opportunities to figure out what the heck people want to say. By “people” I mean our own product managers, developers, and designers, as well as customers who get in touch through our various communication channels. I may even communicate with one of our customers’ customers directly. Customers that I encounter during my workday are a very diverse group of people. They can be anyone from a C-level executive of a company, a software engineer, a designer or a Product Manager of any level of seniority, or an end user. To them, the phrase “my app is crashing” can have quite a few different meanings. So what does it mean to “crash”? Let’s start with a few fun cases and move on to more technical “crashes” later on.

What does the dictionary say?

According to the dictionary[4], the verb “crash” means

“to cause (a computer system, component, or program) to crash”

or

“of a computer system, component, or program: to suffer a sudden major failure usually with attendant loss of data”

These are pretty good explanations of what it means for an (iOS) application to “crash”. They may lack some detail, but then they are from an entry in a dictionary, and not from a comprehensive guide on the meaning of the word “crash” for the iOS platform.[5]

Hard to use apps

This category of “crashes” is not strictly about “crashes”. Nonetheless, it comes up somewhat regularly. It boils down to the fact that an iOS application is harder to use than it should be. As a result, the user is under the impression that the app is not working correctly and has contacted the customer support.
Imagine the following example that happened to me during my commute to work a few months ago:

You are using a simple notes app on your iPhone. It enables you to write notes and save them (similar to iOS’ stock “Notes” app). Its main UI is a UINavigationController with a UITableViewController at the root that allows scrolling through a list of your notes’ titles. It supports adding notes, deleting notes, and, on selecting an item in the list, editing a note. Adding or editing an existing note pushes a UIViewController on the stack and shows a large UITextField to edit your note. Now, imagine you just worked on an interesting blog post for an hour or so (while you were commuting), and want to navigate back to your list of notes by swiping back to the previous scene instead of hitting the “Save” button at the top right of the UINavigationBar. The application presents a UIAlertController. You hit the “Ok” option. Unfortunately, you did not understand what the alert said and you lose your blog post.

What happened here? The app worked as intended, and even showed an alert, yet failed to communicate the destructive nature of my action. For example, labeling the button “Discard” instead of “Ok” could have prevented this mistake. While data was definitely lost, this case cannot be considered a “crash”, yet, people might refer to it as one.

How would you collect information similar to the example outlined above? Crash reporting tools cannot help you for this category; this case is – strictly speaking – not a “crash”. That said, iOS 11 and later come with a built-in screen recorder.[6] Make it easy for your QA testers and customers to get in touch and submit their feedback with screenshots or videos.[10]

It’s a bug in your app but not a “crash”

Following apps that fail to meet their user’s expectation, let’s talk about non-fatal bugs. A bug can of course lead to an application crash, but don’t necessarily has to; let me explain.

Back in 2015, a QA tester reported a bug in an app that I was developing for a customer. They described that they would watch a video (it was an app to stream videos), and the video player (which used the AVFoundation framework) would suddenly “crash”. I was confused. How could the video player “crash”, but not the app itself? What actually happened was that there was an issue with their internet connection. The video player was supposed to show a message and let the user know about the problem. Unfortunately, the UILabel for the message was hidden by another view in the pretty complex view hierarchy on the screen. I would never have thought about this as a “crash”, but what happened actually matches one of the definitions of “crash” (“[…] to suffer a sudden […] failure …”).

Like our previous category of “crashes”, it cannot be reported by crash reporting tools. It can be found during testing using automated UI tests[11] or (manual) QA testing. For the later, use a tool to allow for easy reporting of issues as mentioned in the previous paragraph.[10]

The iOS watchdog process

Moving on, we’re encountering our first “crash” category that actually exits the app: app terminations by the iOS watchdog. Although it has been around since the dawn of iOS, not all developers are familiar with it.

iOS’ watchdog process generates a “watchdog timeout” when an app takes too long to launch, terminate, or respond to system events. Note that this is not an application crash, but an app “kill” by iOS itself. App “kills” cannot be reported by 3rd-party crash reporting solutions even though there is an underlying exception. From Apple’s Tech Note 2151[7]:

The exception code 0x8badf00d indicates that an application has been terminated by iOS because a watchdog timeout occurred. The application took too long to launch, terminate, or respond to system events. One common cause of this is doing synchronous networking on the main thread. Whatever operation is on Thread 0 needs to be moved to a background thread, or processed differently, so that it does not block the main thread.

While seeing 0x8badf00d in your stack traces in iTunes Connect[8] is typically a good indicator that there is a bug in your app, it can also be expected behavior, e.g. iOS will stop CPU intensive processes if the iDevice is running hot to prevent overheating.[9]

What are your options for this? First, during development, make sure to do some basic performance testing and app launch testing.
To check for CPU intensive operations on the main thread, utilize the “Main Thread Checker”.[12] Some crash reporting tools also have heuristics to detect cases where an app has been killed by the watchdog process.[13]

Detecting low memory situations

A very common case for app terminations is app terminations as part of the regular iOS application lifecycle.[15] Once iOS runs low on memory, its applicationDidReceiveMemoryWarning:[16] and UIApplicationDidReceiveMemoryWarningNotification[17] callbacks will message the app ahead of the termination to let it free up memory before it will eventually terminate the app to free resources. Make sure to utilize these methods to save any data before iOS will, as a last resort, send the applicationWillTerminate:[18] message and ultimately terminate your application.

As with the previous categories of “crashes” (app terminations by the watchdog process), iOS will not allow for any 3rd-party crash reporting here so they need to rely on heuristics again. For example, they will listen for the applicationWillTerminate: callback and perform some basic “termination reporting” at the next launch. Use a 3rd-party crash reporting solution that support heuristics like this but be mindful as they usually report false-positives, too.

Application kills

The watchdog process and low memory situations are not the only causes for an app’s termination. The next category in our list of “crash” causes are application “kills”. A very common cause for them is when a user decides to “force close” the application.[14] From Apples’ “App Programming Guide for iOS”:

In addition to the system terminating your app, the user can terminate your app explicitly using the multitasking UI. User-initiated termination has the same effect as terminating a suspended app. The app’s process is killed and no notification is sent to the app.

This kind of app “kills” is even more problematic to 3rd party crash reporting tools than the previous terminations, as their heuristics mostly rely on them receiving messages from iOS, i.e. the applicationWillTerminate callback. Some 3rd party tools again use heuristics to detect app kills by the users, which are – as mentioned before – a mixed bag as they usually cause false-positives. You need to decide for yourself if you are ok with false-positives, especially if your app supports long-running background operations.

Also note that app “kills” are considered the same as the regular “kills” of suspended iOS apps. For this case, make sure you are familiar with iOS’ application lifecycle and persist your data before the app is suspended.

Exiting the app

Now that we have discussed application termination and “kills”, let’s move to the next category of “crashes”. While a deliberate exit of the applications process is technically not a “crash”, they can definitely be perceived as one. To exit an iOS app, you can call exit(0) or [[NSThread mainThread] exit]. While I don’t think that this is a common thing to do for the average app, there might be valid use-cases. App Center’s in-app update feature is one of them.
I have reviewed code in “ordinary” apps in the past where some error condition would exit the app:

if(error) {
    // this is bad and should never happen.
    exit(0);
}

Long story short: just don’t! This is a legit way to end a process, there is nothing for crash reporters to report: no crash, no crash report. 😉

Again using a crash reporting solution that supports detecting unintentional app terminations is a last resort with the aforementioned likeliness of false-positives if you want to detect application exits.

Heureka! It’s a crash!

This is brings us to the final category of “crashes”. Their description is quite simple. A “true crash” is neither a hard to use application, a non-fatal bug, an app termination, an app “kill” nor a deliberate exit of the app by its developer. A crash is when an app suffers a major failure that is fatal to the app (the user is taken to the home screen end the lifecycle of the app was ended abruptly).

Yet, all categories that were discussed fit the definition in the dictionary:

“of a computer system, component, or program: to suffer a sudden major failure usually with attendant loss of data”

So when is a “crash” a “crash”?

Some of the categories above blur the lines quite a bit. Technically, none of them are crashes. Ultimately, it doesn’t matter because your goal as an app developer is to help your users achieve a task and make them happy.
Make sure to make a crash-free experience a top priority from the start, no matter what kind of crash it is, and don’t rely on your crash reporting tool to alert you to them. And, in case someone complains about a “crash”, make sure you ask the right questions to make sure you understand what kind of “crash” they are talking about.


 

1 HockeyApp

2 App Center

3 Wikipedia entry for ‘stack trace’

4 Entry for the word ‘crash’ at Merriam-Webster.com

5 Thinking about it, some of my family members sometimes refer to their internet connection not working as “my internet has crashed”, which is actually correct if we follow the definition in the dictionary.

6 iOS 11 Screen recording

7 Apple Tech Note 2151: Understanding and Analyzing Application Crash Reports

8 iTunes Connect

9 WWDC 2018 Session ‘Understanding Crashes and Crash Logs

10 HockeyApp has a basic feedback functionality while other tools support more sophisticated workflows.

11 Apple’s documentation on User Interface Testing

12 Main Thread Checker

13 HockeySDK-iOS API to detect whether an app has not terminated cleanly

14 How to force close an iOS app

15 App Termination

16 Documentation for applicationDidReceiveMemoryWarning:

17 Documentation for UIApplicationDidReceiveMemoryWarningNotification

18 Documentation for applicationWillTerminate:

Distributing your iOS app with an arm64e slice

Learn how to distribute your iOS app with the arm64e CPU architecture slice.


Introduction

This year’s incarnation of Apple’s iOS devices features the new Apple A12 Bionic CPU. Performance improvements aside, it is the first commercially available 7nm silicone, and the first CPU with the arm64e CPU architecture (as usual, AnandTech covers Apple’s new CPU extensively). Its new security extensions might require you to make adjustments to your code. Apple explains the various usecases and provides necessary steps in their documentation. The new pointer authentication requirements of arm64e resulted in additional work for my team at App Center as well.
It turns out to be quite challenging to distribute an app with an arm64e slice today. While Xcode 10.1 added support for developers to test their app with new architecture, it is not possible to submit an app to the App Store or TestFlight. In fact, the arm64e architecture will be stripped away if you use Xcode’s Organizer window. This happens even when exporting for AdHoc or Enterprise distribution (see the Xcode 10.1 release notes), and even when exporting from the commandline via xcodebuild.

In this post, I will show you how you can circumvent this limitation and export an iOS app with the arm64e slice. You can distribute it to your testers (using one of the alternatives to TestFlight of course).


Export the .xcarchive with the new architecture

First, you need Xcode 10.1 installed on your machine. Open your app’s project or workspace in Xcode 10.1 and add the arm64e architecture as described in the documentation.

Image1

Once you have done that, archive your application the same way you normally would.

This assumes that your app’s dependencies support the arm64e architecture. We recently added arm64e support to the App Center SDK for iOS. I plan on covering our journey in another post.

After Xcode has finished archiving the app, the Organizer window will open. Right-click on the archive that you just created and select “Show in Finder”.

Image2

I prefer to copy the .xcarchive package somewhere where I can find it easily. For my example, I’m choosing a folder on the desktop. To check if the binary in your .xcarchive package contains the slice for the arm64e architecture, run

lipo -info $PATH_TO_THE_BINARY_INSIDE_THE_XCARCHIVE.

The easiest way to get the path to the binary is: right-click on the .xcarchive package that you created, select “Show Package Contents”, and locate your .app package inside Products/Applications/. Then right-click on the .app package, select “Show Package Contents” once more, and find your app’s binary.

In my example, I need to run:

lipo -info /Users/benny/Desktop/AwesomeApp/AwesomeApp.xcarchive/Products/Applications/AwesomeApp.app/AwesomeApp.

This prints out the following:

Architectures in the fat file: /Users/benny/Desktop/AwesomeApp/AwesomeApp.xcarchive/Products/Applications/AwesomeApp.app/AwesomeApp are: arm64 arm64e

Awesome, the .xcarchive already contains the arm64e slice!


Create the .ipa file

The next step is to create the .ipa file that you can install on your new iPhone/iPad.

My initial idea was to switch back to Xcode 9.4.1, run xcodebuild -exportArchive -archivePath ./AwesomeApp.xcarchive/ -exportPath ./ -exportOptionsPlist ExportOptions.plist, and trick xcodebuild into exporting an .ipa file which contains the arm64e slice.

Unfortunately, this doesn’t work; signing the .dylibs and .frameworks will fail because Xcode 9.4.1’s code_sign tool will not be able to do it’s job – the .xcarchive was created in Xcode 10.1, after all.

Luckily, I learned about the ditto tool from a co-worker the other day. cd into the folder that contains your app’s .xcarchive and create a temporary directory structure:

mkdir -p tmp/app/Payload.

Copy your .app package into the directory using:

ditto AwesomeApp.xcarchive/Products/Applications/AwesomeApp.app tmp/app/Payload/AwesomeApp.app,

followed by:

ditto -ck --rsrc --sequesterRsrc --keepParent tmp/app/Payload/ AwesomeApp.ipa


Congratulations, you’re pretty much done. Use lipo -info on the app’s binary inside the .ipa package (unzip the .ipa, then use “Show Package Contents” ) to verify that it contains the arm64e architecture. It should print something like this:

/Users/benny/Desktop/AwesomeApp/Payload/AwesomeApp.app/AwesomeApp is architecture: arm64 arm64e

Now, go distribute your .ipa with the new arm64e architecture to your testers and squash all those bugs!

Hacktoberfest 2018

It’s October, and because I’m a German who hates Lederhosen, I feel like I need to compensate ;). Just kidding, have you heard of Hacktoberfest? It’s an awesome initiative that was started 5 years ago. A lot of companies are on board this year, including Microsoft. To learn more about Hacktoberfest, have a look at the the official website.

I’m definitely contributing something to one of the Microsoft Open Source repositories and likely elsewhere.

You should join Hacktoberfest. You might even get a free t-shirt!

 

 

The impact of the App Center SDK for iOS on your app’s binary size

TL;DR

Although the .framework packages of the App Center SDK look big at first glance, they have a minimal impact on your app’s release binary size. If you add all of App Center’s services to your app, your app’s installation size should increase by about 800KB. Your mileage may vary a little.

Background information

Time and again, we get customers who integrate our SDKs and are concerned about the size of our SDK when they integrate the SDK. This is understandable, as the App Center SDK is distributed as .framework packages that range from 1.7MB for App Center Analytics, to 13.4MB for App Center Crashes. The thing is, those sizes are misleading. Our SDK is, in fact, designed to have a minimal footprint when it is integrated into an app. In fact, a minimal footprint is why we created a modular SDK in the first place (I’ve blogged about this in the App Center blog), developers can pick the functionality they want, and don’t have to worry about their app size at all. So why are the .frameworks so big. The answer is really simple, they are “fake” fat frameworks, statically linked frameworks in disguise. The reason for this is rather simple: performance at app launch time. But let’s focus on their impact on an apps binary size. As the term “fat framework” implies, they contain slices for all architectures, meaning all device CPUs as well as the iPhone/iPad simulators. If you build your app for a specific target, the compiler will actually strip the unused ones. In addition to that, depending on your compiler/build settings, the compiler will strip away even more stuff, because it is smart. With Bitcode enabled, things get even more compacted and then, there’s also App Thinning. If your app’s size is important to you, you should check those techniques out.

How I measured

I created a really simple app using Xcode 9.2 and its default template for an app with one single view controller. I added two buttons, one that calls trackEvent: on the App Center Analytics module, and one to crash the app intentionally. You can check out the project on Github.

  • I did not make any changes to Xcode’s default build settings.
  • I created a release build using Xcode and then exported the .ipa manually using Xcode’s Organizer.
  • I did not use Bitcode.
  • I did not use App Thinning.
  • I installed the app on my iPhone 7 that’s running iOS 11.3 using the Apple Configurator app.
  • I launched the app once to make sure the SDK is initialized.
  • I checked Settings > General > iPhone Storage and made sure it refreshed the data for my AppSizeSample app after each install.
  • I removed the app after each install.
  • I installed each variant of the app twice (and deleted it manually before).

Results

Take these with a grain of salt as I didn’t change any of Xcode’s default settings, and because I only measured with 1 app and 1 device. Your mileage may vary but my measurements should give you a pretty good idea of how “big” the impact of the SDK is.

App Center modules used Exported IPA size Installation size
None (blank app) 24KB 132KB
App Center Analytics 120KB 377KB
App Center Crash 239KB 705KB
App Center Analytics & Crash 250KB 741KB
App Center Distribute 163KB 528KB
AppCenter Analytics, Crash & Distribute 307KB 913KB
App Center Push 115KB 377KB
All App Center modules 314KB 930KB

The numbers are interesting as they don’t just “add up” because all modules share the same “core”, a module named “App Center” that contains shared logic for persistence and network i/o plus I suspect the compiler to be pretty smart about even more things.

In addition to that, the App Center SDK will use between 29KB and up to 1.2MB of space for “Documents & Data”. The amount varies depending on your internet connection as App Center buffers up to 300 event logs and 300 crash logs when offline.

Anyway, I hope this answers some questions.

What else?

  1. Check out App Center, it’s awesome. We’re on Twitter, too. Get in touch if you have feedback.
  2. Our SDK is open source on Github. Check that one out, too.
  3. If you’re interested in learning more about static vs dynamic frameworks, Samantha has you covered. You should read her blog anyways. 😉 And send her some money through her Paypal because her blog is awesomesauce.
  4. Do what I did on your device and let me know your numbers!

Final words

Thanks for reading, please get in touch if you have further questions or feedback.

Cheers,

Benjamin

Core-Value Observing

How can we talk about core values and rules for companies, our pull requests or our code a lot, but we don’t talk about individual core values? Documentation gets pretty scarce when you look up “core values for software developers,” so let’s talk about what they are and how to change them!

And that’s what I did. About two weeks ago, I stood in front of an audience, at the Xcoders Meeting, and talked about a topic that I have been thinking, reading and writing for the past year or so. Well, the talk has its roots well before that but you’ll learn about them once you watch me talk.

I want to keep on talking about it. If you host a meetup, a gathering of your team where you invite people from outside, or whatever, please get in touch. =)

If you’re curious, slides are available (with notes).

How to set up an Android Development environment inside a Hyper-V VM in Windows 10

TLDR;

You can set up Android Studio inside a Hyper-V virtual machine. The caveat ist that you need to use the Visual Studio Android Emulator instead of Google’s Android Emulator.

Motivation

I did a 75 minute lab at TechReady 24, an internal conference at Microsoft this week. For this, I needed to provide a virtual machine with the development environment that would be installed to over 40 machines inside the room where the lab took place. The requirement for this was – we’re talking Microsoft after all – that the virtual machine uses Hyper-V, Microsoft’s own virtualization technology. As the lab was about our Android SDK, we need an Android Development Environment inside the VM which turns out to be rather tricky and poorly documented. All I found about this was a mere: “it’s possible”. The tricky part is that you can’t use Google’s own emulator inside a Hyper-V VM as HAXM requires Hyper-V to be disabled. So here’s how to setup an Android Dev Environment using the Visual Studio Android Emulator inside a Hyper-V VM.

Required Software/Hardware

I have used my 2015 Lenovo X1 Carbon laptop for this. It worked out well but more than 8 GB RAM would have been really nice as the VM only gets 4GB of RAM on a host with 8GB of RAM.

  • Windows 10 Enterprise/Professional/Education
  • 8GB RAM or more – According to the docs Hyper-V requires a minimum of 4GB on the host machine. The Visual Studio Android Emulator requires at least 2GB of RAM in the virtual machine, so a host machine with 4GB of RAM just won’t cut it. My laptop has 8GB of RAM and I definitely felt the need for more.
  • A 64-bit Processor with Second Level Address Translation (SLAT).
  • CPU support for VM Monitor Mode Extension (VT-C on Intel CPUs).

Task 1: Setup your VM

1. Make sure Hyper-V is enabled on the host.

Just apply some Google-fu here. It’s not that complicated.

2. [Optional] Create virtual switches

The Hyper-V Management Console application allows to define “virtual switches” that map your physical network to a virtual one. On my machine, this was already configured correctly.

3. Create your new VM

Open the Hyper-V Manager and create a new VM running Windows 10 using the wizard in the Hyper-V Management application. I won’t go into the details on how to do it as the wizard is pretty self-explanatory. The following 4 settings are crucial for our Android dev setup and no-one really tells you about this. I needed to apply some serious Google-fu:

  1. Set a static amount of RAM for the VM. The VM will need at least 4GB of it. 6 or 8 GB is better. My laptop could barely handle 4GB of RAM for the VM but it did the trick.
  2. Set the VM to have 2 CPUs even if your machine has just 1 CPU.
  3. Connect the virtual machine to your virtual switches to make sure the VM has internet access.
  4. Allow Mac Network Spoofing.

Before we go ahead, run your new VM for the first time and complete the initial Windows installation. Verify that the VM can access the internet. Once you’ve done that, shut down the VM.

4. Enable nested virtualization

You will be running the Visual Studio Android Emulator – which itself is a virtual machine –from inside a virtual machine and both are using Hyper-V. This means you need to enable nested virtualization which isn’t configured per defaul. Thanks to this blogpost it doesn’t get very complicated, because you did Steps 3.1 through 3.4, right?!

  • Now, on your host machine, open PowerShell and run the following command. Replace <VMName> with the name of your VM. Set-VMProcessor -VMName <VMName> -ExposeVirtualizationExtensions $true

The official Hyper-V manual explains other configs to enable nested virtualization. After a lot of trial and error, I ended up with the VM configuration and the PowerShell command that I described above.

Task 2: Install Android Studio and Visual Studio Android Emulator

1. Android Studio

Get and install Android Studio. Easy. When you try to run your app in the emulator, Android Studio will complain that you need to disable Hyper-V to run its emulator. Ignore the message.

2. Visual Studio Android Emulator

  1. Make sure Hyper-V is enabled INSIDE the VM.
  2. Download the VS Android Emulator. After installation is complete, download one profile and start the device. It takes a few minutes, especially for the first launch. It’s really straight forward but there is one thing of note here: Launching the device will take forever (as in “will never finish”) if the VM doesn’t have enough RAM. It will never tell you about this. If your VM has 4GB of RAM, you should be able to run a profile for a device with up to 1GB of RAM. A 2GB profile won’t work. I wanted a device with API level 23 and had to choose a different profile than the default one because of this limitation. Also: Choosing a device profile with 512GB of RAM improved the launch time of the emulator considerably.
  3. Verify that the emulator itself is connected to the internet. If you run into issues here, it’s basically the same steps as with as for the VM on the host pc open the Hyper-V Management app. There should be a VM for each profile that you created in the VS Android Emulator. Check if the VM has its network switches configured correctly. If the “host”-VM was configured correctly, it should work out of the box.

3. Run your android app in Android Studio.

Run your new Visual Studio Android Emulator. Android Studio should recognize the running emulator as a device and allow you to run your app on it. You won’t be able to launch the VS Emulator from within Android Studio.