Mocking Static Method Protocols

I ran into an issue on my project where I wanted to add unit tests for my AnalyticsService facade class, which hooks up to Firebase Analytics behind the scenes. The issue is that the Firebase class uses only static method calls for interaction: Analytics.log(“event_name”)

I created an AnalyticsBackEnd protocol that declared a static method and added an extension to the Analytics class to conform to it:

protocol AnalyticsBackEnd {
static func log(_ name: String)
extension Analytics: AnalyticsBackEnd {}

The sticking point was the initializer for the AnalyticsService class: how do I say that I want to receive a class conforming to AnalyticsBackEnd without passing an instance of the class? The solution ended up looking like this:

class AnalyticsService {
let backEnd: AnalyticsBackEnd.Type
init(backEnd: AnalyticsBackEnd.Type = Analytics.self) {
self.backEnd = backEnd
func log(event: String) {

In this way, the AnalyticsService uses the Analytics by default, but allows me to initialize it with the MockAnalyticsBackEnd class for unit testing purposes:

func testAnalyticsLogging() {
let subject = AnalyticsService(backEnd: MockAnalyticsBackEnd.self)
// Test...

Core Image: Correcting Orientation

UIImage and UIImageView work together to ensure that photos are displayed in the correct orientation, based on the EXIF metadata for the image. However, Core Image does not respect orientation by default and this can cause Core Image backed rendering tasks (thumbnails, effects, etc.) to produce output with an unexpected results.

Lots of the correction techniques available in answers on Stack Overflow and other programming forums are wildly inefficient, producing a second full-resolution copy of the source image. Other techniques are based on the UIKit coordinate system (origin in upper-left corner, positive y pointing down) rather than the Core Image coordinate system (origin in lower-left corner, positive y pointing up).

Fortunately, there is a simple solution built into Core Image as of iOS 11: orientationTransform(for:). When invoked on a CIImage instance, it returns a CGAffineTransform which will convert the image from its current orientation to the .up orientation. This transform can be combined with any other transformed being performed or applied via the CIImage.transformed(by:) instance method. There is another form of the call which calculates and applies the transform, returning the resulting CIImage. See the documentation here.

Trying to correct orientation yourself is tedious and error-prone, using these methods will make it a snap and save you many hours of work.

Note: The enum cases used by the method are NOT the same as the UIImage.Orientation enum values. I’ve made a helper extension for UIImage.Orientation to convert it into the appropriate enum case for the orientation transform method.

extension UIImage.Orientation {
/// Get the equivalent `CGImagePropertyOrientation` enum case.
var cgOrientation: CGImagePropertyOrientation {
switch self {
case .up: return .up
case .down: return .down
case .left: return .left
case .right: return .right
case .upMirrored: return .upMirrored
case .downMirrored: return .downMirrored
case .leftMirrored: return .leftMirrored
case .rightMirrored: return .rightMirrored

The Pros and Cons of RxSwift…

I recently wrapped up a major app project for Google’s Cloud Next ’18 conference. We made extensive use of RxSwift throughout the project, centered around a couple of key objectives:

  1. Model-layer parity with Android – Design nearly-identical APIs for the view model and persistence layers on iOS and Android to help avoid the “iOS does one thing, but Android does another” class of bugs. Internally the implementations are quite different, but having their public interfaces documented in a Wiki allowed each platform to feel out the requirements for a particular object, then document it for the other team. This saved massive amounts of time over the course of the project and resulted in very few platform discrepancy bugs.

  2. React to schedule changes in real time – Google Cloud Next is a BIG conference; second only to Google I/O in terms of attendance. With limited seating at the various sessions, it is very important that attendees know exactly which sessions have available seats as well as receiving important updates about their reserved sessions ASAP. To accomplish this, we used RxSwift to transform observations of the real-time session details coming out of our Firebase Cloud Firestore back end into data streams that could be easily bound to views in the user’s schedule.

This was my first production experience with Rx and, as such, I experienced a considerable ramp-up curve over the course of the first month. If you are working on your first Rx project, expect to lose about 50% of your productivity for the first month and 20% of it the second month as you come to grips with the different style of data flow that Rx enables as well as learning which tool in the Rx toolbox is appropriate for each different data scenario. The up-side is that techniques learned for Rx on one platform are broadly applicable to others (which was a big reason why we chose to work with it on this project).

Once I became more familiar with Rx, I started being able to model data transformations in my head and implement them with a bare minimum of fuss. This was gratifying, but it always felt like there were some sharp edges around the boundaries between Rx code and more traditional UIKit code governing things like user interaction. I’ve created a quick list of the major points to be aware of when considering RxSwift for your iOS project:


  • Able to describe a common interface for model layer APIs between iOS and Android. This was the biggest win for us on this project, saving many dozens of hours of QA bug fixing time.
  • Avoid nested-closure hell that typifies complex asynchronous data transformations in Foundation/UIKit.


  • Steep learning curve makes ramping new developers onto the project difficult (and toward the end of the project, completely impractical). This is the #1 reason you should consider avoiding RxSwift: when it’s crunch time, you won’t be able to add developers to the project unless they’re already Rx veterans.
  • Debugging Rx data transformations is horrible. When Rx is working as intended, it’s borderline magical. When it has a problem, the debugging process is considerably more difficult. Any breakpoint you hit within a data stream will present a 40+ entry backtrace stack with dozens of inscrutable internal Rx methods separating and obscuring the code you actually wrote.
  • Rx metastasizes throughout your code base. The entry and exit-points where Rx interacts with UIKit are awkward and difficult to parse. We often found ourselves saying “oh, well if this service’s method returned an Observable instead of a variable, we could do this particular transformation more easily…” and so Rx spreads to another class in your app.

In the end, we can’t say we dramatically cut development time by using RxSwift, it simply replaced one class of problems (maintaining cross-platform consistency) with another (figuring out how to best use RxSwift). We will be launching into the next phase of the project soon, updating the app for Google Cloud Next ’19. I’m sure I will have more to talk about once that effort has completed next year.

Getting Swift 4 KVO working…

Here are 2 common pitfalls to avoid when you’re trying to use Swift 4 Key-Value Observation for the first time:

Keep that Observation object!

Calling YourObject.observe(_:options:changeHandler:) returns a NSKeyValueObservation object. The observation will only continue for as long as the observation object exists! If you fail to store the observation object in a persistent variable or array, then the it will immediately deallocate and no observations will occur.

Always Specify Options!

The 2nd parameter of the observe() method has a default value which is unhelpfully just called default in the quick documentation. It is the equivalent of providing an empty option set, which means your change handler closure will be invoked when the value changes, but you will not get any information about the new or old value! Even if this is the behavior you want, it is better to explicitly specify an empty option set so that someone else looking at your code immediately knows not to expect a value in the change handler.

Handling Drag & Drop Raw Photos

There is no shortage of tutorials on the topic of Drag & Drop, but I wanted to get into a particular special case which creators of apps that support dropped images should be aware of. If your user is a photographer that shoots with a SLR and imports their images in camera raw format, those images will not be accepted for drop unless you handle them specially.

Specifically, Raw images conform to the kUTTypeRawImage URI, which is not included in UIImage’s readableTypeIdentifiersForItemProvider. This is because UIImage cannot import raw images directly, requiring a detour through Core Image. Luckily, this is pretty easy to accomplish, as Core Image has a simple way to handle Raw images.

Note: My project is using a UICollection view as the drop target, so I’m working with the methods in the UICollectionViewDropDelegate protocol. Working with custom views should be broadly similar, but I haven’t tried it out yet.

Step 1: Getting the Raw Image

When a Raw image is dropped on an app, the image data is not sent as with UIImage-compatible images accessed via UIDropSession’s loadObjects(ofClass:completion:) convenience method. Instead, a URL is provided to your app. However, if you attempt to read or copy the file at the URL, it will always fail as being unavailable. You may be tempted to try to use NSItemProvider’s loadItem(forTypeIdentifier:options:completionHandler:) method, but it has awful ergonomics in Swift (you can’t implicitly coerce a protocol into a conforming type) and what’s more, even if you do force it to give you the URLs for the Raw images, they will all be unavailable and useless, possibly due to sandbox restrictions.

The correct way to do this is to iterate over the list of UIDragItems, filtering by those which have an NSItemProvider that responds affirmatively to hasItemConformingToTypeIdentifier(_:) for kUTTypeRawImage. You can then iterate over the filtered NSItemProviders and call loadFileRepresentation(forTypeIdentifier:completionHandler:) on each one:

This method makes an accessible copy of the Raw image and returns the URL of the copy to the completion block. Why use this method and not, say, loadDataRepresentation(forTypeIdentifier:completionHandler:)? The answer is memory: Raw images tend to be very large (dozens of MB each) and if your user has dropped a bunch of them on your app, attempting to hold all of them in memory could cause the system to kill your app for eating up too much memory. Using the URL instead of the contents of the file consumes basically no memory until a specific image needs to be loaded for processing. In tests, I was able to drop 10+ Raw images onto my app for processing and never see the memory go above about 70MB, dropping back down to 20-30MB when processing completed.

Warning: The URLs provided by loadDataRepresentation only seem to be valid for the scope of the completion closure. You shouldn’t try to hold on to them and load them later, because it will fail. Instead, copy the file in your app’s sandbox (such as the Caches directory) and use the URL of the local copy to access the Raw image later.

Step 2: Converting the Raw Image

Now we have a local URL for the Raw images, but we still can’t do much with them since UIImage is unable to initialize with Raw image data. Thankfully, Apple has a simple and robust conversion mechanism based on Core Image. Instantiating a CIFilter using the init(imageURL:options:) method creates an instance of CIRawFilter, which can process all of the various Raw image formats that Apple officially supports. The filter’s outputImage property (a CIImage) can then be sent to a CIContext for rendering to a CGImage or used directly to apply filter effects and image adjustments:

In practice, the first run of this conversion is pretty slow as the CIContext sets itself up, taking up to several seconds. Subsequent uses are very fast, requiring only a faction of a second. At the end of this process, you have a UIImage that can be used just like a regular dropped image.

ARKit + RGB Sampling

I’ve been working on an ARKit app to paint with “pixels” floating in space. When the ARSession invokes its delegate method for ARFrame capture, I want to capture the colors the camera sees at the detected feature points. I then create a simple 3D box at that point with the sampled color and can then pan around the pixel. This is pretty neat when you scan someone’s face and then have them leave.

Anyway, that whole “sample the color of the camera’s captured image at an arbitrary 2D coordinate” turned out to be a dramatically more difficult problem than I had anticipated. Obstacles include:

  • Image is in CVPixelBuffer format.
  • The pixel buffer is in YCbCr planar format (the camera’s raw format), not RGB.
  • Converting individual samples from YCbCr to RGB is non-trivial and involves doing matrix multiplication.
  • There are several different conversion matrices out for handling different color spaces, just in case you wanted to convert an image captured off a VHS tape, I guess?
  • Apple’s Accelerate framework can do this conversion on the entire image very quickly, but the setup is quite complex and consists of invoking a chain of complex C functions. Once properly configured, it is spectacularly fast, converting an entire camera image in roughly 1/2 of a millisecond.
  • The Accelerate framework has not received much love since Apple’s switch to the unified documentation style last year: hundreds of functions appear nowhere in the documentation. The only way to figure out that they exist and how to use them is to browse the Accelerate header files, which are robustly commented.
  • Swift’s type safety is a big pain in the butt when you’re dealing with unsafe data structures like image buffers.

Setting up ARKit to display the “pixels” took about 2 hours (my first ARKit experiment and my first exposure to SceneKit). Getting the colors samples to color the pixels took about 2 days. I don’t feel like this learning process is anything that is particularly valuable for your average ARKit developer to master, so I’ve tidied it up and released it as a gist.

Check it out: CapturedImageSampler.swift

Usage: when your app receives a new ARFrame via the ARSession’s delegate callback, instantiate a new CapturedImageSampler with it. You are then free to query it for the color of a particular coordinate. I’m using scalar coordinates so that the sampling is scale-independent. If you want to find the color under a user’s tap, for instance, simply convert the x and y coordinates to scalars by dividing them by the screen width and screen height, respectively. When you’re done sampling (which must occur before the next frame arrives), simply discard the CapturedImageSampler by letting it go out of scope. Do not retain the sampler, use it asynchronously or pass it between threads. It should not live longer than the ARFrame that created it.

A word of warning: this object is not at all thread-safe due to the private use of a shared static buffer. I chose this implementation for maximum performance, since a new buffer does not need to be allocated for every frame received from ARSession. However, if you get into a situation where 2 instances of CapturedImageSampler are simultaneously attempting to access the shared buffer you will have a very bad day. If you need to have a thread-safe version of this, I suggest you make the rawRGBBuffer property non-static and add a “release” method that frees up the buffer’s memory when you’re done with it. Failure to manage this process correctly will result in a catastrophic memory leak that will get your app terminated within a couple of seconds.

Quick note on CoreML performance…

I just got done doing some benchmarking using the Oxford102 model to identify types of flowers on an iPhone 7 Plus from work. The Oxford102 is a moderately large model, weighing in around 229MB. As soon as the lone view in the app is instantiated, I’m loading an instance of the model into memory, which seems to allocate about 50MB.

The very first time the model is queried after a cold app launch, there is a high degree of latency. Across several runs I saw an average of around 900ms for the synchronous call to model to return. However, on subsequent uses the performance increases dramatically, with an average response time of around 35ms. That’s good enough to provide near-real-time analysis of video, when you factor in the overhead of scaling the source image to the appropriate input size for the model (in this case, 227×227). Even if you were only updating the results every 3-4 frames, it would still feel nearly instantaneous to the user.

From a practical standpoint, it would probably be a good idea to exercise the model once in the background before using it in a user-noticeable way. This will prevent the slow “first run” from being noticed.

A note on the Swift 4 Package Manager Syntax

I ran into some issues setting up a new Swift 4 project from Swift Package Manager. Specifically, my main.swift file couldn’t import the dependencies I specified in my Package.swift file. It turns out, you have to import your dependencies in the root dependencies: section, then refer to them by module name in the targets() portion of the package.

// swift-tools-version:4.0
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "nerdScrape",
dependencies: [
// Dependencies declare other packages that this package depends on.
// .package(url: /* package url */, from: "1.0.0"),
.package(url: "", from: "0.6.0"),
targets: [
// Targets are the basic building blocks of a package. A target can define a module or a test suite.
// Targets can depend on other targets in this package, and on products in packages which this package depends on.
name: "nerdScrape",
dependencies: ["Commander"]
view raw Package.swift hosted with ❤ by GitHub

Omitting the declaration in your target means the module won’t be available to your app and your import statements will generate compiler errors for nonexistent modules.

Protobuf Server

Quick Links:
iOS AppVapor ServerPerfect Server

One of the primary challenges in learning to work with Protocol Buffers is finding an API to communicate with. Adoption is currently not wide-spread and I had trouble finding public APIs willing to communicate via protobufs. So, I decided to create my own API using server-side Swift, thus fulfilling the requirements (tenuously) for calling myself a full-stack developer. I looked at two of the most popular Swift web server frameworks currently available Vapor and Perfect.

The Contenders

Both offer easy setup via assistant tools (Vapor Toolbox and Perfect Assistant, respectively). However, the default projects’ setups are philosophically quite different. Vapor’s default setup is a fully-fledge web server with example HTML files, CSS, communication over multiple protocols, etc. Perfect’s default setup is extremely spartan, relying on the developer to add features as needed. Going head-to-head on documentation, I’d give the slight edge to Vapor, but both clearly explain how to handle requests and responses. Vapor has the reputation for having a larger and more approachable support community if you have questions, but I didn’t engage with either community so I cannot verify this.

Adding Protobufs to either default project is as simple as adding a dependency for it to the Package.swift file and running swift build:

.Package(url: "", Version(0,9,29))

Note: At the time of writing, the Swift Protobuf team considers 0.9.29 to be their release candidate, and may soon move the project to a full 1.0 release.

Once that is done, running swift build in the terminal from the root directory of the project will download the Swift Protobuf library and integrate it with your project. At this point, you’re ready to include the model files created from the .proto definitions. If you are unfamiliar with how to compile .proto files into Swift, I recommend this article as a primer. Once the models are in your Sources/ directory, you can use them in your request handling code to serialize and deserialize binary messages.

Working with Protobufs

Making a simple API server actually involves gutting the default implementation of both the Vapor and Perfect default projects, which are both set up to serve HTML responses. If you want to send Protobuf data to the server from your client app, you will need to use a POST route, as GET cannot transmit binary data. If you are simply going to request data from the server then GET is appropriate. If you’re receiving data, simply access the body data of the POST request and feed it into the init(serializedData:) initializer of your Protobuf model object to get copy you can manipulate as you see fit.

To send a Protobuf response to the client app, just follow these general steps:

  1. Create a new instance of the Protobuf model object.
  2. Assign values to the properties.
  3. Call try object.serializedData() to get the Data representation of the object.
  4. Assign the data to the body of the response.
  5. Set the content-type header to application/octet-stream. (This is optional, but is a good practice.)
  6. Send the response with a 200 OK response code.

The iOS app linked above shows the basics of using Protobufs with URLSession to parse the object(s) being returned by the server.

Protobufs 💕 BLE

Bluetooth Low Energy + Protocol Buffers

a.k.a. BLEProtobufs


Protocol buffers (protobufs) are the hot new data transport scheme created and open-sourced by Google. At it’s core, it is a way to describe well-structured model objects that can be compiled into native code for a wide variety of programming languages. The primary implementation provided by Google supports Objective-C, but not Swift. However, thanks to extensible capabilities, Apple has been able to release a Swift plug-in that enables the protocol declarations to be compiled into Swift 3 code.

The companion to this is a framework (distributed along with the plug-in) that handles the transformation of the model objects to and from JSON or a compressed binary format. It is this later capability that we are interested for the purposes of communicating with Bluetooth Low Energy (BLE) devices.

The primary selling point of protobufs is their ability to describe the data contract between devices running different programming languages, such as an iOS app and a .Net API server. There are dozens of excellent blog posts scattered about the web on protobufs, so that is all I will say about them here.

Here is the protobuf declaration for the message I will be sending between devices via BLE:

syntax = "proto3";
message Packet {
float time = 1;
sint32 rx = 2;
sint32 ry = 3;
sint32 rz = 4;
view raw Packet.proto hosted with ❤ by GitHub

A Quick BLE Primer

There are two primary actors in a BLE network: peripherals and centrals. Peripherals are devices which exist to provide data; they advertise their presence for all nearby devices to see. When connected to, they deliver periodic data updates (usually on the order of 1-2 times per second or less). The second type of device is known as a “central”, it can connect to multiple peripherals in order to read data from and write data to them.

A peripheral’s data is arranged into semantically-related groups called “services”. Within each service exists one or more data points, known as characteristics. Centrals can subscribe to the peripheral’s characteristics and will be notified when the value changes. The BLE standard favors brevity and low power consumption, so the default data payload of a characteristic is only 20 bytes (not kilobytes).

Data from a characteristic is received as just that, a plain Data object containing the bytes of the value. Thus, it is often incumbent upon the iOS developer to parse this data into native types like Int, Float, String, etc. This process is complex and error-prone, as working with individual bytes is not a common use case for Swift.

Enter Protobufs

As I mentioned above, protocol buffers can encode themselves in a compressed binary format. This makes them ideal for data transport over BLE where space is at a premium. In the example project I link to below, I am transmitting a timestamp in the form of an NSTimeInterval (double) cast to a float and three Int32 values representing the spacial orientation of the host device. I converted the rotational units from floating-point radians to integer- based degrees because integers compress much better than floating-point numbers in protobufs. After I set the properties the model object, I request its Data representation, which I save as the value of the characteristic. The data payload ranges from 5 to 12 bytes, based largely on the magnitude of the orientation angles (larger magnitude angles compress less). This is well below the 20 byte goal size.

In action:

var packet = Packet()
if let motionData = self.motionData {
packet.rx = Int32(motionData.attitude.pitch * rad2Deg)
packet.ry = Int32(motionData.attitude.yaw * rad2Deg)
packet.rz = Int32(motionData.attitude.roll * rad2Deg)
let now = Date()
packet.time = Float(now.timeIntervalSinceReferenceDate)
guard let data = try? packet.serializedData() else {
NSLog("Unable to create data from protocol buffer.")
// Write the data to the appropriate characteristic.

On the central (receiving) end, the app is notified via a delegate callback whenever the subscribed characteristic’s value changes. I take the Data value from the characteristic argument and pass it to the initializer of the protobuf-backed model object. Voila! Instant model object with populated properties that I can do with what I please.

In action:

do {
let packet = try Packet(serializedData: data)
let date = Date(timeIntervalSinceReferenceDate: Double(packet.time))
NSLog("[\(formatter.string(from: date))] x: \(packet.rx), y: \(packet.ry), z: \(packet.rz)")
} catch {
NSLog("Could not parse protobuf data: \(error.localizedDescription)")

I have a pair of example projects available. The sending app is designed to be run on an iOS device and the receiving app is a simple OS X command line app built using Swift Package Manager (because frameworks + Swift CLI apps = hell). I’ve written the core of both apps using only Foundation and CoreBluetooth, so the sending and receiving roles should be easy to swap between different platforms.

Peripheral (sender) app

Central (receiver) app