Jon's Blog

So You Want to Build a VR App

Mon, 09 Aug 2021 00:00:00 GMT

In this article, I describe the essential considerations that will affect the scope of your VR game early on, how they differ from traditional game development, and how to best meet them head-on.

You can read it here.

Why XCFrameworks Matter

Tue, 12 Jan 2021 00:00:00 GMT

XCFrameworks are the future of framework development on iOS. I’ve written an article covering just what they are and why they’re important, which you can read here.

OpenCV, InputArray, OutputArray, and You

Wed, 13 Mar 2019 00:00:00 GMT

When you’re writing OpenCV code in C++, you’ll eventually want to pass whatever Mat objects you have into a function or a class method. For the purposes of this post, let’s say we want to write a utility function that prepares a BGR888 Mat for edge detection. This typically involves converting the image to grayscale, as well as blurring it to remove superficial details.

It’s tempting to write the following function signature:

// Revision 1
Mat preprocess(const Mat& input) {
  Mat gray, output;
  cvtColor(input, gray, COLOR_BGR2GRAY);
  blur(gray, output, Size(3, 3));
  return output;
}

This is good C++ code; we pass a reference to the input Mat so that we don’t incur a copy, and we use const to mark the reference as read-only. However, OpenCV provides some utilities that we can use to improve this function.

Mat is not the only container for representing an image in OpenCV. If you’re looking to leverage OpenCV’s Transparent API for GPU acceleration, you’ll want to use UMat as your container instead. If you have an existing codebase full of utility functions written like above, you’re going to have to make copies that take a UMat reference if you want to use those.

Fortunately, OpenCV has a solution to this problem in the form of the InputArray proxy class. As per the docs, InputArray lets you pass in a Mat, UMat, or even a std::vector, among others.

Here’s what our function looks like using InputArray:

// Revision 2
Mat preprocess(InputArray input) {
  Mat gray, output;
  cvtColor(input, gray, COLOR_BGR2GRAY);
  blur(gray, output, Size(3, 3));
  return output;
}

Note that we aren’t passing a reference to input. InputArray internally typedefs as a reference, so that’s taken care of for you; that said, Mat and UMat are internally backed by smart pointers, so we don’t need to worry about getting a reference anyway! Using InputArray also has the nice benefit of making your image read-only as far as the OpenCV SDK is concerned, which helps the compiler find issues and makes your code easier to read.

OpenCV provides a similar proxy class called OutputArray, which wraps the same types but is treated as write-only. Let’s rewrite preprocess() with that:

// Revision 3
void preprocess(InputArray input, OutputArray output) {
  Mat gray;
  cvtColor(input, gray, COLOR_BGR2GRAY);
  blur(gray, output, Size(3, 3));
}

Now instead of returning a new Mat object, whatever we pass into output will be written to.

Why would we want to do this? Well, when you’re writing computer vision code, you need to be careful about how your image data is being passed around. After all, the images you’re passing around are uncompressed; a 4000x4000 image in BGR888 format would take around 48 megabytes of space in RAM. That’s not a lot, but it adds up fast, particularly if you’re parallelizing or running on a mobile platform. If we gave that image to Revision 2 of the preprocess() function, we would be allocating two copies of the image; one temporary (gray) and one that we keep and return ( output).

With Revision 3, we’ve moved the decision to allocate the output from inside of the function to wherever it gets called. If you want a copy, call preprocess(someMat, someOtherMat). If you want to overwrite the original Mat, just call preprocess(someMat, someMat), using the same object for the input and the output. If you’ve wondered why so many of OpenCV’s functions write outputs to an OutputArray parameter instead of returning a Mat, this is why!

Now let’s say we always want preprocess() to operate in-place. We can make use of OpenCV’s InputOutputArray to completely avoid allocating new images in preprocess() and make the function a little easier to read:

// Revision 4
void preprocess(InputOutputArray input) {
  cvtColor(input, input, COLOR_BGR2GRAY);
  blur(input, input, Size(3, 3));
}

Easy.

As a footnote, you might be wondering to yourself, “Why not just avoid all of this and use UMat everywhere?” The simple answer is that OpenCV functions that consume a UMat will run on the GPU if it can, falling back on the CPU if not. If you pass a Mat, it will always run on the CPU. Sometimes you’ll want to explicitly force an algorithm to run on the CPU (or bail on using UMat when you run into some platform-specific bug), so having the ability to quickly switch between the two will save you a lot of time.

Optional outputs with `InputArray` and `OutputArray`

A nice trick you can do with InputArray is making parameters optional. Let’s suppose we want to write a function that takes a Mat and a mask (also a Mat):

void someFunction(InputArray input,
        OutputArray output,
        InputArray mask) {
  // 1. Use the mask on the input
  // 2. Write something into the output
}

Masks in OpenCV’s own functions are typically optional; that is, you can pass in noArray() and no masking operation will take place. We can do that in our function as well:

void someFunction(InputArray input,
        OutputArray output,
        InputArray mask = noArray()) {
  if (!mask.empty()) {
    // 1. Use the mask on the input
  }
  // 2. Write something into the output
}

This is a trivial detail, but writing the function this way makes it so that if you don’t want a mask, you can just omit the parameter entirely instead of writing in noArray() yourself.

A more interesting case is the ability to make an OutputArray optional. Let’s suppose we’re writing a function that tells us how many closed edges it could find in an image (and for good measure, let’s use Revision 3 of the preprocess function we wrote earlier):

int getNumberOfClosedEdges(InputArray input) {
  Mat processed, edges;
  preprocess(input, processed);
  Canny(processed, edges, 100, 170, 3);
  // Imaginary function that returns the number
  // of closed edges in `edges`
  return numberOfClosedEdges(edges);
}

All we care about when we’re using this function is the number of closed edges. However, you can use the result of the Canny edge detection (edges) for visualization purposes or other processing tasks later on, so wouldn’t it be nice if we could optionally choose to keep it?

OutputArray has a read-only needed() property, which returns false if its backing object is a noArray() instance. Let’s use that with our function:

int getNumberOfClosedEdges(InputArray input,
        OutputArray outputEdges = noArray()) {
  Mat processed, edges;
  preprocess(input, processed);
  Canny(processed, edges, 100, 170, 3);

  if (outputEdges.needed()) {
    edges.copyTo(outputEdges);
  }

  // Imaginary function that returns the number
  // of closed edges in `edges`
  return numberOfClosedEdges(edges);
}

Now we have an optional parameter to write the Canny results into if we want them. This technique is particularly useful for cases where you have an image processing pipeline with several steps and you want to store intermediates for visualization or debugging.

Randomizing UIColor with Method Swizzling

Fri, 10 Aug 2018 00:00:00 GMT

Lately, I’ve helped contribute to Slide for Reddit, an open-source Reddit client for iOS written in Swift. We’re reconsidering some aspects of our UI customization options, and I’m of the opinion that the user just doesn’t have the tools they need to make the app as ugly as humanly possible. As a joke, I wanted to randomize all the colors throughout the app just to see what it would look like. That led to an interesting question: is it possible to write an extension for UIColor that forces all instances to be randomized when they are used?

This was a surprisingly deep rabbit hole to jump into.

First Steps

It is not possible to override a property or method inside of a Swift extension. This precludes doing something fishy like overriding UIColor’s initializer. In any case, we want the color to change every time it is accessed, so we would need to override a property of UIColor.

Interestingly, UIColors are not used directly for drawing; instead, they serve a CGColor instance via their cgColor property when accessed. If we could override cgColor in our extension, we could return a random color from there, but this is not allowed. Or at least, it isn’t allowed in pure Swift.

The Objective-C runtime allows us to perform method swizzling, which is a technique that replaces the implementation of a selector with another implementation at run time. After a brief read on this topic, I had the following code:

extension UIColor {

  class func jc_swizzle() {
    let originalSelector = #selector(getter: cgColor)
    let swizzledSelector = #selector(randomCGColor)
    let originalMethod = class_getInstanceMethod(self, originalSelector)!
    let swizzledMethod = class_getInstanceMethod(self, swizzledSelector)!
    method_exchangeImplementations(originalMethod, swizzledMethod)
  }

  @objc func randomCGColor() -> CGColor {
    return CGColor(
      colorSpace: CGColorSpaceCreateDeviceRGB(),
      components: [
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // R
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // G
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // B
        1.0, // A
        ])!
  }

}

In jc_swizzle(), we swap the getter for UIColor.cgColor with the extension function randomCGColor(). This in theory does what we want! Let’s see what it looks like:

What?! That looks good! That can’t be right. What happened?

Class Clusters

As it turns out, Apple makes use of an Objective-C pattern called a class cluster in UIColor. Under the hood, there are many private subclasses of UIColor, but they’re all hidden from any public interface. That means we just overrode the method of a class that doesn’t actually ever get used.

So then, let’s find out what class some random UIColor instance resolves to. For UIColor.red, that ends up being UICachedDeviceRGBColor. Let’s work on that:

public extension UIColor {

  public static func jc_swizzle() {
    // This resolves to `UICachedDeviceRGBColor`
    let _class: AnyClass! = object_getClass(UIColor.red)
    let originalMethod = class_getInstanceMethod(_class, #selector(getter: cgColor))
    let swizzledMethod = class_getInstanceMethod(_class, #selector(randomCGColor))

    if let originalMethod = originalMethod, let swizzledMethod = swizzledMethod {
      method_exchangeImplementations(originalMethod, swizzledMethod)
    }
  }

  dynamic func randomCGColor() -> CGColor {
    return CGColor(colorSpace: CGColorSpaceCreateDeviceRGB(), components: [
      CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // R
      CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // G
      CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // B
      1.0, // A
      ])!
  }

}

This works. On the highlighted line, we get the class object for UIColor.red (UICachedDeviceRGBColor), which we then pass as the first argument to class_getInstanceMethod instead of UIColor.self. By swizzling UICachedDeviceRGBColor we can affect many, many more colors.

But who knows how many other UIColor subclasses there are? I couldn’t stop here. I wanted to ruin all the colors. The code that results from this path of inquiry is horrible and evil and will absolutely get you rejected from the App Store. Apple doesn’t like developers referencing private headers, after all. Let’s do it anyway.

How can we reference the private internal objc classes that back UIColor? Well, we can use NSClassFromString() to get them if we know their names. We know about UICachedDeviceWhiteColor and UICachedDeviceRGBColor but who knows how many others there are under the hood?

Well, I knows. The answer is ten:

UIDeviceRGBColor
UICachedDeviceRGBColor (derives from UIDeviceRGBColor)
NSColor
UIDisplayP3Color
UIPlaceholderColor
UICIColor
UIDeviceWhiteColor
UICachedDeviceWhiteColor (derives from UIDeviceWhiteColor)
UICGColor
UICachedDevicePatternColor (derives from UICGColor)

I found these by grepping through a dump of the private headers in iOS 11.4. If you’d like to do this yourself, run grep -r -n ".\+:\s*UIColor". in the terminal (you’ll also want to run this again for each subclass you discover, replacing UIColor with the new class name just in case there are any subclasses of those). Let’s use these:

public extension UIColor {

  private struct StaticVars {
    // Names of classes to swizzle. Derived from dumped private headers.
    static let classesToSwizzle: [String] = [
      "UIColor",
      "UIDeviceRGBColor",
      "NSColor",
      "UIDisplayP3Color",
      "UIPlaceholderColor",
      "UICIColor",
      "UIDeviceWhiteColor",
      "UICGColor",
      "UICachedDeviceRGBColor",
      "UICachedDeviceWhiteColor",
      "UICachedDevicePatternColor",
    ]

    static let randomColorBlock: @convention(block) (AnyObject?) -> CGColor = { _ -> (CGColor) in
      return CGColor(colorSpace: CGColorSpaceCreateDeviceRGB(), components: [
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // R
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // G
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // B
        1.0, // A
        ])!
    }
  }

  public static func jc_swizzle() {
    for className in StaticVars.classesToSwizzle {
      if let classFromString = NSClassFromString(className),
        let originalMethod = class_getInstanceMethod(classFromString.self, #selector(getter: cgColor)) {
        method_setImplementation(originalMethod, imp_implementationWithBlock(
          unsafeBitCast(StaticVars.randomColorBlock, to: AnyObject.self)
          ))
      }
    }
  }

}

This is much, much more effective:

Note how everything is affected, including the individual drawable elements in the status bar:

This is a fine solution, but it doesn’t satisfy the engineer in me. What happens when iOS updates? What if there are classes that I missed?

Things Get Worse

Shortly after writing this code, I was made aware of objc_getClassList(). This creates a list of class objects for every single class loaded at runtime. In fact, on my phone running iOS 11.3.1, that list contains 25,265 elements. Let’s parse these instead of keeping a list of strings:

public extension UIColor {

  private struct StaticVars {
    static let randomColorBlock: @convention(block) (AnyObject?) -> CGColor = { _ -> (CGColor) in
      return CGColor(colorSpace: CGColorSpaceCreateDeviceRGB(), components: [
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // R
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // G
        CGFloat(Float(arc4random()) / Float(UINT32_MAX)), // B
        1.0, // A
        ])!
    }
  }

  private static func jc_swizzle() {

    // Returns true if the given class has `superclass` anywhere in its class hierarchy
    func isClassSubclassOf(_ base: AnyClass?, superclass: AnyClass) -> Bool {
        var _class: AnyClass? = base
        while _class != nil && _class != superclass {
            _class = class_getSuperclass(_class)
        }
        return _class == superclass
    }

    var classCount = objc_getClassList(nil, 0)
    var allClasses = UnsafeMutablePointer<AnyClass?>.allocate(capacity: Int(classCount))
    var autoreleasingAllClasses = AutoreleasingUnsafeMutablePointer<AnyClass?>(allClasses)
    classCount = objc_getClassList(autoreleasingAllClasses, classCount)

    for i in 0 ..< classCount {
      if let currentClass: AnyClass = allClasses[Int(i)],
        isClassSubclassOf(currentClass, superclass: UIColor.self),
        let originalMethod = class_getInstanceMethod(currentClass.self, #selector(getter: cgColor)) {
        method_setImplementation(originalMethod, imp_implementationWithBlock(
          unsafeBitCast(StaticVars.randomColorBlock, to: AnyObject.self))
        )
      }
    }

  }

}

Here we use objc_getClassList to look through every single class registered at runtime, swizzling the ones that derive from UIColor. This has two benefits. Firstly, because we’re not directly referencing any private derivatives of UIColor, we’re probably (probably) not going to get rejected from the App Store for using this code. Secondly it is better from the standpoint that it will be resistant to changes in iOS in the future. This is guaranteed to replace all UIColor instance getters with the new random color logic, satisfying my original goal.

With that, we have an answer to “Can I?” Perhaps the next article should go into “Should I?”

Spurious (Spatial) Correlations

Fri, 13 Apr 2018 00:00:00 GMT

We all know you can lie with graphs, but how well can you lie with spatial data?

In grad school, I took a course called Spatial Analysis. In it, I learned how to analyze geospatial data and look for trends. For our term project, we were each to take a spatial dataset relevant to our fields and apply our skills. I recalled Tyler Vigen’s Spurious Correlations, a website that draws correlations between (hopefully) unrelated data sets such as the divorce rate in Maine and margarine sales per capita. There is a wealth of freely available spatial data made available by government agencies and ESRI, so I thought it would be funny to try out similar techniques with these data sets for the state of Maine.

These are my results.

Eagles and K-Cups

With the exception of Washington County, eagles seem to make their homes in areas where household K-Cup ownership is at or higher than the national average.

Eagles and K-Cups

Schools and Aquifers

589 of Maine’s 795 schools are within 3 km of an aquifer. This means that if you are at a school, there is a 74% chance that you are within walking distance of a significant source of groundwater.

U.S. Cellular Coverage and Oil Spill Hotspots

US Cellular offers great coverage in areas where hazardous oil spills occur the most.

Cemeteries and Broadband Coverage

3875 of Maine’s 3929 cemeteries are located exactly within broadband coverage areas (98.63%).

Red Cross Facilities and Bedrock Formation Eras

In what I can only consider to be a deliberate move, Red Cross has built their facilities in Maine on top of bedrock formed before the Devonian era. This means that if you got into a time machine and went back to the period at which the bedrock was formed beneath any Red Cross facility, you would not find sharks, ferns, trees, insects, or land vertebrates.

Disclaimer:

These datasets are in no way scientifically rigorous.

Sources

Maine Department of Environmental Protection
Maine Office of GIS
University of Southern Maine’s Digital Atlas
ESRI Online Catalog

Jon's Blog

So You Want to Build a VR App

Why XCFrameworks Matter

OpenCV, InputArray, OutputArray, and You

Optional outputs with InputArray and OutputArray

Links

Randomizing UIColor with Method Swizzling

First Steps

Class Clusters

Things Get Worse

Spurious (Spatial) Correlations

Eagles and K-Cups

Schools and Aquifers

U.S. Cellular Coverage and Oil Spill Hotspots

Cemeteries and Broadband Coverage

Red Cross Facilities and Bedrock Formation Eras

Sources

Optional outputs with `InputArray` and `OutputArray`