Opening Portals to the Database: Making Core Data Easy

database

Written by Valentyn Kuznietsov, Maximilian Landsmann, and Yen-Chia Lin.

Working with databases on iOS can be difficult. Powerful but complex APIs can be a reason for subtle mistakes, slow development, and hard verification of code. For example, Core Data requires profound knowledge to use it safely (especially in the multithreading environment). In addition to that, educating every team member on how to use Core Data properly is not always possible. But the persistence layer should not be hard to use in the first place.

To solve this problem, we developed a database library on top of Core Data that is easy to master and hard to misuse. In this post, we will talk about the evolution of our database framework, describe our new solution, and showcase how our automation tools help to reduce manual work.

Runtastic Database Layer History  

To begin with, in 2009, we chose SQLite (libsqlite) as our database. This library implements the SQLite database engine, having exceptional performance and reliability characteristics.

However, using the “plain“ SQLite also meant that we had to implement certain features ourselves, such as entity versioning and migration to new database schemas. In addition to that, it would have been nice to utilize object-relational mapping (ORM), work with NSPredicates (no more SQL queries in code), have lazy data loading, and automatic relationship handling.

Later, in 2012, we took a closer look at Core Data. It is an extremely powerful framework, provided by Apple as well in the iOS SDK. Core Data offered us a solution to all requirements we had at that time.

As a first building block, we used the open-source Magical Record library and implemented multiple wrappers. Magical Record took care of the database stack setup (e.g. creating a persistence coordinator, setting a root main context for every model file), propagating changes from child NSManagedObjectContext to the root context, and some other aspects of Core Data usage.

With that, creating a user would look like this:

let context = DBUser.createNewContext()
var dbUser: DBUser?
context.performAndWait {
    dbUser = DBUser.createObject(in: context)
    dbUser?.id = "Batman"
    dbUser?.firstName = "Bruce"
    dbUser?.lastName = "Wayne"
        
    do {
        try context.save()
    } catch {
        // something went terribly wrong!
    }
}

In 2015, we suspected a critical flaw in the core of our database framework. It looked like we had problems with thread safety (due to the database framework complexity) that led to sporadic crashes of our apps and was almost impossible to reproduce.

Further analysis with enabled Xcode scheme launch argument -com.apple.CoreData.ConcurrencyDebug 1 revealed more issues – fortunately, they were consistent with the crash reports we received.

In 2016, we started thinking about the “persistence layer future“ framework. We defined the following goals:

  • it should be easy to master (no deep knowledge of Core Data in multi-threading environment needed)
  • no leakage of database technology in the public API
  • no cumbersome APIs (e.g. NSManagedObjectContext.performAndWait)
  • only lightweight data types in the public API (no more NSManagedObject subclass parameters) in the interface
  • the solution has to be thread safe

We also realized that it would have been impossible to switch our database layer easily. Many of our libraries already used the old database framework, and our apps were also quite big. It was apparent that the new solution has to work in parallel with the old system on the same database files, allowing us to slowly migrate our code, feature by feature.

In 2017, we started to work on this idea and built the first proof of concept. In 2018, we released it internally and shipped the Balance app with it.

What is Portal?

Portal is the codename of our new database layer, an abstraction for any database technology using the repository design pattern. Portal provides generic APIs for database CRUD functions. It allows developers to safely work with the database without digging into database specifics (or in our case, Core Data) at all.

The core idea behind Portal is that the repository is just a simple Swift protocol with only lightweight entities (lightweight structs or classes as opposed to heavy NSManagedObject) that can be freely passed around in code (including different threads and dispatch queues).

For example, in the case of the User entity, we will create a UserPortal. The Portal implementation correctly performs CRUD operations on the database, but converts them to simple entities before returning them through the public interface. Similarly, when there is a need to update an existing entity in the database, Portal accepts an entity as input, converts to a database object, and correctly performs all required operations.

All the layers outside of Portal (above orange part of the diagram) only work on lightweight entities. Usage of Core Data is only done inside the Portal layer (bottom blue part).

The usage of Portal looks like this:

let portal = UserPortal()
let user: User = ...
portal.create(user)
portal.update(user)
portal.delete(user)
let predicate    = NSPredicate(format: "firstName = %@ AND lastName = %@", "Bruce", "Wayne")
let count        = portal.count(matching: predicate)
let findResult   = portal.findAll(matching: predicate)
let deleteResult = portal.deleteAll(matching: predicate)

Portal Interface Overview

Portal Protocol

Portal is the main protocol of the solution. It defines generic database CRUD functions and has no dependency on any particular database technology (e.g. Core Data). Portal declares an associated type that must conform to the PortalEntity protocol. Other function variants like sorting and fetch limiting are also provided.

public protocol Portal {
    
    associatedtype EntityObject: PortalEntity
    func count(matching predicate: NSPredicate) -> Int
    func findAll(matching predicate: NSPredicate) -> ArrayResult<EntityObject>
    func create(_ entities: [EntityObject]) -> ArrayResult<EntityObject>
    func update(_ entities: [EntityObject]) -> ArrayResult<EntityObject>
    func deleteAll(matching predicate: NSPredicate) -> SingleResult<Bool>
}
extension Portal {
    public func count() -> Int { … }
    public func findFirst(predicate: NSPredicate) -> SingleResult<EntityObject> { … }
    …
}

Portal Entity Protocol

Every Portal entity must have a primary ID. Therefore, it must conform to this protocol.

public protocol PortalEntity {
    
    // customize primary Id property name
    static var entityIdPropertyName: String { get }
    // primary Id
    var entityId: String { get }
}

For example, the user entity could look like this:

final class User: PortalEntity {  
    let userId: String
    let firstName: String
    let lastName: String
    // MARK: - PortalEntity
    static var entityIdPropertyName: String {
        return "userId"
    }
    
    var entityId: String {
        return userId
    }
}

Portal Entity Convertible Protocol

Database objects that can be converted to a PortalEntity and vice versa, must implement this protocol. For Core Data, this protocol is usually implemented by NSManagedObject subclasses.

public protocol PortalEntityConvertible {
    
    associatedtype EntityObject: PortalEntity
    /// DB Object <-- Entity
    func update(with entity: EntityObject)
    
    /// DB Object --> Entity
    func toEntity() -> EntityObject
}

For the user object, the implementation looks like the following:

final class DBUser: NSManagedObject, PortalEntityConvertible {
    
    func update(with entity: User) {
        userId    = entity.userId
        firstName = entity.firstName
        lastName  = entity.lastName
    }
    
    func toEntity() -> User {
        return User(userId: userId,
                    firstName: firstName,
                    lastName: lastName)
    }
}

Core Data Portal Class

CoreDataPortal is an implementation of the Portal protocol with a generic database object. It encapsulates everything related to Core Data, and direct access to the Core Data APIs is only allowed in this class. Developers using Portal can also subclass CoreDataPortal to provide their custom implementations of their NSFetchRequest if needed.

CoreDataPortal provides the correct Core Data usage and is implemented once. It auto-creates private NSManagedObjectContext, wraps all calls inside performAndWait closures, and merges child contexts to the parent context in a thread-safe way.

The following is an example of how we used our old database framework:

func findBruceWaynes() -> [String] {
    let predicate = NSPredicate(format: "firstName == %@ AND lastName == %@", "Bruce", "Wayne")
    let context = DBUser.createNewContext()
    guard let users = DBUser.findAll(by: predicate, in: context) as? [DBUser] else {
        return []
    }
    var userIds: [String] = []
    context.performAndWait {
        userIds = users.map { $0.id }
    }
    return userIds
}

Notice that every access to Core Data must be wrapped inside context.performAndWait closures in order to be thread-safe. Because of the nature of closures, this provides some inconveniences if some properties are needed outside the closure.

And here is the same simple method using Portal:

func findBruceWaynes() -> [String] {
    let predicate = NSPredicate(format: "firstName == %@ AND lastName == %@", "Bruce", "Wayne")
    guard case let .success(users) = userPortal.findAll(matching: predicate) else {
        return []
    }
    return users.map { $0.id }
}

Creating a New Portal

To create a new Portal, these steps have to be performed:

  1. Define a Core Data model and generate NSManagedObject subclasses
  2. Create an entity
  3. Make the entity conform to PortalEntity
  4. Make the DB class conform to PortalEntityConvertible
  5. (Optional) Create a subclass of CoreDataPortal for custom fetch queries
  6. Repeat for every Core Data model file (at Runtastic we have more than 30)

This manual work is repetitive, error-prone, and takes time. In addition to that, when a developer would like to add a new property to the database model file, he would need to touch many files. These considerations bring us to automation and code generation.

Automation

To minimize the effort of transitioning to Portal or creating a new database model, we developed a Ruby gem alongside the Portal framework. With it, creating a Portal is quite easy. It is possible just to define a Core Data model, run the gem, and all necessary Portal related code is generated. All of the steps described above are automated by this gem.

It makes use of the tools MotoSwift and Sourcery, which rely on input templates written in the Stencil template language and generate Swift code.

MotoSwift’s purpose is to generate NSManagedObject subclasses from Core Data model files. Xcode generated classes are not modern, swifty database classes. We were influenced by some other blog posts in that regard.

Sourcery is a handy tool for code generation, as it scans your Swift code and generates code based on specific existing implementation details. We use it to generate the entities, protocol conformances, and CoreDataPortal subclasses as needed.

We view automation as a vital point in raising a large development team’s efficiency. Considering the number of database files we are migrating to Portal, automation allows for a much faster transition.

Conclusion

This approach saves a lot of time in the long run. Providing developers with proper and verified implementation behind simple and bulletproof APIs is worth the effort, as it greatly decreases chances of introducing nasty bugs in the future. Combined with automation and code generation, developing a database layer for a new feature is literally effortless.

Q&A

Question: How do you handle model change notifications?
Answer: It is possible and it is already implemented. However, it was not included in Portal 1.0, as we would like to perform additional tests before allowing it for production.

Question: What about lazy loading?
Answer: Unfortunately, this aspect of Core Data is lost. Nevertheless, our research had shown that this use case is almost never needed in our codebase.

Question: What about database migration?
Answer: Lightweight migrations are performed by Core Data itself, heavyweight migrations must be done by a developer outside of Portal. Portal does not perform any migrations, at least not in the current version.

Question: Is Objective-C supported?
Answer: Yes.

Question: Are there plans to open-source it?
Answer: Not at the moment, as we would like to gather more feedback by using it internally for some time. In addition to that, unfortunately, the internals of CoreDataPortal are tied to our current database stack (as it has to work in parallel with our old framework) – this code must be carefully reviewed or removed before the public release.

***

RATE THIS ARTICLE NOW

Runtastic Tech Team We are made up of all the tech departments at Runtastic like iOS, Android, Backend, Infrastructure, DataEngineering, etc. We’re eager to tell you how we work and what we have learned along the way. View all posts by Runtastic Tech Team »