General Findings About NSPersistentCloudKitContainer

Update, 29 June 2022: Added sections 10 & 11 and note in section 4 since receiving feedback from other users and extra information from WWDC 22.

We have recently released Streaks 7.5, which changes how iCloud sync works in the app so it uses NSPersistentCloudKitContainer. This is a framework Apple released in iOS 13, which is used to automatically sync data in a Core Data database to all of your other devices.

It is a really elegant solution that mostly works well, but in rolling out this update, we've come to learn a few things about it.

1. Monitoring Events

In iOS 14, Apple added a notification so you can determine what NSPersistentCloudKitContainer is actually doing under the hood:

• NSPersistentCloudKitContainer.eventChangedNotification

Having access to this information is critical to tell your users when data is being synced. Because of this, if you're adopting NSPersistentCloudKitContainer, bump your minimum target to iOS 14.

All devices that support iOS 13 also support iOS 14. The only caveat here is that is you would also need to a minimum macOS target of Big Sur, since this notification isn't available on Catalina.

let observer = NotificationCenter.default.addObserver(forName: NSPersistentCloudKitContainer.eventChangedNotification, object: nil, queue: .main) { [weak self] notification in

    guard let event = notification.userInfo?[NSPersistentCloudKitContainer.eventNotificationUserInfoKey] as? NSPersistentCloudKitContainer.Event else {
        return
    }

    let isFinished = event.endDate != nil
    
    switch (event.type, isFinished) {
        case (.import, false): // Started downloading records
        case (.import, true): // Finished downloading records

        case (.export, false): // Started uploading records
        case (.export, true): // Finished uploading records

        ...
    }
}

Also, just because an event finishes, doesn't mean everything is in sync yet. You may experience many consecutive events of the same type all within a short timeframe.

2. Know The Errors

When handling events such as in the code above you can access the error property of the event.

So far, I've only seen two different categories of error:

1. CloudKit errors. In this instance, the error can be cast to CKError.
2. Generic errors with a domain of NSCocoaErrorDomain.

CloudKit Errors

If an error occurs, it will frequently give you a CKError, but these aren't necessarily the same as when accessing CloudKit directly. For example, if you receive a CKError.Code.partialFailure, it's not possible retrieve the errors that would normally be contained.

if let error = event.error as? CKError {
    switch error.code {
        case .quotaExceeded: // iCloud is full
        ...
    }
}

Speaking of the .quotaExceeded error: this is a common problem, since non-paying users only receive 5GB of iCloud storage. However, this can be a tricky issue to detect, as the error might be reported as a .partialFailure.

We still haven't come up with an elegant solution for this, but one strategy we have considered it to access CloudKit directly to see what happens. Try to create a new CKRecord in an unrelated record type so you can semi-reliably detect the error. This would go something like:

1. Error occurs in NSPersistentCloudKitContainer event
2. If it's .quotaExceeded, tell the user.
3. Otherwise, if it's .partialFailure (or otherwise unknown error), create a record on a separate CloudKit record type and/or zone.
4. Check if an error is returned to further diagnose a possible problem
5. When NSPersistentCloudKitContainer returns successfully in future, assume the prior error is cleared.

This feels slightly hacky to me though, but still likely useful.

Other Errors

For errors that aren't a CKError, I haven't been able to find documentation for many of the errors that occur, but these are the ones I've discovered (I'm sure there are more).

let nsError = event.error as NSError

// nsError.domain is most likely `NSCocoaErrorDomain`

switch nsError.code {
    case 133000: // Data integrity error
    case 133020: // Merge error
    case 134301: // ???
    case 134400: // Not logged in to iCloud
    case 134404: // Constraint conflict
    case 134405: // iCloud account changed
    case 134407: // ???
    case 134419: // Too much work to do
    case 134421: // Unhandled exception
}

The most interesting ones here are:

• 134419. This seems to occur when the container has too much work to do. In other words, system resources are being managed so the export/import cannot be completed.
  
  I've seen this most frequently on Apple Watch. In this case, the data will eventually sync, but it's unclear when the system will try again (some reports indicate this may be while the Apple Watch is on the charging puck).
  
  It is worth mentioning this I have seen this on iOS devices also, but primarily on Apple Watch.
• 134400. There seems to be an iOS bug that periodically occurs where a given app thinks the user isn't logged in to iCloud. I'm not 100% sure if this is what occurs if the app isn't enabled in iCloud settings, but I've definitely frequently seen this where everything is configured correctly. I believe this is a system bug, as I've seen it often when accessing CloudKit directly too (CKError.code.notAuthenticated).

3. Apple Watch Handling

There's some quirky behaviour with Apple Watch, which can make it challenging to keep everything up-to-date between the Apple Watch and its linked iPhone.

1. Firstly, the biggest issue is the 134419 error described above, especially when the app is freshly installed and it is importing all of the existing data.
2. If the app isn't a standalone app (i.e. you must have the app installed on the paired iPhone also), then only one of these devices receives remote push notificaitons.
   
   For example, if you also have an iPad and make a change on there, then when the data changes on CloudKit, all other devices should receive a silent notification so they can wake up and fetch new data.
   
   If the iPhone is awake/unlocked, it will receive that silent notificaiton, otherwise the Apple Watch receives it (I may be slightly incorrect on this, but hopefully that describes the overall behaviour).
   
   This means that either the Apple Watch or the iPhone will have the most up-to-date information, but not both. At least, not right away.

Initially with Streaks 7.5, we decided to not use Watch Connectivity at all for keeping task data in sync. We've since adopted a hybrid approach, where the iPhone sends its most up-to-date snapshot to the Apple Watch, and then Streaks for Apple Watch determines whether to use the data from the phone or the data synced from iCloud.

There's definitely upsides to implementing this hybrid approach, but things get a little tricky if the user switches frequently between using your iPhone app and Apple Watch app.

Because of this, I believe this approach should prioritise keeping complications up to date, since that is the data the user is most likely to quickly see on their Apple Watch after using the iPhone.

4. Background Updating

NSPersistentCloudKitContainer can update your database while the app is in the background (for example, after receiving a silent push notification).

However, it appears that this updating is performed using a queue with a quality-of-service priority of .utility , and it's not possible to making this run at a higher priority.

Utility tasks have a lower priority than default, user-initiated, and user-interactive tasks, but a higher priority than background tasks. Assign this quality-of-service class to tasks that do not prevent the user from continuing to use your app. For example, you might assign this class to long-running tasks whose progress the user does not follow actively.

This means that your data will update in the background, but may not be immediately processed. Therefore, if you are trying to keep your WidgetKit widgets up-to-date or your Apple Watch complications up-to-date, this may not always happen.

Update, 29 June 2022: One thing Apple recommend is to use to background tasks from your main app to request more time to keep data in sync across devices.

5. Priming the Database Records

When configuring CloudKit, the typical process is:

1. Create your database schema (.xcdatamodel)
2. Run your app from Xcode (which by default will use "development" CloudKit container)
3. Create an entity and populate the properties
4. The necessary record type will be created in development CloudKit
5. In CloudKit Dashboard, deploy the record types to production

It can be easy to forget to deploy new fields to your record types in CloudKit, and this can be really difficult to track down.

Therefore, when developing your app, ensure you call NSPersistentCloudKitContainer.initializeCloudKitSchema.

6. Update Database Records Sparingly

When you make a change to a database record, an .export event is triggered on the current device. This subsequently triggers an .import event on each of your other devices.

Since NSPersistentCloudKitContainer uses transactions to keep the data in sync, it needs to propagate through every database changed one by one.

This means:

1. If you install the app on a new device, that device might have a lot of data it needs to import on first run. This initial import can take minutes, or longer, and is a frequent cause of the 134419 error.
2. This problem is especially bad on Apple Watch, since it is so heavily resource-constrained, but also generally gets its Internet access from the paired iPhone, so the network may cut out at times.

Additionally, minimise the calls to NSManagedObjectContext.save(). I use a method which roughly looks like:

extension NSManagedObjectContext {
    func saveIfChanged() {
        guard self.hasChanges else {
            return
        }
        
        do {
            try self.save()
        }
        catch {
            // Handle error
        }
    }
 }

Be aware that hasChanges will return true, even if no data has changed:

entity.foo = "Bar"
// context.hasChanges is true
context.saveIfChanges()

// context.hasChanges is false

entity.foo = "Bar"
// context.hasChanges is true, even though `foo` is unchanged`

Instead, do something like:

let newValue = "Bar"

if entity.foo != newValue {
    entity.foo = newValue
}

There's probably a more elegant way to do this, and I'm also not 100% sure what impact an unnecessary save like this will have on the database or on CloudKit, but since it's unnecessary, might as well avoid it!

Avoid Update Loops

Ensure that you don't accidentally trigger an endless update loop between devices.

For example, if you make database changes after an .import event completes, that will trigger a subsequent import on the originating device, and so on.

7. Updating Your View Model

After receiving new data from another device, ensure that the new data is presented as necessary to the user.

To achieve this, use the standard Core Data notifications and tools, such as:

• Notification.Name.NSManagedObjectContextObjectsDidChange
• Notification.Name.NSManagedObjectContextDidMergeChangesObjectIDs
• NSFetchedResultsController

8. Keeping App Extension Up-To-Date

In Streaks, the Core Data files are stored in a shared app group so other extensions can also read from the database:

• Intents extension
• Intents UI extension
• Notifications extension
• Widgets extension
• Today extension (now deprecated, but we still support it).

This is relatively straightforward to achieve, but there are some caveats:

1. The Intents and Today extension can both write to the database
2. All extensions need to be able to read the database
3. If Intents/Today update the database, the main app needs to see this data.

To correctly handle database changes into an extension that's already running, you need to handle the Notification.Name.NSPersistentStoreRemoteChange event.

I recommend reading Persistent History Tracking in Core Data by Antoine van der Lee about this. However, Apple recommend not removing the transactions at all when using NSPersistentCloudKitContainer, as it may cause a reset, resulting in all the data needing to be downloaded again.

9. Core Data Merge Policies

Depending on specifically how you use Core Data, you may run into merge errors if your NSManagedObjectContext merge policies are not correctly set.

This manifests itself in two ways when calling try context.save():

• An exception is thrown containing details of the merge conflict
• It will appear as though your data isn't syncing across devices

I've found that NSMergePolicy.mergeByPropertyObjectTrump seems to work the best to handle this error. You need to set the merge policy on all contexts.

In order to use Core Data correctly, you should frequently be using background contexts, created either using newBackgroundContext() or performBackgroundTask(). This allows you to offload work to a background thread, rather than blocking the UI thread.

To achieve this - and avoid excessive refactoring - extend NSPersistentCloudKitContainer and override these methods:

class AppNSPersistentCloudKitContainer: NSPersistentCloudKitContainer {
    open override func newBackgroundContext() -> NSManagedObjectContext {
        let context = super.newBackgroundContext()
        context.mergePolicy = .mergeByPropertyObjectTrump

        return context
    }

    override func performBackgroundTask(_ block: @escaping (NSManagedObjectContext) -> Void) {
        super.performBackgroundTask { context in
            context.mergePolicy = .mergeByPropertyObjectTrump
            block(context)
        }
    }
}

Doing this, your setup would look like:

let container = AppNSPersistentCloudKitContainer()

// Configure container, including:
container.viewContext.mergePolicy = .mergeByPropertyObjectTrump

And your usage would look like:

let context = container.newBackgroundContext()
context.perform {
    // Perform queries here
}

// or ...

container.performBackgroundTask { context in
    // Perform queries here
}

10. Changing iCloud Account

One issue we have experienced is that the local Core Data database was periodically being cleared, making it appear to the user that all of their data was lost.

Eventually, we discovered that if the iCloud account changes on a device (for example, user logs out or toggles off an app's access to iCloud), then NSPersistentCloudKitContainer will remove the data. This is intentional behaviour, intended as a privacy feature.

The problem we were experiencing was that iOS was (incorrectly) telling the app that its access to iCloud had been revoked, even though the user hadn't logged out or toggled off permission to access iCloud. This is the 134400 error mentioned above.

After a short delay (up to a day or so), iOS would once again allow access to iCloud, resulting in the user's data being downloaded again from iCloud. However, to the user it just looked like their data was gone.

To work around this issue, we use CKContainer.accountStatus to check if the user has access to iCloud (as well as observer Notification.Name.CKAccountChanged).

You can disable syncing to iCloud by setting cloudKitContainerOptions to nil (you can use still use NSPersistentCloudKitContainer even when not syncing to iCloud).

let description: NSPersistentStoreDescription

if accountStatus == .available {
    description.cloudKitContainerOptions = .init(containerIdentifier: "...")
}
else {
    description.cloudKitContainerOptions = nil
}

11. Extensions Should Operate Offline

In Section 8 above, I talked about keeping app extensions up-to-date. One thing not mentioned is how to handle cloud sync inside the extensions.

Apple recommends only syncing from the main app, for two primary reasons:

1. Extensions rarely (if ever) get enough time to complete an export operation
2. Your extension will end up competing with your main app for resources

To achieve this, set NSPersistentStoreDescription.cloudKitContainerOptions to nil in your extension.

In reality, we have enabled sync in our Intents extension, so when users use Shortcuts, it will update other devices faster. To achieve this, we also had to add the appropriate entitlements to the extension.

More Info Needed!

If you have any further hints and tips for using NSPersistentCloudKitContainer, please reach out either to @qzervaas on Twitter, or at hello@crunchybagel.com, as I will aim to keep this article updated.