This repository has no description
0

Configure Feed

Select the types of activity you want to include in your feed.

feat: self-contained iOS MLKit — zero consumer setup (v4.15.1)

The root problem with v4.15.0's MLKit shipment was that downstream apps
had to: (1) add -ObjC to OTHER_LDFLAGS, (2) copy MLKit resource bundles
into the app bundle via an Xcode build phase, (3) force the KMP framework
to be dynamic. This PR eliminates all three consumer-side steps.

How:

1. `-ObjC` propagation — the cinterop mlkitAccurate.def already carries
`linkerOpts = -ObjC`, which Kotlin/Native applies at the consumer's
framework link step. For the KMP default (dynamic framework), this
preserves MLKit's ObjC class/category metadata inside the resulting
dylib; the consumer's app-link step just embeds the dylib, so it
doesn't need -ObjC of its own. No change needed in consumer
OTHER_LDFLAGS.

2. Resource bundles — MLKit's .tflite / .binarypb weights (~9 MB) are
now staged (Gradle `stageMlkitResources` task) into the library's
iosMain Compose Multiplatform resource set. Compose packages them
into the consumer's iOS app bundle under
`compose-resources/composeResources/...generated.resources/files/mlkit/`.
At first MlKitPose.detect() call, the new
`MlKitResourceBootstrap.ensureInitialized()` runs once — reads the
bundled bytes via `Res.readBytes`, writes them to
`NSCachesDirectory()/MLKitResources/<BundleName>.bundle/<file>`,
and registers that path with a small NSBundle swizzle.

3. NSBundle swizzle — `src/nativeInterop/mlkitRedirect/MLKitResourceRedirect.m`
swizzles `-[NSBundle URLForResource:withExtension:]` at +load time.
When MLKit's internal lookup for its three resource bundles
(MLKitPoseDetectionAccurateResources / MLKitPoseDetectionCommonResources /
MLKitXenoResources) falls through the main bundle, the swizzle
redirects to the Caches-directory copy the Kotlin bootstrap wrote.
Compiled to libMLKitRedirect.a via the new `compileMlkitRedirect`
Gradle task and folded into the MLKit cinterop .def's
staticLibraries list.

4. Dynamic framework — KMP's default is dynamic, so no consumer action
is required unless they explicitly set `isStatic = true`. The
sample iosApp's composeApp/build.gradle.kts is simplified to the
default dynamic config, and the Xcode iosApp target's OTHER_LDFLAGS
`-ObjC` entry + "Copy MLKit Resource Bundles" shell-script build
phase are removed — proving the library is self-contained.

Result: downstream consumers (kima) can bump from 4.14.0 → 4.15.1 with
zero other changes and MLKit pose detection works on iOS.

Version bumped to 4.15.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

+265 -38
+79 -4
posedetection/build.gradle.kts
··· 4 4 5 5 mavenPublishing { 6 6 publishToMavenCentral(SonatypeHost.CENTRAL_PORTAL) 7 - coordinates("com.performancecoachlab.posedetection", "posedetection-compose", "4.15.0") 7 + coordinates("com.performancecoachlab.posedetection", "posedetection-compose", "4.15.1") 8 8 9 9 pom { 10 10 name.set("Pose Detection") ··· 148 148 val mlkitStagingDir = layout.buildDirectory.dir("mlkit-staging") 149 149 val mlkitArchivesDir = layout.buildDirectory.dir("mlkit-archives") 150 150 val mlkitDefFile = layout.buildDirectory.file("mlkit-archives/mlkitAccurate.def") 151 + val mlkitRedirectSrc = 152 + project.file("src/nativeInterop/mlkitRedirect/MLKitResourceRedirect.m") 153 + val mlkitRedirectDir = layout.buildDirectory.dir("mlkit-redirect") 154 + val mlkitRedirectLib = 155 + layout.buildDirectory.file("mlkit-redirect/ios_arm64/libMLKitRedirect.a") 156 + val mlkitComposeResDir = layout.buildDirectory.dir("mlkit-compose-resources") 157 + 158 + // Compile our NSBundle swizzle for iosArm64 and archive into libMLKitRedirect.a. 159 + // The cinterop .def includes this static library alongside MLKit's, so the 160 + // swizzle gets linked into the consumer's KMP framework automatically. 161 + val compileMlkitRedirect = tasks.register<Exec>("compileMlkitRedirect") { 162 + group = "mlkit" 163 + description = "Compile NSBundle MLKit redirect swizzle to static archive." 164 + inputs.file(mlkitRedirectSrc) 165 + outputs.file(mlkitRedirectLib) 166 + val out = mlkitRedirectLib.get().asFile 167 + val obj = out.resolveSibling("MLKitResourceRedirect.o") 168 + out.parentFile.mkdirs() 169 + commandLine( 170 + "sh", "-c", 171 + "xcrun --sdk iphoneos clang " + 172 + "-c -fobjc-arc -fmodules " + 173 + "-arch arm64 -mios-version-min=16.2 " + 174 + "${mlkitRedirectSrc.absolutePath} -o ${obj.absolutePath} && " + 175 + "xcrun ar rcs ${out.absolutePath} ${obj.absolutePath}" 176 + ) 177 + } 178 + 179 + // Copy MLKit resource bundles (.tflite / .binarypb) from the synced pods into 180 + // the library's iosMain Compose Multiplatform resources, so they're packaged 181 + // into the consumer app bundle automatically. At runtime, Kotlin extracts them 182 + // to NSCachesDirectory and the NSBundle swizzle redirects MLKit's lookups. 183 + val stageMlkitResources = tasks.register<Copy>("stageMlkitResources") { 184 + group = "mlkit" 185 + description = "Stage MLKit resource bundles as Compose MP resources." 186 + dependsOn("syncMlkitBinaries") 187 + val pods = mlkitStagingDir.get().asFile.resolve("Pods") 188 + from(pods.resolve("MLKitPoseDetectionAccurate/Resources/MLKitPoseDetectionAccurateResources")) { 189 + into("files/mlkit/MLKitPoseDetectionAccurateResources") 190 + } 191 + from(pods.resolve("MLKitPoseDetectionCommon/Resources/MLKitPoseDetectionCommonResources")) { 192 + into("files/mlkit/MLKitPoseDetectionCommonResources") 193 + } 194 + from(pods.resolve("MLKitXenoCommon/Frameworks/MLKitXenoCommon.framework/MLKitXenoResources.bundle")) { 195 + into("files/mlkit/MLKitXenoResources") 196 + } 197 + into(mlkitComposeResDir) 198 + } 199 + 200 + // Wire the staged MLKit resources into the Compose Multiplatform resource set 201 + // for the ios source tree. Compose packages these into the consumer's app 202 + // bundle at `compose-resources/<pkg>.generated.resources/files/mlkit/...`. 203 + compose.resources { 204 + customDirectory( 205 + sourceSetName = "iosMain", 206 + directoryProvider = mlkitComposeResDir, 207 + ) 208 + } 209 + 210 + // Compose's `generateResourceAccessors*` tasks read from the resource dir, so 211 + // make them wait for the stage task (which populates the dir). 212 + tasks.matching { it.name.startsWith("generateResourceAccessors") } 213 + .configureEach { dependsOn(stageMlkitResources) } 214 + tasks.matching { it.name.startsWith("generateExpectResourceCollectors") } 215 + .configureEach { dependsOn(stageMlkitResources) } 216 + tasks.matching { it.name.startsWith("convertXmlValueResources") } 217 + .configureEach { dependsOn(stageMlkitResources) } 218 + tasks.matching { it.name.startsWith("copyNonXmlValueResources") } 219 + .configureEach { dependsOn(stageMlkitResources) } 220 + tasks.matching { it.name.startsWith("prepareComposeResources") } 221 + .configureEach { dependsOn(stageMlkitResources) } 151 222 152 223 val syncMlkitBinaries = tasks.register<Exec>("syncMlkitBinaries") { 153 224 group = "mlkit" ··· 165 236 val generateMlkitDefFile = tasks.register("generateMlkitDefFile") { 166 237 group = "mlkit" 167 238 description = "Generate the MLKit cinterop .def with absolute archive paths." 168 - dependsOn(syncMlkitBinaries) 239 + dependsOn(syncMlkitBinaries, compileMlkitRedirect) 169 240 outputs.file(mlkitDefFile) 170 241 // Capture as locals so the closure doesn't hold project-level refs — 171 242 // configuration-cache-safe. 172 243 val pods = mlkitStagingDir.get().asFile.resolve("Pods").absolutePath 173 244 val archives = mlkitArchivesDir.get().asFile.absolutePath 245 + val redirectArchives = mlkitRedirectDir.get().asFile.absolutePath 174 246 val defFile = mlkitDefFile.get().asFile 175 247 doLast { 176 248 val defContent = buildString { ··· 191 263 "libMLKitPoseDetectionAccurate.a libMLKitPoseDetectionCommon.a " + 192 264 "libMLKitVision.a libMLKitCommon.a libMLImage.a libMLKitXenoCommon.a " + 193 265 "libGTMSessionFetcher.a libGoogleDataTransport.a libGoogleToolboxForMac.a " + 194 - "libGoogleUtilities.a libFBLPromises.a libnanopb.a" 266 + "libGoogleUtilities.a libFBLPromises.a libnanopb.a libMLKitRedirect.a" 195 267 ) 196 - appendLine("libraryPaths.ios_arm64 = $archives/ios_arm64") 268 + appendLine("libraryPaths.ios_arm64 = $archives/ios_arm64 $redirectArchives/ios_arm64") 197 269 appendLine( 198 270 "linkerOpts = -ObjC -lc++ -lsqlite3 -lz " + 199 271 "-framework Accelerate -framework CoreML" 200 272 ) 273 + // Expose the swizzle control function to Kotlin. 274 + appendLine("---") 275 + appendLine("void mlkit_set_resource_dir(const char *path);") 201 276 } 202 277 defFile.writeText(defContent) 203 278 }
+6
posedetection/src/iosArm64Main/kotlin/com/performancecoachlab/posedetection/camera/MlKitPose.kt
··· 73 73 cropHPx: Float, 74 74 timestamp: Long, 75 75 ): Skeleton? { 76 + // Extract MLKit's resource bundles from Compose MP resources to 77 + // Caches and point the NSBundle swizzle at them — so MLKit finds 78 + // its .tflite / .binarypb files without any app-bundle build-phase 79 + // script on the consumer side. 80 + MlKitResourceBootstrap.ensureInitialized() 81 + 76 82 // Pre-rotate the pixels so the CGImage handed to MLKit is already 77 83 // physically upright at oriented (logical/display) dimensions. MLKit 78 84 // returns landmarks in the input CGImage's raw pixel space, so when
+97
posedetection/src/iosArm64Main/kotlin/com/performancecoachlab/posedetection/camera/MlKitResourceBootstrap.kt
··· 1 + package com.performancecoachlab.posedetection.camera 2 + 3 + import cocoapods.MLKitPoseDetectionAccurate.mlkit_set_resource_dir 4 + import kotlinx.cinterop.ExperimentalForeignApi 5 + import kotlinx.cinterop.addressOf 6 + import kotlinx.cinterop.usePinned 7 + import kotlinx.coroutines.runBlocking 8 + import org.jetbrains.compose.resources.ExperimentalResourceApi 9 + import org.jetbrains.compose.resources.InternalResourceApi 10 + import posedetection.posedetection.generated.resources.Res 11 + import platform.Foundation.NSData 12 + import platform.Foundation.NSFileManager 13 + import platform.Foundation.NSURL 14 + import platform.Foundation.NSURL.Companion.fileURLWithPath 15 + import platform.Foundation.NSUserDomainMask 16 + import platform.Foundation.NSCachesDirectory 17 + import platform.Foundation.create 18 + import platform.Foundation.writeToFile 19 + 20 + /** 21 + * One-time bootstrap: extracts MLKit's bundled resource files (packaged into 22 + * the library's Compose Multiplatform resources by Gradle's 23 + * `stageMlkitResources` task) to the app's Caches directory, then registers 24 + * that path with the NSBundle swizzle (via `mlkit_set_resource_dir`). From 25 + * that point on, MLKit's `[NSBundle URLForResource:withExtension:]` calls for 26 + * its resource bundles fall back to the Caches copy, so the consumer's iOS 27 + * app bundle doesn't need any build-phase script to copy them in. 28 + * 29 + * Intentionally synchronous at first call — runBlocking lets us stay inside 30 + * the FrameProcessor's non-suspending captureOutput pipeline. The extraction 31 + * is ~9 MB and happens once. 32 + */ 33 + @OptIn(ExperimentalForeignApi::class, ExperimentalResourceApi::class, InternalResourceApi::class) 34 + internal object MlKitResourceBootstrap { 35 + // Matches the sub-tree Gradle's stageMlkitResources produces under 36 + // build/mlkit-compose-resources/files/mlkit/... 37 + private val BUNDLE_FILES: Map<String, List<String>> = mapOf( 38 + "MLKitPoseDetectionAccurateResources" to listOf( 39 + "pose_landmark_detector_accurate.tflite", 40 + "pose_person_detector_accurate.tflite", 41 + ), 42 + "MLKitPoseDetectionCommonResources" to listOf( 43 + "pose_tracking_graph.binarypb", 44 + "pose_non_tracking_graph.binarypb", 45 + ), 46 + "MLKitXenoResources" to listOf( 47 + "yuv_to_rgb_graph.binarypb", 48 + ), 49 + ) 50 + 51 + // `lazy` on Kotlin/Native is thread-safe by default — evaluates once. 52 + // Accessing .value triggers the one-time extraction + swizzle-setter. 53 + private val bootstrap: Unit by lazy { 54 + runBlocking { extract() } 55 + } 56 + 57 + fun ensureInitialized() { 58 + bootstrap 59 + } 60 + 61 + private suspend fun extract() { 62 + val fm = NSFileManager.defaultManager 63 + val cachesRoot = (fm.URLsForDirectory(NSCachesDirectory, NSUserDomainMask).firstOrNull() 64 + as? NSURL)?.path ?: return 65 + val mlkitDir = "$cachesRoot/MLKitResources" 66 + fm.createDirectoryAtPath( 67 + mlkitDir, 68 + withIntermediateDirectories = true, 69 + attributes = null, 70 + error = null, 71 + ) 72 + 73 + for ((bundleName, files) in BUNDLE_FILES) { 74 + val bundlePath = "$mlkitDir/$bundleName.bundle" 75 + fm.createDirectoryAtPath( 76 + bundlePath, 77 + withIntermediateDirectories = true, 78 + attributes = null, 79 + error = null, 80 + ) 81 + for (file in files) { 82 + val resourcePath = "files/mlkit/$bundleName/$file" 83 + val bytes = Res.readBytes(resourcePath) 84 + val outPath = "$bundlePath/$file" 85 + bytes.usePinned { pinned -> 86 + val data = NSData.create( 87 + bytes = pinned.addressOf(0), 88 + length = bytes.size.toULong(), 89 + ) 90 + data.writeToFile(outPath, atomically = true) 91 + } 92 + } 93 + } 94 + 95 + mlkit_set_resource_dir(mlkitDir) 96 + } 97 + }
+76
posedetection/src/nativeInterop/mlkitRedirect/MLKitResourceRedirect.m
··· 1 + // Redirects MLKit's NSBundle URLForResource:withExtension: lookups for its 2 + // resource bundles (MLKitPoseDetectionAccurateResources, 3 + // MLKitPoseDetectionCommonResources, MLKitXenoResources) to a directory the 4 + // library's Kotlin init code populates at runtime. Without this, MLKit's 5 + // `.tflite` / `.binarypb` weights have to be copied into the consumer's app 6 + // bundle via an Xcode build phase — forcing per-app setup. 7 + // 8 + // Kotlin side calls mlkit_set_resource_dir() with a Caches-directory path 9 + // after extracting the resource files there. The swizzled URLForResource 10 + // method first defers to the original (so non-MLKit lookups are unaffected), 11 + // then falls back to the registered directory for the specific bundle names 12 + // MLKit queries. 13 + 14 + #import <Foundation/Foundation.h> 15 + #import <objc/runtime.h> 16 + 17 + static NSString *_mlkitResourceDir = nil; 18 + 19 + __attribute__((visibility("default"))) 20 + void mlkit_set_resource_dir(const char *path) { 21 + if (path) { 22 + _mlkitResourceDir = [NSString stringWithUTF8String:path]; 23 + } else { 24 + _mlkitResourceDir = nil; 25 + } 26 + } 27 + 28 + static NSSet<NSString *> *mlkitBundleNames(void) { 29 + static NSSet *names; 30 + static dispatch_once_t once; 31 + dispatch_once(&once, ^{ 32 + names = [NSSet setWithObjects: 33 + @"MLKitPoseDetectionAccurateResources", 34 + @"MLKitPoseDetectionCommonResources", 35 + @"MLKitXenoResources", 36 + nil]; 37 + }); 38 + return names; 39 + } 40 + 41 + @interface NSBundle (PoseDetectionMLKitRedirect) 42 + @end 43 + 44 + @implementation NSBundle (PoseDetectionMLKitRedirect) 45 + 46 + + (void)load { 47 + static dispatch_once_t once; 48 + dispatch_once(&once, ^{ 49 + Method origM = class_getInstanceMethod(self, 50 + @selector(URLForResource:withExtension:)); 51 + Method newM = class_getInstanceMethod(self, 52 + @selector(pd_mlkit_URLForResource:withExtension:)); 53 + method_exchangeImplementations(origM, newM); 54 + }); 55 + } 56 + 57 + - (NSURL *)pd_mlkit_URLForResource:(NSString *)name 58 + withExtension:(NSString *)ext { 59 + // Swizzled — this now calls the ORIGINAL implementation. 60 + NSURL *fromOriginal = [self pd_mlkit_URLForResource:name withExtension:ext]; 61 + if (fromOriginal) return fromOriginal; 62 + 63 + if (!_mlkitResourceDir || !name || !ext) return nil; 64 + if (![ext isEqualToString:@"bundle"]) return nil; 65 + if (![mlkitBundleNames() containsObject:name]) return nil; 66 + 67 + NSString *candidate = [_mlkitResourceDir stringByAppendingPathComponent: 68 + [NSString stringWithFormat:@"%@.%@", name, ext]]; 69 + BOOL isDir = NO; 70 + if ([[NSFileManager defaultManager] fileExistsAtPath:candidate isDirectory:&isDir] && isDir) { 71 + return [NSURL fileURLWithPath:candidate isDirectory:YES]; 72 + } 73 + return nil; 74 + } 75 + 76 + @end
+5 -7
sample/composeApp/build.gradle.kts
··· 21 21 ).forEach { target -> 22 22 target.binaries.framework { 23 23 baseName = "ComposeApp" 24 - // Dynamic framework so MLKit's ObjC class / category metadata is 25 - // preserved in a dylib's __objc_classlist / __objc_catlist 26 - // sections (static archives strip category .o files during the 27 - // framework repack, causing 'unrecognized selector' at runtime). 24 + // Default KMP behavior (dynamic framework). Library's cinterop 25 + // linkerOpts supply -ObjC at framework link time to preserve MLKit 26 + // ObjC category metadata inside the dylib; the consumer iOS app 27 + // just loads the framework at runtime and categories are already 28 + // registered. No consumer-side flags / build phases required. 28 29 isStatic = false 29 - if (target.name == "iosArm64") { 30 - linkerOpts += "-ObjC" 31 - } 32 30 } 33 31 } 34 32
+2 -27
sample/iosApp/iosApp.xcodeproj/project.pbxproj
··· 88 88 A93A953329CC810C00F8E227 /* Sources */, 89 89 A93A953429CC810C00F8E227 /* Frameworks */, 90 90 A93A953529CC810C00F8E227 /* Resources */, 91 - ABCDEF0123456789ABC00003 /* Copy MLKit Resource Bundles */, 92 91 ); 93 92 buildRules = ( 94 93 ); ··· 164 163 runOnlyForDeploymentPostprocessing = 0; 165 164 shellPath = /bin/sh; 166 165 shellScript = "export JAVA_HOME=/Applications/Android\\ Studio.app/Contents/jbr/Contents/Home/\nexport JDK_HOME=/Applications/Android\\ Studio.app/Contents/jbr/Contents/\ncd \"$SRCROOT/..\"\n./gradlew :sample:composeApp:embedAndSignAppleFrameworkForXcode\n"; 167 - }; 168 - ABCDEF0123456789ABC00003 /* Copy MLKit Resource Bundles */ = { 169 - isa = PBXShellScriptBuildPhase; 170 - buildActionMask = 2147483647; 171 - name = "Copy MLKit Resource Bundles"; 172 - files = ( 173 - ); 174 - inputFileListPaths = ( 175 - ); 176 - inputPaths = ( 177 - ); 178 - outputFileListPaths = ( 179 - ); 180 - outputPaths = ( 181 - ); 182 - runOnlyForDeploymentPostprocessing = 0; 183 - shellPath = /bin/sh; 184 - shellScript = "set -e\nMLKIT_PODS=\"$SRCROOT/../../posedetection/build/mlkit-staging/Pods\"\nDEST=\"$TARGET_BUILD_DIR/$PRODUCT_NAME.app\"\ncopy_bundle() {\n # $1 = pod resource dir, $2 = bundle name\n local src=\"$1\"\n local bundle_name=\"$2\"\n if [ ! -d \"$src\" ]; then\n echo \"warning: MLKit resources not at $src\"\n return\n fi\n local dst=\"$DEST/$bundle_name.bundle\"\n rm -rf \"$dst\"\n mkdir -p \"$dst\"\n cp -R \"$src/\" \"$dst/\"\n # Codesign the bundle if signing is enabled\n if [ \"${CODE_SIGNING_REQUIRED:-YES}\" = \"YES\" ] && [ -n \"${EXPANDED_CODE_SIGN_IDENTITY:-}\" ]; then\n /usr/bin/codesign --force --sign \"$EXPANDED_CODE_SIGN_IDENTITY\" --preserve-metadata=identifier,entitlements --timestamp=none \"$dst\" 2>/dev/null || true\n fi\n}\n\ncopy_bundle \"$MLKIT_PODS/MLKitPoseDetectionAccurate/Resources/MLKitPoseDetectionAccurateResources\" \"MLKitPoseDetectionAccurateResources\"\ncopy_bundle \"$MLKIT_PODS/MLKitPoseDetectionCommon/Resources/MLKitPoseDetectionCommonResources\" \"MLKitPoseDetectionCommonResources\"\ncopy_bundle \"$MLKIT_PODS/MLKitXenoCommon/Frameworks/MLKitXenoCommon.framework/MLKitXenoResources.bundle\" \"MLKitXenoResources\"\n"; 185 166 }; 186 167 /* End PBXShellScriptBuildPhase section */ 187 168 ··· 331 312 "@executable_path/Frameworks", 332 313 ); 333 314 MARKETING_VERSION = 1.0; 334 - OTHER_LDFLAGS = ( 335 - "$(inherited)", 336 - "-ObjC", 337 - ); 315 + OTHER_LDFLAGS = "$(inherited)"; 338 316 PRODUCT_BUNDLE_IDENTIFIER = com.nate.posedetection.iosApp; 339 317 PRODUCT_NAME = PoseDetection; 340 318 SWIFT_EMIT_LOC_STRINGS = YES; ··· 361 339 "@executable_path/Frameworks", 362 340 ); 363 341 MARKETING_VERSION = 1.0; 364 - OTHER_LDFLAGS = ( 365 - "$(inherited)", 366 - "-ObjC", 367 - ); 342 + OTHER_LDFLAGS = "$(inherited)"; 368 343 PRODUCT_BUNDLE_IDENTIFIER = com.nate.posedetection.iosApp; 369 344 PRODUCT_NAME = PoseDetection; 370 345 SWIFT_EMIT_LOC_STRINGS = YES;