An realtime pose detection library for Android and Compose Multiplatform. Android version uses CameraX and GoogleML Kit, while iOS version uses AVFoundation with VisionKit and CoreML.

Quick Start#

Import the Compose library

implementation("com.performancecoachlab.posedetection:posedetection-compose:1.2.2")

Add camera use to your android manifest

<uses-feature
        android:name="android.hardware.camera"
        android:required="false" />
<uses-permission android:name="android.permission.CAMERA" />

Add camera use to you iOS info.plist

<key>NSCameraUsageDescription</key>
<string>We need access to your camera to analyse your performance.</string>

Usage#

Request camera permissions

var permissionGranted by remember { mutableStateOf(false) }
PermissionProvider().apply {
    if (!hasCameraPermission()) RequestCameraPermission(onGranted = {
        permissionGranted = true
    }, onDenied = { permissionGranted = false }) else permissionGranted = true
}

Create a Skeleton Repisitory

val skeletonRepository = remember { SkeletonRepository() }

Initialise the camera feed

if (permissionGranted) CameraView(skeletonRepository = skeletonRepository)  

Create a Pose to detect

val upRightPose = Pose(
    leftShoulder = Pose.PoseRange(0.0, 40.0),
    rightShoulder = Pose.PoseRange(0.0, 40.0),
    leftHip = Pose.PoseRange(160.0, 180.0),
    rightHip = Pose.PoseRange(160.0, 180.0),
    leftKnee = Pose.PoseRange(160.0, 180.0),
    rightKnee = Pose.PoseRange(160.0, 180.0)
)

Listen for skeleton updates and detect specific poses

val skeleton by skeletonRepository.skeletonFlow.collectAsState()
val poseDetected = skeleton?.let { 
    upRightPose.matches(it)
}

Analyse pre recorded video files. Initialise the video extraction for android with your application context.

VideoExtractionContext.setUp(applicationContext)

extract frames from the video and request analysis

rememberCoroutineScope().launch {
    try {
        extractFrame(url, frame, VideoExtractionContext)?.let { frame ->
            frameAnalyser.analyseFrame(frame)?.let { skeleton ->
                bitmap = frame.drawSkeleton(skeleton)
            }
        }
    } catch (e: Exception) {
        e.printStackTrace()
    }
}

Check out the sample app for full example of how to use the library.

License#

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.