ML Kit Analyzer

Google’s ML Kit provides on-device machine learning Vision APIs for detecting faces, scanning barcodes, labeling images, and more. ML Kit Analyzer makes it easier to integrate ML Kit with your CameraX app.

ML Kit Analyzer is an implementation of the ImageAnalysis.Analyzer interface. It overrides the default target resolution (if needed) to optimize for ML Kit usage, handles the coordinate transformations, and passes the frames to ML Kit, which returns the aggregated analysis results.

Implement ML Kit Analyzer

To implement ML Kit Analyzer, we recommend using the CameraController class, which works with PreviewView to display UI elements. When implemented using CameraController, ML Kit Analyzer handles the coordinate transformations between the original ImageAnalysis stream and PreviewView for you. It receives the target coordinate system from CameraX, calculates the coordinate transformation, and forwards it to ML Kit’s Detector class for analysis.

To use ML Kit Analyzer with CameraController, call setImageAnalysisAnalyzer() and pass it a new ML Kit Analyzer object with the following in its constructor:

• A list of ML Kit Detectors, which CameraX invokes sequentially in order.

• The target coordinate system that determines the coordinates of the ML Kit output:

  • COORDINATE_SYSTEM_VIEW_REFERENCED: the transformed PreviewView coordinates.
  • COORDINATE_SYSTEM_ORIGINAL: the original ImageAnalysis stream coordinates.

• An Executor that invokes the Consumer callback and delivers the MlKitAnalyzer.Result, or the aggregated ML Kit result of a camera frame, to the app.

• A Consumer, which CameraX invokes when there is new ML Kit output.

The following code implements ML Kit Analyzer using CameraController to set up a BarcodeScanner to detect QR codes:

// create BarcodeScanner object
val options = BarcodeScannerOptions.Builder()
  .setBarcodeFormats(Barcode.FORMAT_QR_CODE)
  .build()
val barcodeScanner = BarcodeScanning.getClient(options)

cameraController.setImageAnalysisAnalyzer(
            ContextCompat.getMainExecutor(this),
            MlKitAnalyzer(
                listOf(barcodeScanner),
                COORDINATE_SYSTEM_VIEW_REFERENCED,
                ContextCompat.getMainExecutor(this)
            ) { result: MlKitAnalyzer.Result? ->
    // The value of result.getResult(barcodeScanner) can be used directly for drawing UI overlay.
    }
)

// create BarcodeScanner object
BarcodeScannerOptions options = new BarcodeScannerOptions.Builder()
   .setBarcodeFormats(Barcode.FORMAT_QR_CODE)
   .build();
BarcodeScanner barcodeScanner = BarcodeScanning.getClient(options);

cameraController.setImageAnalysisAnalyzer(executor,
    new MlKitAnalyzer(List.of(barcodeScanner), COORDINATE_SYSTEM_VIEW_REFERENCED,
    executor, result -> {
   // The value of result.getResult(barcodeScanner) can be used directly for drawing UI overlay.
 });

In the code sample above, ML Kit Analyzer passes the following to BarcodeScanner’s Detector class:

• The transformation Matrix based on COORDINATE_SYSTEM_VIEW_REFERENCED that represents the target coordinate system.
• The camera frames.

If BarcodeScanner runs into any issues, then its Detector throws an error, and ML Kit Analyzer propagates it to your app. If successful, then ML Kit Analyzer returns MLKitAnalyzer.Result#getValue(), which in this case is the Barcode object.

You can also implement ML Kit Analyzer using the ImageAnalysis class that is part of camera-core. However, because ImageAnalysis is not integrated with PreviewView, you must manually handle the coordinate transformations. For more information, see the ML Kit Analyzer reference documentation.

Additional resources

For a working camera app with ML Kit Analyzer functionality, see the CameraX-MLKit sample.