How Do I Get The Raw Android Camera Buffer In C Using Jni?

I've been searching Google and StackOverflow exhaustively and cannot find this. Maybe I'm missing something obvious. Thanks! (This is because the Java implementation of the previ

Solution 1:

I made a little investigation on topic. This presentation (from p.277, Chinese) helped a lot.

Camera preview call chain

As others mentioned, you can get a buffer using a Camera.setPreviewCallback method. Here's how it happens there (a verbose version):

1. User calls Camera.startPreview() which is a native function.
2. android_hardware_Camera_startPreview calls startPreview method of C++ Camera class.
3. Camera calls a startPreview method of ICamera interface
4. ICamera makes an IPC call to remote client.
5. It calls a setCameraMode method of CameraService class.
6. CameraService sets a window to display a preview and calls a startPreview method of CameraHardwareInterface class.
7. The latter tries to call a start_preview method on particular camera_device_t device. I didn't looked up further but it should perform a call to the driver.
8. When image arrives, dataCallback of CameraService is invoked.
9. It passes data to handlePreviewData method of client.
10. Client either copies the buffer or sends it directly to the ICameraClient.
11. ICameraClient sends it over IPC to the Camera.
12. Camera calls a registered listener and passes buffer to JNI.
13. It invokes a callback in Java class. If user provided a buffer with Camera.addCallbackBuffer then it copies to the buffer first.
14. Finally Java class Camera handles the message and invokes a onPreviewFrame method of Camera.PreviewCallback.

As you can see 2 IPC calls were invoked and buffer was copied at least twice on steps 10, 11. First instance of raw buffer which is returned by camera_device_t is hosted in another process and you cannot access it due to security checks in CameraService.

Preview surface

However, when you set a preview surface using either Camera.setPreviewTexture or Camera.setPreviewDisplay it is be passed directly to the camera device and refreshed in realtime without participation of all the chain above. As it's documentation says:

Handle onto a raw buffer that is being managed by the screen compositor.

Java class Surface has a method to retrieve it's contents:

publicstaticnative Bitmap screenshot(int width, int height, int minLayer, int maxLayer);

But this API is hidden. See i.e. this question for a way to use it.

Solution 2:

There is no public API to do what you want; the only official (that is, guaranteed to work) method is the Java-level preview callbacks set up through calling Camera.setPreviewCallback(). In Android > 3.0, you can also use Camera.setPreviewTexture() to route preview data to the GPU, and process it there using GLES (or read it back to the CPU). The GPU path is what the ICS AOSP camera application uses for its video effects.

Presumably, OpenCV and others have looked through the Android framework native code, and have bypassed the Java Camera API, talking to the services below directly.

This is fairly dangerous, because there is absolutely no guarantee that those interfaces won't change between Android versions, since they are not part of the public API. Using them may be fine now, and then when a user upgrades their device, your app will stop working.

Solution 3:

Have you taken a look at OpenCV for Android. Their advanced tutorials show how to use JNI and there is a NativeProcessor object in their camera package.