~paparazzi-uav/paparazzi/v5.0-manual

Let's consider how we can use OpenCL in this scenario. In particular let's try two ways: direct calls to OpenCL API and recently introduced OpenCV T-API

(aka [Transparent API](https://docs.google.com/presentation/d/1qoa29N_B-s297-fp0-b3rBirvpzJQp8dCtllLQ4DVCY/present)) - implicit OpenCL accelerations of some OpenCV algo-s.

Application structure

---------------------

Starting Android API level 11 (Android 3.0) [Camera API](http://developer.android.com/reference/android/hardware/Camera.html)

allows use of OpenGL texture as a target for preview frames.

Android API level 21 brings a new [Camera2 API](http://developer.android.com/reference/android/hardware/camera2/package-summary.html)

that provides much more control over the camera settings and usage modes,

it allows several targets for preview frames and OpenGL texture in particular.

Having a preview frame in an OpenGL texture is a good deal for using OpenCL because there is an

[OpenGL-OpenCL Interoperability API (cl_khr_gl_sharing)](https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/cl_khr_gl_sharing.html),

allowing sharing OpenGL texture data with OpenCL functions without copying (with some restrictions of course).

Let's create a base for our application that just configures Android camera to send preview frames to OpenGL texture and displays these frames

on display without any processing.

A minimal `Activity` class for that purposes looks like following:

@code{.java}

public class Tutorial4Activity extends Activity {

private MyGLSurfaceView mView;

@Override

public void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

requestWindowFeature(Window.FEATURE_NO_TITLE);

getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,

WindowManager.LayoutParams.FLAG_FULLSCREEN);

getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON,

WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);

mView = new MyGLSurfaceView(this);

setContentView(mView);

}

@Override

protected void onPause() {

mView.onPause();

super.onPause();

}

@Override

protected void onResume() {

super.onResume();

mView.onResume();

}

@endcode

And a minimal `View` class respectively:

@code{.java}

public class MyGLSurfaceView extends GLSurfaceView {

MyGLRendererBase mRenderer;

public MyGLSurfaceView(Context context) {

super(context);

if(android.os.Build.VERSION.SDK_INT >= 21)

mRenderer = new Camera2Renderer(this);

else

100

mRenderer = new CameraRenderer(this);

101

102

setEGLContextClientVersion(2);

103

setRenderer(mRenderer);

104

setRenderMode(GLSurfaceView.RENDERMODE_WHEN_DIRTY);

105

}

106

107

@Override

108

public void surfaceCreated(SurfaceHolder holder) {

109

super.surfaceCreated(holder);

110

}

111

112

@Override

113

public void surfaceDestroyed(SurfaceHolder holder) {

114

super.surfaceDestroyed(holder);

115

}

116

117

@Override

118

public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {

119

super.surfaceChanged(holder, format, w, h);

120

}

121

122

@Override

123

public void onResume() {

124

super.onResume();

125

mRenderer.onResume();

126

}

127

128

@Override

129

public void onPause() {

130

mRenderer.onPause();

131

super.onPause();

132

}

133

}

134

@endcode

135

136

__Note__: we use two renderer classes: one for legacy [Camera](http://developer.android.com/reference/android/hardware/Camera.html) API

137

and another for modern [Camera2](http://developer.android.com/reference/android/hardware/camera2/package-summary.html).

138

139

A minimal `Renderer` class can be implemented in Java (OpenGL ES 2.0 [available](http://developer.android.com/reference/android/opengl/GLES20.html) in Java),

140

but since we are going to modify the preview texture with OpenCL let's move OpenGL stuff to JNI.

141

Here is a simple Java wrapper for our JNI stuff:

142

143

@code{.java}

144

public class NativeGLRenderer {

145

static

146

{

147

System.loadLibrary("opencv_java3"); // comment this when using OpenCV Manager

148

System.loadLibrary("JNIrender");

149

}

150

151

public static native int initGL();

152

public static native void closeGL();

153

public static native void drawFrame();

154

public static native void changeSize(int width, int height);

155

}

156

@endcode

157

158

Since `Camera` and `Camera2` APIs differ significantly in camera setup and control, let's create a base class for the two corresponding renderers:

159

160

@code{.java}

161

public abstract class MyGLRendererBase implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {

162

protected final String LOGTAG = "MyGLRendererBase";

163

164

protected SurfaceTexture mSTex;

165

protected MyGLSurfaceView mView;

166

167

protected boolean mGLInit = false;

168

protected boolean mTexUpdate = false;

169

170

MyGLRendererBase(MyGLSurfaceView view) {

171

mView = view;

172

}

173

174

protected abstract void openCamera();

175

protected abstract void closeCamera();

176

protected abstract void setCameraPreviewSize(int width, int height);

177

178

public void onResume() {

179

Log.i(LOGTAG, "onResume");

180

}

181

182

public void onPause() {

183

Log.i(LOGTAG, "onPause");

184

mGLInit = false;

185

mTexUpdate = false;

186

closeCamera();

187

if(mSTex != null) {

188

mSTex.release();

189

mSTex = null;

190

NativeGLRenderer.closeGL();

191

}

192

}

193

194

@Override

195

public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture) {

196

//Log.i(LOGTAG, "onFrameAvailable");

197

mTexUpdate = true;

198

mView.requestRender();

199

}

200

201

@Override

202

public void onDrawFrame(GL10 gl) {

203

//Log.i(LOGTAG, "onDrawFrame");

204

if (!mGLInit)

205

return;

206

207

synchronized (this) {

208

if (mTexUpdate) {

209

mSTex.updateTexImage();

210

mTexUpdate = false;

211

}

212

}

213

NativeGLRenderer.drawFrame();

214

}

215

216

@Override

217

public void onSurfaceChanged(GL10 gl, int surfaceWidth, int surfaceHeight) {

218

Log.i(LOGTAG, "onSurfaceChanged("+surfaceWidth+"x"+surfaceHeight+")");

219

NativeGLRenderer.changeSize(surfaceWidth, surfaceHeight);

220

setCameraPreviewSize(surfaceWidth, surfaceHeight);

221

}

222

223

@Override

224

public void onSurfaceCreated(GL10 gl, EGLConfig config) {

225

Log.i(LOGTAG, "onSurfaceCreated");

226

String strGLVersion = GLES20.glGetString(GLES20.GL_VERSION);

227

if (strGLVersion != null)

228

Log.i(LOGTAG, "OpenGL ES version: " + strGLVersion);

229

230

int hTex = NativeGLRenderer.initGL();

231

mSTex = new SurfaceTexture(hTex);

232

mSTex.setOnFrameAvailableListener(this);

233

openCamera();

234

mGLInit = true;

235

}

236

}

237

@endcode

238

239

As you can see, inheritors for `Camera` and `Camera2` APIs should implement the following abstract methods:

240

@code{.java}

241

protected abstract void openCamera();

242

protected abstract void closeCamera();

243

protected abstract void setCameraPreviewSize(int width, int height);

244

@endcode

245

246

Let's leave the details of their implementation beyond of this tutorial, please refer the

247

[source code](https://github.com/Itseez/opencv/tree/master/samples/android/tutorial-4-opencl/) to see them.

248

249

Preview Frames modification

250

---------------------------

251

252

The details OpenGL ES 2.0 initialization are also quite straightforward and noisy to be quoted here,

253

but the important point here is that the OpeGL texture to be the target for camera preview should be of type `GL_TEXTURE_EXTERNAL_OES`

254

(not `GL_TEXTURE_2D`), internally it keeps picture data in _YUV_ format.

255

That makes unable sharing it via CL-GL interop (`cl_khr_gl_sharing`) and accessing its pixel data via C/C++ code.

256

To overcome this restriction we have to perform an OpenGL rendering from this texture to another regular `GL_TEXTURE_2D` one

257

using _FrameBuffer Object_ (aka FBO).

258

259

### C/C++ code

260

261

After that we can read (_copy_) pixel data from C/C++ via `glReadPixels()` and write them back to texture after modification via `glTexSubImage2D()`.

262

263

### Direct OpenCL calls

264

265

Also that `GL_TEXTURE_2D` texture can be shared with OpenCL without copying, but we have to create OpenCL context with special way for that:

266

267

@code{.cpp}

268

void initCL()

269

{

270

EGLDisplay mEglDisplay = eglGetCurrentDisplay();

271

if (mEglDisplay == EGL_NO_DISPLAY)

272

LOGE("initCL: eglGetCurrentDisplay() returned 'EGL_NO_DISPLAY', error = %x", eglGetError());

273

274

EGLContext mEglContext = eglGetCurrentContext();

275

if (mEglContext == EGL_NO_CONTEXT)

276

LOGE("initCL: eglGetCurrentContext() returned 'EGL_NO_CONTEXT', error = %x", eglGetError());

277

278

cl_context_properties props[] =

279

{ CL_GL_CONTEXT_KHR, (cl_context_properties) mEglContext,

280

CL_EGL_DISPLAY_KHR, (cl_context_properties) mEglDisplay,

281

CL_CONTEXT_PLATFORM, 0,

282

0 };

283

284

try

285

{

286

cl::Platform p = cl::Platform::getDefault();

287

std::string ext = p.getInfo<CL_PLATFORM_EXTENSIONS>();

288

if(ext.find("cl_khr_gl_sharing") == std::string::npos)

289

LOGE("Warning: CL-GL sharing isn't supported by PLATFORM");

290

props[5] = (cl_context_properties) p();

291

292

theContext = cl::Context(CL_DEVICE_TYPE_GPU, props);

293

std::vector<cl::Device> devs = theContext.getInfo<CL_CONTEXT_DEVICES>();

294

LOGD("Context returned %d devices, taking the 1st one", devs.size());

295

ext = devs[0].getInfo<CL_DEVICE_EXTENSIONS>();

296

if(ext.find("cl_khr_gl_sharing") == std::string::npos)

297

LOGE("Warning: CL-GL sharing isn't supported by DEVICE");

298

299

theQueue = cl::CommandQueue(theContext, devs[0]);

300

301

// ...

302

}

303

catch(cl::Error& e)

304

{

305

LOGE("cl::Error: %s (%d)", e.what(), e.err());

306

}

307

catch(std::exception& e)

308

{

309

LOGE("std::exception: %s", e.what());

310

}

311

catch(...)

312

{

313

LOGE( "OpenCL info: unknown error while initializing OpenCL stuff" );

314

}

315

LOGD("initCL completed");

316

}

317

@endcode

318

319

@note To build this JNI code you need __OpenCL 1.2__ headers from [Khronos web site](https://www.khronos.org/registry/cl/api/1.2/) and

320

the __libOpenCL.so__ downloaded from the device you'll run the application.

321

322

Then the texture can be wrapped by a `cl::ImageGL` object and processed via OpenCL calls:

323

@code{.cpp}

324

cl::ImageGL imgIn (theContext, CL_MEM_READ_ONLY, GL_TEXTURE_2D, 0, texIn);

325

cl::ImageGL imgOut(theContext, CL_MEM_WRITE_ONLY, GL_TEXTURE_2D, 0, texOut);

326

327

std::vector < cl::Memory > images;

328

images.push_back(imgIn);

329

images.push_back(imgOut);

330

theQueue.enqueueAcquireGLObjects(&images);

331

theQueue.finish();

332

333

cl::Kernel Laplacian = ...

334

Laplacian.setArg(0, imgIn);

335

Laplacian.setArg(1, imgOut);

336

theQueue.finish();

337

338

theQueue.enqueueNDRangeKernel(Laplacian, cl::NullRange, cl::NDRange(w, h), cl::NullRange);

339

theQueue.finish();

340

341

theQueue.enqueueReleaseGLObjects(&images);

342

theQueue.finish();

343

@endcode

344

345

### OpenCV T-API

346

347

But instead of writing OpenCL code by yourselves you may want to use __OpenCV T-API__ that calls OpenCL implicitly.

348

All that you need is to pass the created OpenCL context to OpenCV (via `cv::ocl::attachContext()`) and somehow wrap OpenGL texture with `cv::UMat`.

349

Unfortunately `UMat` keeps OpenCL _buffer_ internally, that can't be wrapped over either OpenGL _texture_ or OpenCL _image_ - so we have to copy image data here:

350

@code{.cpp}

351

cl::ImageGL imgIn (theContext, CL_MEM_READ_ONLY, GL_TEXTURE_2D, 0, tex);

352

std::vector < cl::Memory > images(1, imgIn);

353

theQueue.enqueueAcquireGLObjects(&images);

354

theQueue.finish();

355

356

cv::UMat uIn, uOut, uTmp;

357

cv::ocl::convertFromImage(imgIn(), uIn);

358

theQueue.enqueueReleaseGLObjects(&images);

359

360

cv::Laplacian(uIn, uTmp, CV_8U);

361

cv:multiply(uTmp, 10, uOut);

362

cv::ocl::finish();

363

364

cl::ImageGL imgOut(theContext, CL_MEM_WRITE_ONLY, GL_TEXTURE_2D, 0, tex);

365

images.clear();

366

images.push_back(imgOut);

367

theQueue.enqueueAcquireGLObjects(&images);

368

cl_mem clBuffer = (cl_mem)uOut.handle(cv::ACCESS_READ);

369

cl_command_queue q = (cl_command_queue)cv::ocl::Queue::getDefault().ptr();

370

size_t offset = 0;

371

size_t origin[3] = { 0, 0, 0 };

372

size_t region[3] = { w, h, 1 };

373

CV_Assert(clEnqueueCopyBufferToImage (q, clBuffer, imgOut(), offset, origin, region, 0, NULL, NULL) == CL_SUCCESS);

374

theQueue.enqueueReleaseGLObjects(&images);

375

cv::ocl::finish();

376

@endcode

377

378

- @note We have to make one more image data copy when placing back the modified image to the original OpenGL texture via OpenCL image wrapper.

379

- @note By default the OpenCL support (T-API) is disabled in OpenCV builds for Android OS (so it's absent in official packages as of version 3.0),

380

but it's possible to rebuild locally OpenCV for Android with OpenCL/T-API enabled: use `-DWITH_OPENCL=YES` option for CMake.

381

@code{.cmd}

382

cd opencv-build-android

383

path/to/cmake.exe -GNinja -DCMAKE_MAKE_PROGRAM="path/to/ninja.exe" -DCMAKE_TOOLCHAIN_FILE=path/to/opencv/platforms/android/android.toolchain.cmake -DANDROID_ABI="armeabi-v7a with NEON" -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON path/to/opencv

384

path/to/ninja.exe install/strip

385

@endcode

386

To use your own modified `libopencv_java3.so` you have to keep inside your APK, not to use OpenCV Manager and load it manually via `System.loadLibrary("opencv_java3")`.

387

388

Performance notes

389

-----------------

390

391

To compare the performance we measured FPS of the same preview frames modification (_Laplacian_) done by C/C++ code (call to `cv::Laplacian` with `cv::Mat`),

392

by direct OpenCL calls (using OpenCL _images_ for input and output), and by OpenCV _T-API_ (call to `cv::Laplacian` with `cv::UMat`) on _Sony Xperia Z3_ with 720p camera resolution:

393

* __C/C++ version__ shows __3-4 fps__

394

* __direct OpenCL calls__ shows __25-27 fps__

395

* __OpenCV T-API__ shows __11-13 fps__ (due to extra copying from `cl_image` to `cl_buffer` and back)

Older »