TingPush Camera AI

Integration docs

Adding a new AI model to the camera (no code changes)

Bản tiếng Việt

The RV1126BP camera can run custom AI models: oil spills, smoke / fire, helmets, blocked walkways… Each model is 2 files, <id>.rknn + <id>.json. Once installed, the app gains a new feature alongside plate reading and people / vehicle counting, with an on / off switch, active hours, a detection zone, sensitivity, and how to alert (notification / phone call).

  • No code changes, no firmware rebuild, no camera restart.
  • Only models that are switched on are loaded on the NPU; switching one off in the app frees it immediately.
  • Up to 8 custom models per camera. With several models on, the NPU shares its time between them: see each model's ms per run on the camera's setup page (Trạng thái → Model AI tùy chỉnh, Status → Custom AI models) or in the app.

The TingPush apps are in Vietnamese; screen labels are quoted as they appear, with a translation in parentheses.

 Ex01 / training machine     server (Admin → AI models)           app (camera owner)              camera
 best.pt ─► make_custom_model.py ─► package oil-1.0.tar.gz ──► AI features → AI models → Install ─► download, check SHA-256,
                                                                                             swap files, reload
                                                              turn on "Oil spill", draw zone ─► runs every 3 seconds
                                                                                             ─► "custom" event
                                                                                             ─► notification / call

1. Train the model

The model must be a YOLOv8 / YOLO11 box detector (Ultralytics, task=detect) with 1..16 classes. Train it the same way as the plate model in Ex01 (yolo detect train model=yolo11n.pt data=… imgsz=640).

Data tips (using oil spills as the example):

  • Photos taken from the camera itself, at its real mounting angle, covering day / night / lighting, different floors, large and small spills.
  • Lots of "not oil" images (unlabelled): water puddles, freshly mopped floors, shadows, light reflections, dirt, paper, plastic bags. Without them the camera keeps raising false alarms.
  • Start with 1 class (oil). If the camera needs to tell things apart, add classes (oil, water) and only alert on oil (--alert oil).
  • yolo11n is fast enough; yolo11s is more accurate but about 2.5× slower on the NPU.

2. Packaging for the camera — lpr_camera/tools/make_custom_model.py

Run on Linux / WSL (rknn-toolkit2 is Linux-only, Python 3.11):

python3.11 -m venv rk && . rk/bin/activate
pip install rknn-toolkit2==2.3.2 onnx==1.16.1 "setuptools<82" ultralytics

python make_custom_model.py --pt runs/detect/train/weights/best.pt --id oil --name "Oil spill" \
    --calib datasets/oil/images/val --mode presence --confirm 30 --level alarm --version 1.0 --verify
# -> out/oil.rknn  out/oil.json  out/oil-1.0.tar.gz
Option Meaning
--id model ID: a-z 0-9 _, up to 14 characters, not one of plate / chars / object / face…. The same ID = a new version of that model
--name name shown in the app and in notifications ("Detected Oil spill at Warehouse floor")
--calib a folder of 50–300 real images from the camera for INT8 quantisation (the closer to reality, the more accurate)
--mode presence alerts once the object has stayed in place for --confirm seconds, once; no repeat while it's still there. Someone briefly walking in front doesn't count as it disappearing. Oil spills, obstacles, left items
--mode instant alerts as soon as it's seen; no repeat while it stays in frame. Smoke, fire, missing helmets
--confirm presence: seconds of continuous presence (default 30)
--interval seconds between model runs (default presence 3, instant 1) — larger means less NPU load
--forget how long it must be gone to count as cleared (default presence 60, instant 10)
--cooldown minimum seconds between two alerts from the model (default 60)
--conf minimum score 0..1 (default 0.5; users can adjust it in the app)
--level info regular notification · alarm urgent alert · call phone call by default (the owner can still choose in the app)
--alert only alert on these classes, e.g. --alert "no helmet"
--gray the model was trained on greyscale images (like the plate model)
--verify compares the INT8 model with the original on the NPU simulator; below 90%, add calibration images

rknn-toolkit2 must be exactly 2.3.2 (matching librknnrt in the camera firmware). The script adds box coordinate normalisation to 0..1 before quantising (like object.rknn) — models exported directly to rknn by Ultralytics will not work correctly.

The <id>.json file (editable by hand, e.g. to rename or change alerting without retraining — a package containing only the .json installs too):

{
  "name": "Oil spill", "desc": "Oil spills on the warehouse floor", "version": "1.0",
  "model": "oil.rknn", "classes": ["oil"], "alert_classes": ["oil"],
  "mode": "presence", "conf": 0.5, "interval_s": 3, "confirm_s": 30, "forget_s": 60, "cooldown_s": 60,
  "level": "alarm", "input": "rgb", "normalized": true
}

3. Installing the model on the camera

Remotely (recommended): admin web → Quản trị → Model AI (Admin → AI models) → upload oil-1.0.tar.gz (or select oil.rknn + oil.json together). The camera owner opens the app → camera → Tính năng AI → Model AI (AI features → AI models) → Cài (Install). The camera downloads the package, checks SHA-256, checks each file (RKNN header, valid JSON), swaps them into /userdata/lpr/models/custom/ and reloads — a few seconds, no restart. A broken package replaces nothing.

The same route updates built-in models: a package with plate.rknn / chars.rknn / object.rknn replaces the plate / people-vehicle models (same number of classes and same export method as before — changing the number of plate classes still requires a firmware rebuild).

By hand: copy oil.rknn + oil.json to the camera's /userdata/lpr/models/custom/ and restart; or put them in lpr_camera/models/custom/ before build.bat / make package to ship them inside the firmware package.

4. Turning it on in the app

Tính năng AI → Oil spill (AI features → Oil spill): switch it on, set active hours + days of the week, the detection zone (drawn on the image, e.g. the floor only; empty = whole image), sensitivity, confirmation time and repeat suppression. Gửi thử (Send test) creates a sample event to test notifications / calls. Cảnh báo (Alerts): each model has its own row (push notification / phone call).

5. Events sent

Type custom, on the same path as intrusion / faces (instant over WebSocket; the copy with the scene image and a close-up of the detected area goes through /api/device/events):

{"type":"custom","summary":"Phát hiện Oil spill tại Sàn kho (đã 31 giây)",
 "data":{"model":"oil","model_name":"Oil spill","class":"oil","class_index":0,"score":0.83,
         "box":[0.40,0.70,0.60,0.80],"duration_s":31,"mode":"presence","level":"alarm","zone":"Sàn kho"}}

summary is generated in Vietnamese ("Detected Oil spill at Sàn kho (for 31 seconds)"). View events in the app / web (Events) and through the public API GET /api/v1/events?type=custom, like other AI events.

6. When you still need code changes

  • Models that aren't YOLO detectors: segmentation, image classification, face recognition, OCR…
  • New logic: measuring spill area, vehicle speed, combining 2 models ("person" + "no helmet")…
  • Changing the plate model's number of classes / thresholds (model_config.h).