Could not begin read transaction (another read transaction is still active on this thread)

[darshansatra1]

So I'm running a foreground service using the plugin flutter_foreground_task. Before I was using Hive DB, but since the foreground service creates a new isolate I was not able to access the same box in the main isolate, and other isolates since Hive does not support multi-threading so I tried Object Box. It was working awesome and the main isolate was able to see the changes made in other isolates. But the issue began when I found out that Read operations throw an error when another block accesses(Write/Read) the box at the same time. I read the documentation and it says there can be more than one reader, however it was not the case while I was running the foreground service.

Basic info:

• ObjectBox version: ^1.1.1
• Flutter/Dart SDK: 2.2.4-0.0.pre.1
• Null-safety enabled: yes
• Reproducibility: always
• OS: Windows
• Device/Emulator: OnePlus Nord

• flutter doctor -v

[√] Flutter (Channel stable, 2.2.4-0.0.pre.1, on Microsoft Windows [Version 10.0.19041.1110], locale en-IN)
    • Flutter version 2.2.4-0.0.pre.1 at C:\src\flutter
    • Framework revision a34a20c34d (3 weeks ago), 2021-07-04 14:41:40 +0530
    • Engine revision 241c87ad80
    • Dart version 2.13.4

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at C:\Users\Darshan\AppData\Local\Android\sdk
    • Platform android-30, build-tools 30.0.2
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java       
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Android Studio (version 4.1.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] IntelliJ IDEA Ultimate Edition (version 2021.1)
    • IntelliJ at C:\Program Files\JetBrains\IntelliJ IDEA 2020.2.3
    • Flutter plugin version 57.0.5
    • Dart plugin version 211.7665

[√] VS Code (version 1.58.2)
    • VS Code at C:\Users\Darshan\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.24.0

[√] Connected device (3 available)
    • AC2001 (mobile) • 4f0ec2e7 • android-arm64  • Android 11 (API 30)
    • Chrome (web)    • chrome   • web-javascript • Google Chrome 91.0.4472.114
    • Edge (web)      • edge     • web-javascript • Microsoft Edge 92.0.902.55

• No issues found!

Expected behavior

It was not suppose to throw error since any number of readers are allowed.

Code

@override
List<HealthDataModel> fetchAllHealthData() {
  try {
    if (healthDataBox == null) return [];
    return healthDataBox?.getAll() ?? [];
  } catch (e) {
    print(e);
    return [];
  }
}

• pubspec.yaml.

name: health
description: 

publish_to: "none"

version: 1.0.5+6

environment:
  sdk: ">=2.12.0 <3.0.0"

dependencies:
  flutter:
    sdk: flutter
  http: any
  background_fetch: ^1.0.0
  flutter_foreground_task: ^2.0.4
  objectbox: ^1.1.1
  objectbox_flutter_libs: ^1.1.1

dev_dependencies:
  flutter_test:
    sdk: flutter
  json_serializable: ^4.1.1
  build_runner: ^2.0.2
  flutter_native_splash: ^1.1.8+4
  objectbox_generator: ^1.1.1

flutter_native_splash:
  color: "#F8F8FC"
  image: assets/images/splash screen icon.png
  android: true
  ios: true

flutter:
  uses-material-design: true
  assets:
    - assets/images/
    - assets/icons/ 
  fonts:
    - family: Arial
      fonts:
        - asset: fonts/Arial.ttf 

• Affected entity classes. Although this error happens randomly for all entity

import 'package:objectbox/objectbox.dart';

@Entity()
class HealthDataModel {
  @Id()
  int id;
  @Property(type: PropertyType.date)
  DateTime? date;
  int? baseStep;
  int? steps; 
  HealthDataModel({
    this.id = 0,
    this.date,
    this.baseStep,
    this.steps, 
  });
}

Logs, stack traces

I/flutter (31054): ObjectBoxException: failed to create transaction: 10199 Could not begin read transaction (another read transaction is still active on this thread)
E/Box     (31054): Storage error (code -30783)

Additional context

Add any other context about the problem here.

• Is there anything special about your app?

It tracks your steps even when the app is killed.

• May transactions or multi-threading play a role?

Yes, isolates are being used that's why multi-threading is required.

• Did you find any workarounds to prevent the issue?

• No

[vaind]

That's pretty weird... Box.getAll() is synchronous and that isn't interrupted so I don't see how anything could try to run on the same thread at the same time. Even though multiple isolates may be scheduled on the same thread by Dart VM, the isolate must "yield" first (using await) before another one can execute.

Can you provide some code actually running the parallel operations? It would be perfect if you could reduce the issue to the minimal flutter app where the issue can be reproduced since I'm not familiar with flutter_foreground_task.

I've had a brief look at flutter_foreground_task and maybe that's even the issue with how the callback is executed directly, not jumping to the UI thread when executing dart's callback? But that's really a stretch since I don't know enough about the plugin and its inner workings... nor about calling dart callbacks from native plugin code (Kotlin) for that matter.

[darshansatra1]

Okay so what is happening is as the application starts, I start the foreground service. Now as I'm using a pedometer, it emits a stream of steps which we have to listen to. I'm listening to the stream in the main isolate and also in the foreground service. So when a new value occurs in the stream, my foreground service first fetches the previous steps; does some calculation, and then store that calculated steps in the DB and as the main isolate is also listening to the stream, when an event occurs there, I try to fetch the new steps from the DB so that I can show it in the UI. Now as these two the two get events are happening at the same time in different threads, it is throwing me this exception. I would really really appreciate your help here.

[vaind]

Now as these two the two get events are happening at the same time in different threads, it is throwing me this exception.

That's the thing - if they were happening in different threads, then there would be no issue - the error is that it tries to start another TX on the same thread... "(another read transaction is still active on this thread)

OK, I have a clearer picture of what you're doing and with a little more info I may be able to try and reproduce it... Could you please add:

• What ObjectBox API calls are you using 1. in the main isolate; 2. in the foreground service callback. Say box.get, box.getAll, query.find, etc. etc - please try to be specific because otherwise it may be impossible to reproduce.
• How exactly are you launching attaching the foreground service callback? Docs look like there's just one function FlutterForegroundTask.start() - is that the one you're using?

[darshansatra1]

Foreground Service:

Future<void> startForegroundService({
  required int steps,
  required int yesterdaySteps,
}) async {
  if (await FlutterForegroundTask.isRunningTask) return;
  await FlutterForegroundTask.init(
    notificationOptions: NotificationOptions(
      channelId: 'steps',
      channelName: 'Steps',
      channelImportance: NotificationChannelImportance.NONE,
      priority: NotificationPriority.LOW,
      visibility: NotificationVisibility.VISIBILITY_SECRET,
      iconData: NotificationIconData(
        name: "stat_name",
        resType: ResourceType.mipmap,
        resPrefix: ResourcePrefix.ic,
      ),
    ),
    foregroundTaskOptions: ForegroundTaskOptions(
      autoRunOnBoot: true,
    ),
  );
  await startPeriodicTask(steps: steps, yesterdaySteps: yesterdaySteps);
}

Future<void> startPeriodicTask({
  required int steps,
  required int yesterdaySteps,
}) async {
  await FlutterForegroundTask.start(
    notificationTitle: "Today: $steps steps",
    notificationText: 'Yesterday: $yesterdaySteps steps',
    callback: periodicTaskFun,
  );
}

void periodicTaskFun() async {
  Stream<StepCount> _stepCountStream = Pedometer.stepCountStream;
  StreamSubscription<StepCount>? streamSubscription;
  LocalDataSource? localDataSource;
  FlutterForegroundTask.initDispatcher((timeStamp) async {
    if (streamSubscription != null) {
      return;
    }
    localDataSource = LocalDataSourceImpl();
    await localDataSource!.initialize();
    streamSubscription = _stepCountStream.listen(
      (steps) async { 
        try {
          DateTime date = DateTime(
              steps.timeStamp.year, steps.timeStamp.month, steps.timeStamp.day);
          HealthDataModel healthDataModel;
          List<HealthDataModel> allHealthData =
              localDataSource!.fetchAllHealthData();
          healthDataModel = allHealthData.firstWhere(
              (element) => element.date!.isSameDate(date), orElse: () {
            print("NOT FOUND IN FOREGROUND SERVICE"); 
            HealthDataModel newHealthData;
            newHealthData = HealthDataModel(
              date: date,
              steps: 0, 
            );
            return newHealthData;
          });
          int newSteps = 0;
          if (healthDataModel.baseStep == null) {
            healthDataModel.baseStep = steps.steps;
          }
          if (steps.steps < healthDataModel.baseStep!) {
            int previousSteps = healthDataModel.steps!;
            newSteps = steps.steps + previousSteps;
            healthDataModel.steps = newSteps;
            \\ This is the function getting called to store steps
            await updateHealthData(
                localDataSource: localDataSource!,
                healthDataModel: healthDataModel);
          } else {
            newSteps = steps.steps - healthDataModel.baseStep!;
            if (newSteps != healthDataModel.steps || newSteps == 0) { 
              healthDataModel.steps = newSteps;
              \\ This is the function getting called to store steps
              await updateHealthData(
                localDataSource: localDataSource!,
                healthDataModel: healthDataModel,
              );
            }
          }
          HealthDataModel yesterdayHealthDataModel = allHealthData.firstWhere(
              (element) =>
                  element.date!.compareTo(
                    DateTime.now().subtract(Duration(days: 1)),
                  ) ==
                  0, orElse: () {
            List<String> hoursSteps = [];
            for (int i = 0; i < 24; i++) {
              hoursSteps.add(0.toString());
            }
            return HealthDataModel(
              date: date.subtract(Duration(days: 1)),
              steps: 0,
              hourSteps: hoursSteps,
              activeTime: 0,
              calories: 0,
              distance: 0,
            );
          });
          await FlutterForegroundTask.update(
            notificationTitle: "Today: ${healthDataModel.steps} steps",
            notificationText:
                'Yesterday: ${yesterdayHealthDataModel.steps} steps',
          );
        } catch (e) {
          print(e);
        }
        busy = false;
      },
      cancelOnError: true,
    );
  }, onDestroy: (timeStamp) async {
    localDataSource?.dispose();
    await streamSubscription?.cancel();
  });
}


Future<void> updateHealthData({
  required LocalDataSource localDataSource,
  required HealthDataModel healthDataModel,
}) async {
  await localDataSource.updateHealthDataModel(healthDataModel: healthDataModel);
}

LocalDataSource :

@override
  Future<HealthDataModel> updateHealthDataModel(
      {required HealthDataModel healthDataModel}) async {
    try {
      if (healthDataBox == null) {
        throw CacheException();
      }
      HealthDataModel model = healthDataBox!.getAll().firstWhere(
          (element) => element.date!.isSameDate(healthDataModel.date!),
          orElse: () {
        return healthDataModel;
      });
      healthDataModel.id = model.id;
      int id = await healthDataBox!.putAsync(healthDataModel);
      healthDataModel.id = id;
      return healthDataModel;
    } catch (e) {
      return healthDataModel;
    }
  }

Main Isolate:
So from Main Isolate I'm calling startForegroundService(steps:0,yesterdaySteps:0) to start foreground service.

 void onStepCount() async {
    // return;
    streamSubscription = _stepCountStream.listen((steps) {
      updateStepsEvent(steps: steps);
    }, cancelOnError: true);
  }

void updateStepsEvent({StepCount steps}){
    \\ This function is calling the function fetchHealthDataFromDate
}

@override
  HealthDataModel? fetchHealthDataFromDate({required DateTime date}) {
    try {
      if (healthDataBox == null) return null;
      int index = healthDataBox!
          .getAll()
          .indexWhere((element) => element.date!.isSameDate(date));
      if (index == -1) return null;
      return healthDataBox!.getAll()[index];
    } catch (e) {
      print(e);
    }
  }

Don't worry about the HealthDataModel, I don't think this issue is because of the model.

[darshansatra1]

If it helps, I saw the flutter_foreground_task and they're using PluginUilities like this:

options['callbackHandle'] =
          PluginUtilities.getCallbackHandle(callback)?.toRawHandle();

[vaind]

I've tried creating a reproduction in this branch: https://github.com/objectbox/objectbox-dart/compare/288-issue-repro but couldn't get the foreground callback to execute - it just logs FlutterForegroundTask started. but then it doesn't actually call the callback. Can you try to have a look to see what the issue might be, as you're familiar with the plugin?

[darshansatra1]

Are you trying this in IOS or Android? It won't work in IOS.

I think you're missing some configurations.

Android 
Since this plugin is based on a foreground service, we need to add the following permission to the AndroidManifest.xml file. Open the AndroidManifest.xml file and specify it between the <manifest> and <application> tags.

<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
And we need to add this permission to automatically resume foreground task at boot time.

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />
And specify the service inside the <application> tag as follows.

<service android:name="com.pravera.flutter_foreground_task.service.ForegroundService" />

[vaind]

Thanks, it was the missing permission setting. Some preliminary info: the callback from the foreground task executes in a separate isolate, thus needs to attach to an existing open store, not open a new one. How do you handle this situation? See this for details: https://stackoverflow.com/a/68519353/2386130

[darshansatra1]

Okay so the problem is, I'm opening the box twice instead of opening the second box using the reference of the first box?

[vaind]

You can get the port to the background task (and then send over the store reference) like in this commit: e61ea42

Let me know if that solved your issue.

[darshansatra1]

Thank you so much, I was gonna give up and move to sqflite, fortunately, this port communication worked and I'm really thankful. Great work and great database.