NullReferenceException thrown in Tensorflow.Binding.dll when trying to run EfficientDetD0 #1077

vinayak-barman · 2023-05-18T03:12:22Z

Description

 public class tfDetEngine : DetectionEngine
    {
        private readonly Session _session;

        public tfDetEngine(string modelPath, string labelPath) : base(modelPath, labelPath)
        {
            _session = Session.LoadFromSavedModel(modelPath);
        }

        public override IEnumerable<Detection> Detect(Tensor image)
        {

            using (_session)
            {
                try
                {
                    var tensors = _session?.eval(image);
                    if (tensors != null)
                    {
                        var boxesP = tensors["detection_boxes"].ToArray<float>();

                        var boxes = new float[boxesP.Length / 4][];
                        for (int i = 0; i < boxesP.Length; i += 4)
                        {
                            boxes[i / 4] = new float[] { boxesP[i], boxesP[i + 1], boxesP[i + 2], boxesP[i + 3] };
                        }

                        var scores = tensors["detection_scores"].ToArray<float>();
                        var classes = tensors["detection_classes"].ToArray<float>();

                        var detectedObjects = boxes.Select((box, i) => new Detection
                        {
                            Box = box,
                            Score = scores[i],
                            Class = classes[i]
                        });

                        return detectedObjects;
                    }

                    return new List<Detection>();
                }

                catch (NullReferenceException e)
                {
                    return new List<Detection>();
                }
            }
        }

I keep getting NullReferenceException thrown in Tensorflow.Binding.dll whenever I try to evaluate the model with a (512, 512, 3) sized image tensor

The text was updated successfully, but these errors were encountered:

AsakusaRinne · 2023-05-18T03:43:57Z

Could you please provide the stack trace of the exception?

vinayak-barman · 2023-05-18T04:22:45Z

The stack trace goes here:
at Tensorflow.Tensor._as_tf_output()
at Tensorflow.BaseSession.eval(Tensor tensor)
at Detection.Engines.tfDetEngine.Detect(Tensor image)

AsakusaRinne · 2023-05-18T04:57:20Z

Please check if the image is an EagerTensor. Using an EagerTensor under graph mode may cause this problem.

vinayak-barman · 2023-05-18T05:03:55Z

The tensor isn't an EagerTensor indeed
{tf.Tensor '' shape=(512, 512, 3) dtype=uint8}

AsakusaRinne · 2023-05-18T05:17:58Z

Could you please provide some steps to reproduce it? I don't know what DetectionEngine is.

vinayak-barman · 2023-05-18T05:20:58Z

Here is the DetectionEngine class:

 public abstract class DetectionEngine
    {
        public DetectionEngine(string modelPath, string labelPath)
        {
            ModelPath = modelPath;
            LabelPath = labelPath;
        }
        public string ModelPath { get; set; }
        public string LabelPath { get; set; }
        public abstract IEnumerable<Detection> Detect(Tensor frame);
    }

Here is where I run inference:



        public void StartVideo()
        {
            var devices = new List<int>();
            while (devices.Count == 0)
            {
                devices = GetDevices();
            }

            Environment.SetEnvironmentVariable("TF_CPP_MIN_LOG_LEVEL", "2");  // Disable TensorFlow logs
            Environment.SetEnvironmentVariable("OMP_NUM_THREADS", "4");     // Set the number of threads

            new Thread(() => {
                engine = new tfDetEngine(@"C:\Users\softm\Downloads\efficientdet_d0_coco17_tpu-32\efficientdet_d0_coco17_tpu-32\saved_model", "");
            }).Start();
            

            foreach (int device in devices)
            {
                StartVideoDaemon(device);
            }
        }

        async Task StartVideoDaemon(int id)
        {
            await Task.Factory.StartNew(() =>
            {
                VideoCapture capture = new(id);
                Mat frame = new();
                Bitmap img;
                while (true)
                {
                    capture.Read(frame);
                    if (frame.Empty())
                    {
                        break;
                    }

                    Mat frameN = new Mat();
                    Cv2.Resize(frame, frameN, new Size(512, 512));

                    var detections = engine?.Detect(ToTensor(frameN));

                    using (MemoryStream memory = frameN.ToMemoryStream())
                    {
                        img = new Bitmap(memory);

                        var port = "port" + (id + 1);
                        Dispatcher.UIThread.InvokeAsync(() =>
                        {
                            var image = this.FindControl<Image>(port);
                            if (image != null)
                            {
                                image.Source = img;
                            }
                        });
                    }
                    Task.Delay(10);
                }
            });

        }


        public static unsafe Tensor ToTensor(Mat src)
        {
            NumSharp.Shape shape = (src.Height, src.Width, src.Type().Channels);
            SafeTensorHandle handle;
            var tensor = new Tensor(handle = c_api.TF_AllocateTensor(TF_DataType.TF_UINT8, shape.Dimensions.Select(d => (long)d).ToArray(), shape.NDim, (ulong)shape.Size));

            new UnmanagedMemoryBlock<byte>(src.DataPointer, shape.Size)
                .CopyTo((byte*)c_api.TF_TensorData(handle));

            return tensor;
        }

        List<int> GetDevices()
        {
            var devices = new List<int>();
            int i = 0;
            while (true)
            {
                var cap = new VideoCapture(i);
                if (cap.IsOpened())
                {
                    devices.Add(i);
                    i++;
                    continue;
                }
                break;
            }
            return devices;
        }

vinayak-barman · 2023-05-18T05:21:41Z

I am using [EfficientDet D0 512x512] (http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz)

AsakusaRinne · 2023-05-18T05:27:48Z

Ok, I'll reproduce it and give you a feedback later. Which version are you using?

vinayak-barman · 2023-05-18T05:28:46Z

I am using the latest one, just got it yesterday

…

On Thu, 18 May, 2023 at 11:27 AM, Rinne ***@***.***> wrote: Ok, I'll reproduce it and give you a feedback later. Which version are you using? — Reply to this email directly, view it on GitHub <#1077 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APIIF6IKXZ5VL3YMJMTLQNLXGWXN5ANCNFSM6AAAAAAYF43PJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

AsakusaRinne · 2023-05-18T06:43:43Z

The UnmanagedMemoryBlock and Dispatcher are not defined in the code. Is there any extra information?

vinayak-barman · 2023-05-18T06:48:55Z

On Thu, 18 May, 2023 at 12:43 PM, Rinne ***@***.***> wrote: The UnmanagedMemoryBlock and Dispatcher are not defined in the code. Is there any extra information? — Reply to this email directly, view it on GitHub <#1077 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APIIF6OQQAHYXM3EQJHDOE3XGXAKTANCNFSM6AAAAAAYF43PJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

I am using Avalonia UI with it, that’s where the dispatcher comes from, and speaking of UnmanagedMemoryBlock, it comes from NumSharp.

AsakusaRinne · 2023-05-18T08:34:51Z

The error is caused by some wrong usages. Note that tf.net has no longer depended on NumSharp, please use Tensorflow.Numpy instead. Here's an example without this problem:

using OpenCvSharp;
using System;
using System.Collections.Generic;
using System.Linq;
using Tensorflow;
using Tensorflow.NumPy;
using static Tensorflow.Binding;

namespace TensorFlowNET.UnitTest.Training
{
    public record class Detection(float[] Box, float Score, float Class);

    public abstract class DetectionEngine
    {
        public DetectionEngine(string modelPath, string labelPath)
        {
            ModelPath = modelPath;
            LabelPath = labelPath;
        }
        public string ModelPath { get; set; }
        public string LabelPath { get; set; }
        public abstract IEnumerable<Detection> Detect(NDArray frame);
    }

    public class tfDetEngine : DetectionEngine
    {
        private readonly Session _session;

        public tfDetEngine(string modelPath, string labelPath) : base(modelPath, labelPath)
        {
            _session = Session.LoadFromSavedModel(modelPath);
        }

        public override IEnumerable<Detection> Detect(NDArray image)
        {
            var graph = _session.graph;
            graph.as_default();
            try
            {
                var assigned = tf.convert_to_tensor(image, name: "input_image");
                bool a = tf.Context.executing_eagerly();
                var tensors = _session?.run(assigned);
                if (tensors != null)
                {
                    var boxesP = tensors["detection_boxes"].ToArray<float>();

                    var boxes = new float[boxesP.Length / 4][];
                    for (int i = 0; i < boxesP.Length; i += 4)
                    {
                        boxes[i / 4] = new float[] { boxesP[i], boxesP[i + 1], boxesP[i + 2], boxesP[i + 3] };
                    }

                    var scores = tensors["detection_scores"].ToArray<float>();
                    var classes = tensors["detection_classes"].ToArray<float>();

                    var detectedObjects = boxes.Select((box, i) => new Detection
                    (
                        Box: box,
                        Score: scores[i],
                        Class: classes[i]
                    ));

                    return detectedObjects;
                }

                return new List<Detection>();
            }

            catch (NullReferenceException e)
            {
                return new List<Detection>();
            }
            finally
            {
                graph.Exit();
            }
        }
    }
    [TestClass]
    public class ObjectDetection
    {
        tfDetEngine engine;

        [TestMethod]
        public void DetectionWithEfficientDet()
        {
            StartVideo();
        }

        public void StartVideo()
        {
            var devices = new List<int>();
            while (devices.Count == 0)
            {
                devices = GetDevices();
            }

            Environment.SetEnvironmentVariable("TF_CPP_MIN_LOG_LEVEL", "2");  // Disable TensorFlow logs
            Environment.SetEnvironmentVariable("OMP_NUM_THREADS", "4");     // Set the number of threads

            engine = new tfDetEngine(@"C:\Users\liu_y\Downloads\efficientdet_d0_coco17_tpu-32\saved_model", "");


            foreach (int device in devices)
            {
                StartVideoDaemon(device);
            }
        }

        void StartVideoDaemon(int id)
        {
            VideoCapture capture = new(id);
            Mat frame = new();
            System.Drawing.Bitmap img;
            while (true)
            {
                capture.Read(frame);
                if (frame.Empty())
                {
                    break;
                }

                Mat frameN = new Mat();
                Cv2.Resize(frame, frameN, new Size(512, 512));

                var detections = engine?.Detect(ToTensor(frameN));
            }

        }


        public unsafe NDArray ToTensor(Mat src)
        {
            Shape shape = (src.Height, src.Width, src.Type().Channels);
            SafeTensorHandle handle;
            var tensor = new Tensor(handle = c_api.TF_AllocateTensor(TF_DataType.TF_UINT8, shape.dims.Select(d => (long)d).ToArray(), shape.ndim, (ulong)shape.size));

            new Span<byte>((void*)src.DataPointer, (int)shape.size)
                .CopyTo(new Span<byte>((byte*)c_api.TF_TensorData(handle), (int)shape.size));

            return tensor.numpy();
        }

        List<int> GetDevices()
        {
            var devices = new List<int>();
            int i = 0;
            while (true)
            {
                var cap = new VideoCapture(i);
                if (cap.IsOpened())
                {
                    devices.Add(i);
                    i++;
                    continue;
                }
                break;
            }
            return devices;
        }
    }
}

However it will still fail when executing var boxesP = tensors["detection_boxes"].ToArray<float>();. I know few about object detection so that I can't figure what is expected. Please check it again.

vinayak-barman · 2023-05-18T12:23:51Z

_session.run still leaves me with a NullReferenceException, here is the stack trace

   at Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list)
   at Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict)
   at Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   at Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)
   at Detection.Engines.tfDetEngine.Detect(NDArray image)

AsakusaRinne · 2023-05-18T12:30:05Z

Are you using exactly the code I listed before?

vinayak-barman · 2023-05-18T12:35:34Z

yes, still doesn't work

vinayak-barman · 2023-05-18T12:44:59Z

Could it be that I have an installation issue? I might have a broken installation of Tensorflow. How can I test that TensorFlow.net will run using a simpler model?

AsakusaRinne · 2023-05-18T15:07:54Z

Could it be that I have an installation issue? I might have a broken installation of Tensorflow. How can I test that TensorFlow.net will run using a simpler model?

Please refer to https://github.com/SciSharp/SciSharp-Stack-Examples/tree/master/src/TensorFlowNET.Examples and choose one or two simple models of it

AsakusaRinne · 2023-05-18T15:08:36Z

yes, still doesn't work

I can't reproduce it at this time. Is there any further information?

vinayak-barman · 2023-05-18T15:46:04Z

I think I have issues with my tensorflow installation or something. Everything should work fine. I am using SciSharp.Tensorflow.Redist as my laptop doesn't have a GPU but my PC does. What steps should I follow to make sure that my initial setups are correct? I suspect my setup is faulty, as you have a perfectly running Tensorflow.Net implementation on your end.

AsakusaRinne · 2023-05-18T16:32:17Z

Sorry, my bad. My local environment is actually the master branch. Could you please try it again with master branch? Please clone it and add a project reference but keep tensorflow.redist installed. We'll publish a nightly release if master branch fixes this problem.

vinayak-barman · 2023-05-18T16:47:53Z

Pardon, which repo do I have to clone now?

AsakusaRinne · 2023-05-18T16:57:46Z

Pardon, which repo do I have to clone now?

Please clone this repo and reference the src/TensorFlowNET.Core/tensorflow.net.csproj

vinayak-barman · 2023-05-18T17:00:56Z

But, there is no tensorflow.net.csproj but rather Tensorflow.Binding.csproj

…

On Thu, 18 May, 2023 at 10:57 PM, Rinne ***@***.***> wrote: Pardon, which repo do I have to clone now? Please clone this repo and reference the src/TensorFlowNET.Core/tensorflow.net.csproj — Reply to this email directly, view it on GitHub <#1077 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APIIF6PBM42CU3TYWCZOXCDXGZIJNANCNFSM6AAAAAAYF43PJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

AsakusaRinne · 2023-05-18T17:02:33Z

Yup, that's it. I got it wrong.

vinayak-barman · 2023-05-19T06:13:37Z

I tried it with the source code, but it gave me the same error, but a little detailed. When debugging, I notcied that _session.status is null. Here is the stack trace:

   at Tensorflow.Status.op_Implicit(Status status) in C:\Users\softm\Downloads\TensorFlow.NET\src\TensorFlowNET.Core\Status\Status.cs:line 102
   at Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list) in C:\Users\softm\Downloads\TensorFlow.NET\src\TensorFlowNET.Core\Sessions\BaseSession.cs:line 216
   at Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict) in C:\Users\softm\Downloads\TensorFlow.NET\src\TensorFlowNET.Core\Sessions\BaseSession.cs:line 205
   at Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict) in C:\Users\softm\Downloads\TensorFlow.NET\src\TensorFlowNET.Core\Sessions\BaseSession.cs:line 133
   at Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict) in C:\Users\softm\Downloads\TensorFlow.NET\src\TensorFlowNET.Core\Sessions\BaseSession.cs:line 57

vinayak-barman · 2023-05-19T06:18:59Z

Just curious, what should I put in the place of feed_dict params?

AsakusaRinne · 2023-05-19T09:38:34Z

Sorry for the confusion, I've fixed the error of null reference of status, could you please pull the newest master branch and try again?

AsakusaRinne · 2023-05-19T09:39:48Z

feed_dict

feed_dict is a parameter to specify some initial values for the parameters

vinayak-barman · 2023-05-19T09:51:23Z

Now I get "Invalid slice notation: 'detection_boxes"

AsakusaRinne · 2023-05-19T09:53:18Z

Now I get "Invalid slice notation: 'detection_boxes"

Yes, that's the same with what acts like in my local environment. Since I don't know much about object detection, could you please further explain what you want by var boxesP = tensors["detection_boxes"].ToArray<float>();?

vinayak-barman · 2023-05-19T10:55:50Z

So, I couldn't convert boxes to Float array directly, because it is nullable. I couldn't just write it as ToArray<float[]>(), it should actually be a two dimensional array, and the extra for loop is there trying to convert the float[] to float[][]

AsakusaRinne · 2023-05-19T11:12:32Z

What are tensors and boxesP expected to be in object detection?

vinayak-barman · 2023-05-19T11:40:20Z

tensors is the detections from the model, and boxes are the bounding boxes from detections.

vinayak-barman · 2023-05-19T11:59:46Z

But graph has no nodes, everything shows 0. Maybe the model isn't loaded correctly?

AsakusaRinne · 2023-05-19T17:18:55Z

@Oceania2018 Could you please look at this issue?

AsakusaRinne mentioned this issue May 19, 2023

fix: status null reference of base session. #1079

Merged

NullReferenceException thrown in Tensorflow.Binding.dll when trying to run EfficientDetD0 #1077

NullReferenceException thrown in Tensorflow.Binding.dll when trying to run EfficientDetD0 #1077

Comments

vinayak-barman commented May 18, 2023

Description

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023 • edited Loading

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023

vinayak-barman commented May 18, 2023

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023 via email

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023 via email

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023 • edited Loading

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023

vinayak-barman commented May 18, 2023

AsakusaRinne commented May 18, 2023

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 18, 2023 via email

AsakusaRinne commented May 18, 2023

vinayak-barman commented May 19, 2023

vinayak-barman commented May 19, 2023

AsakusaRinne commented May 19, 2023

AsakusaRinne commented May 19, 2023

vinayak-barman commented May 19, 2023

AsakusaRinne commented May 19, 2023

vinayak-barman commented May 19, 2023

AsakusaRinne commented May 19, 2023

vinayak-barman commented May 19, 2023

vinayak-barman commented May 19, 2023 • edited Loading

AsakusaRinne commented May 19, 2023

vinayak-barman commented May 18, 2023 •

edited

Loading

vinayak-barman commented May 18, 2023 •

edited

Loading

vinayak-barman commented May 19, 2023 •

edited

Loading