Events are blocked or slowed down after a while

jmarrucho · April 1, 2019, 4:31pm

Hi,

I use events in a device written in c ++. But I have blockages after a few minutes … My understanding of events is surely not enough …
So I did this very simple test (similar) in python with 2 events on 2 instances of TangoTest. One or both of them also lock up after a few minutes …

Many useful information

2 device proxy:
‘sys/tg_test/1’
'sys/tg_test/jcm
2 callback (class) called with push_event()

2 events suscribed on change value attribute:
double_scalar_w
short_scalar_w

true loop 100ms

Polling 10ms and change event 0.1
Writing new value every 500ms

reader side:


import PyTango,time

mybool=1

devtangotestdoublew='sys/tg_test/1'
dptangotestdoublew=PyTango.DeviceProxy(devtangotestdoublew)

devtangotestshort='sys/tg_test/jcm'
dptangotestshort=PyTango.DeviceProxy(devtangotestshort)

class CBTANGOTESTDOUBLEW():
   def push_event(self,event):
      print ("           tangotest1: ",event.attr_value.value)
   time.sleep(0.1)

class CBTANGOTESTSHORT():
   def push_event(self,event):
      print ("tangotestjcm: ",event.attr_value.value)
   time.sleep(0.1)

cbtangotestdoublew=CBTANGOTESTDOUBLEW()
evttangotestodublew=dptangotestdoublew.subscribe_event('double_scalar_w',PyTango
.EventType.CHANGE_EVENT,cbtangotestdoublew)

cbtangotestshort=CBTANGOTESTSHORT()
evttangotestshort=dptangotestshort.subscribe_event('short_scalar_w',PyTango.Even
tType.CHANGE_EVENT,cbtangotestshort)

while mybool==1:
    time.sleep(0.1)

writer side:


>>> dev='sys/tg_test/1'
>>> dp=PyTango.DeviceProxy(dev)
>>> devjcm='sys/tg_test/jcm'
>>> dpjcm=PyTango.DeviceProxy(devjcm)
>>>
>>> for i in range (0,1000000):
...    for j in range (1,10):
...       dp.double_scalar_w=j
...       dpjcm.short_scalar_w=j
...       time.sleep(0.5)
...

I would like some help please. Thank you

rbourtembourg · April 2, 2019, 7:55am

Hi,

What version of ZMQ are you using?

Here is an extract of cppTango 9.3.3 release notes:

If you are using ZMQ 4.2.0 or ZMQ 4.2.1, we encourage you to use a different ZMQ version or to upgrade to Tango 9.3.

rbourtembourg · April 2, 2019, 11:38am

You could also be affected by the following issue if you are using Tango Access Control:

github.com/tango-controls/cppTango

events and TangoAccessControl

opened 12:54PM - 20 Dec 18 UTC

closed 03:05PM - 24 Feb 22 UTC

egorbe

Hello, We use TangoAccessControl and I had been experiencing problems with even…ts for a long time. C++/Python clients could sucessfully subscribe to events (i used periodic events) but they stopped arriving after 600 seconds. It turned out that clients should resubscribe every 200 seconds by executing EventConfirmSubscription command on the corresponding administrative device. However, the command was not in the list of allowed commands for class Dserver so it was silently rejected by TangoAccessControl. After I added the command to the list of allowed commands, everything starts to work just fine. ``` MySQL [tango]> select * from property_class where value='EventConfirmSubscription'; +---------+------------------+-------+--------------------------+---------------------+---------------------+---------+ | class | name | count | value | updated | accessed | comment | +---------+------------------+-------+--------------------------+---------------------+---------------------+---------+ | DServer | AllowedAccessCmd | 2 | EventConfirmSubscription | 2018-12-19 10:38:45 | 2018-12-19 10:38:45 | NULL | +---------+------------------+-------+--------------------------+---------------------+---------------------+---------+ ``` I think it would be nice if try/catch block in client/eventkeepalive.cpp would report about denied access instead of silent ommision. Also, I guess the dserver EventConfirmSubscription command should be allowed by default in TANGO database by create/update database scripts. thanks, Evgeny

jmarrucho · April 2, 2019, 12:54pm

Hi,

[quote=“Reynald”]Hi,

What version of ZMQ are you using?

Here is an extract of cppTango 9.3.3 release notes:

If you are using ZMQ 4.2.0 or ZMQ 4.2.1, we encourage you to use a different ZMQ version or to upgrade to Tango 9.3.[/quote]

I use Tango 9.2
but no module ZMQ for python:

import zmq
Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named zmq

Thanks.

rbourtembourg · April 2, 2019, 1:33pm

What is your operating system?

jmarrucho · April 2, 2019, 2:16pm

Debian 9.6

rbourtembourg · April 2, 2019, 3:25pm

So if you installed libtango9 (Tango C++ library) from the official Debian repository, you got the Debian package libzmq5 installed with it, which is actually libzmq 4.2.1. So you are affected by the ZMQ bug.

One solution to overcome this bug is to install cppTango 9.3.3 from the source code or using the Bintray cppTango dev Debian package (install instructions when clicking on the “Set me up!” blue button).
You might need to uninstall libtango9 package to be able to install this development package.

You could also try to find a way to install libzmq >= 4.2.2 or libzmq < 4.2.0.

Hoping this helps.
Reynald

rbourtembourg · April 2, 2019, 4:01pm

If you are familiar with Conda environments, you could also try to install the latest pytango conda package which will install the latest tango conda package (Tango C++ library) too.
I think the bug is only on the event subscriber side. So if your event subscribers are only in Python, this could be an alternative solution. but you will have to run your python Tango clients from a Conda environment.

PS: I hope I didn’t say too many stupid things here. I don’t know much Conda.

jmarrucho · April 3, 2019, 9:28am

Hi,

[quote=“Reynald”]So if you installed libtango9 (Tango C++ library) from the official Debian repository, you got the Debian package libzmq5 installed with it, which is actually libzmq 4.2.1. So you are affected by the ZMQ bug.

One solution to overcome this bug is to install cppTango 9.3.3 from the source code or using the Bintray cppTango dev Debian package (install instructions when clicking on the “Set me up!” blue button).
You might need to uninstall libtango9 package to be able to install this development package.

You could also try to find a way to install libzmq >= 4.2.2 or libzmq < 4.2.0.

Hoping this helps.
Reynald[/quote]

i confirm i have installed Tango 9.2 and libzmq 4.2.1

dpkg -l |grep libzmq
ii libzmq3-dev:amd64 4.2.1-4+deb9u1 amd64 lightweight messaging kernel (development files)
ii libzmq5:amd64 4.2.1-4+deb9u1 amd64 lightweight messaging kernel (shared library)

I will see how to reinstall Tango or libzmq.

Thank you

jkruger · April 4, 2019, 9:08am

I tried this, but it does not work. Neither adding the repository

deb https://dl.bintray.com/tango-controls/debian stretch Release

this will be ignored by the apt update (even if I set this repo to trusted=yes)

nor find the 9.3.3 debian package.

Is there any hint?

Jens

rbourtembourg · April 4, 2019, 11:49am

Hi Jens,
I added this line in /etc/apt/sources.list:

deb [trusted=yes] https://dl.bintray.com/tango-controls/debian stretch main

Then I did:

sudo apt update
sudo apt install libtango9lts-dev

This installed the include files under /usr/local/include and the 9.3.3 lib under /usr/local/lib on my Debian Stretch VM.

Hoping this helps.
Reynald

rbourtembourg · April 4, 2019, 12:01pm

And if you want the debug version of the library (recommended to help to find bugs), you should install:


sudo apt install libtango9lts-dbg

Cheers,
Reynald

jkruger · April 4, 2019, 4:12pm

Hi Reynald,

I tried both version, but nothing helped with the event problem in my case. I got always events with errors, but after installing the zmq version 4.2.5 it works.

Jens

rbourtembourg · April 4, 2019, 4:41pm

Hi Jens,

Thanks for your feedback.
What kind of errors did you get? Do you mean you didn’t receive any event after a while like JCM who created this topic on the forum?
Could you give us more details?

Glad to hear ZMQ 4.2.5 solved your problems.

Cheers,
Reynald

jkruger · April 8, 2019, 9:05am

Hi Reynald,

the story becomes more and more strange. After installing the 4.2.5 version of ZMQ all worked fine for me, but I hadn’t store the output of the ZMQ and TANGO on the screen for my little test program. So I wanted to switch back to the original ZMQ version and deinstalled the 4.2.5 (compiled from sources) version. After starting my test program works now fine as before with the installed 4.2.5 version. Also the restarting of server, client, and Tango-DB didn’t change the behavior. Maybe a system restart will change this.

But What I could tell you from my memory, is that the first event came as expected and all others had the ‘err’ flag set. The error messages gave me the hint that the server would not run, but it wasn’t the case and with the help of Astor and Jive I saw that the events are fired from the server.

I’m using the Debian 9.8 (stretch), the libtango9 (9.2.5a+dfsg1-2), tango-db (9.2.5a+dfsg1-2), and python-pytango (9.2.0-2) packages.

rbourtembourg · April 8, 2019, 9:14am

Hi Jens,

What you are describing sounds like a typical heartbeat problem where you get an error event every 10 seconds, followed by a synchronous read_attribute call because the event client tries to subscribe again because it thinks the remote device server is not running.
Does this corresponds to what you are observing?

jkruger · April 8, 2019, 9:20am

Hi Reynald,

exactly, that is what I saw, before installing and de-installing the 4.2.5 version of ZMQ lib. But now it’s working without any problem.

rbourtembourg · April 8, 2019, 9:29am

Be careful with the ZMQ 4.2.0 or 4.2.1 because as I said to the original poster on this forum thread, there is a bug preventing you to receive more than 1000 events when using these versions, unless you use cppTango 9.3.3

jkruger · April 8, 2019, 9:44am

How sharp is this number? In my short testing program I reached the number of 3000 events and it’s yet working (or in other words: receiving events).

rbourtembourg · April 8, 2019, 10:50am

This number is sharp (1000 by default, corresponding to TANGO_EVENT_BUFFER_HWM) but the problem occurs only if the client subscribes to several publishers.
This is probably why you didn’t see it with your test program.
In this specific case (event client subscribing to at least 2 subscribers), it will receive only 1000 events (it might be 1000 events after having subscribed to the 2nd publisher).
This issue is described in cppTango#444.