Hi
We have created a tango program which has two read bodies for two of its attributes only.
When the client is connected to it,the memory leak keeps increasing linearly. Where does this memory leak occur? How can we stop the memory leak?
Thanks and regard
TGMRT.
Can you be more specific? You need to provide the source code if you want us to help. Have you run valgrind on the program to try to determine where the leak is?
Andy
We have just run program which is generated by POGO that contains two attributes (one read only and other read/write ). Still we are facing memory leakage.
Please find an attachment of program.
Thanks and Regards,
TGMRT
Hi,
I have downloaded, compiled and ran your Tango class. A strange one! Nothing coded inside the read or write attribute methods!
I have also started a client reading the speed attribute. I have one exception at each reading (because on the server side the
attribute set_value() method is not called) but I do not see any leak. Valgrind also does not report anything.
Where do you notice this memory leak? On the server side, client side?
Regards
Emmanuel
Hello,
we are using tango ver8.1.2 and fedora 19 pre-installed ‘system monitor’ utility to check memory usage by DS process i.e. server process
We also tried ‘TangoTest’ example bundled with Tango installation for the same purpose, there also memory usage keeps-on increasing. Within 16 minutes its Memory usage becomes 151.9MB from 41MB initially.
Thanks and Regards,
TGMRT
I would be surprised if there is a memory leak in TangoTest. It is installed and running in many places and this is the first time a leak is reported. On the Tango9 virtual box for example I have it running with VmSize = 863876 kb and it does not increase. What is different on your system? I presume you compiled tango 8.1.2 and the TangoTest yourself. But this still does not explain the leak you are seeing. The memory usage you list is very low for a Tango server. Where does this number come from? Can you cat the /proc/pid/status file to get the whole picture.
As Manu said we do not see a leak in your (empty) server. Can you try to run it on the Tango virtual box to see if the problem comes from your installation?
Andy
Thanks for your reply.
Here are the results with cat proc/ pid/status
Step 1: I ran the Tango Test program. Attaching Proc_0.txt and Sysmon_0.png. Sysmon is monitored using the system monitor functionality of Fedora 19.
Step 2: After 10 min. once again I took readings. Pl. refer proc_1.txt and sysmon_1.png
The memory was quite stable in sysmon_1. But proc Vmsize reported increase of 65536 kb.
Step 3: Started the client - ATK panel from jive (1 instance) And noted the results in Proc_2.txt and sysmon_2.png
The memory was increasing in sysmon. Also noticed that proc Vmsize increased.
Step 4: Allowed to run it for 15 min. more. And noted the results in Proc_3.txt and sysmon_3.png
The memory was increasing in sysmon. Also noticed that proc Vmsize increased.
I am doing observations in VM.I will let you know its result also.
Can you provide the pointers as to how do we go ahead about fixing this issue?
Also, I am not clear about your statement in the earlier reply “…Can you try to run it on the Tango virtual box to see if the problem comes from your installation?” Can you please elaborate on this? I remember, i followed the instructions shipped with the TANGO tools for installation. How do I go ahead about rectifying it?
Thanks and regards
TGMRT
Hi,
Sorry to ask you one more thing but because we do not have the problem here we need your inputs.
Is it possible for you to install valgrind on your host?
Then run your device server under valgrind control with the following command line
valgrind -v --log-file= --leak-check=yes --leak-resolution=high --num-callers=50 TangoTest test
(assuming TangoTest is in your PATH and test is its instance name)
Let it run during few minutes with your clients being active. Stop the device server and send us the file generated by Valgrind
Regards
Emmanuel
Hello,
thanks for your reply.
Here is the attachment of log file as required.
And please let me know if there was anything wrong with tango installation or any other factor affecting to this problem, so that we can correct it.
Thanks and Regards,
TGMRT
Hi
Here is the attachment of log file which is generated by running TangoTest for 25 minutes.
I hope it will be helpful.
Thank & Regards,
TGMRT
Hi,
Valgrind reports many “definitely lost” bytes which could be seen at a leak. BUT I don’t think it is.
If you read carefully where Valgrind reports about lost bytes, you will see that it is memory allocated in the
class init_device() method. If you do not use the device Init commnamd, this code is executed once at device start-up.
Reading/Writing attributes or executing commands do not trigger this init_device() method execution an dthis is why I would
not really consider this as a real leak.
Anyway, what is sure is that some memory freeing code is missing in the delete_device() method of this class. I have fixed this
in my local copy of TangoTest and I will soon commit this bug fix.
BUT as I explained above, this does not explain a permanent memory increase except if you repeat the Init command again and again.
Is it what you are doing?
Another interesting point: Do you poll some TangoTest device attribute with a modified polling buffer depth (much bigger than the default depth is 10)?
Re,
Well I have just looked into your second valgrind report you just send. In this one there is a real leak of 2976 bytes in 62 blocks. This leak is known and fixed if you apply the patches provided in this site. Go to the Download/Patches page and
get and apply the 4 patches available for Tango 8.1.2
Regards
Emmanuel
Hello,
I have applied patches.
Here are the results with cat proc/ pid/status after applying patches
Step 1: I ran the Tango Test program. Attaching Proc_0.txt.
Step 2: After 10 min. once again I took readings into proc_1.txt
proc Vmsize reported no increment.
Step 3: Started the client - ATK panel from jive (1 instance) And noted the results in Proc_2.txt
I noticed that proc Vmsize increased by around 66000 Kb.
Step 4: Allowed to run it for 15 min. more. And noted the results in Proc_3.txt
Again I noticed that proc Vmsize increased further by around 45000 Kb.
I have also attached valgrind output log file.
Can you provide the solution as how do we go ahead about fixing this issue?
Thanks & Regards,
TGMRT
Hello
We are working with TANGO installed in Fadora 19.
I am just asking whether anyone has tried TANGO on linux system instead of Virtual box?
Thanks & Regards,
TGMRT
Hello,
Yes, Tango is used on many Linux hosts (Red Hat, Debian and other). For instance at the ESRF for the accelerators control system, we have more than 160 hosts running Linux! On top of that, you have to add the Linux crates used on our beam-lines…
Back to your problem.
I am aware of one more possible case which could explain your memory behavior but this one means that your event system is
wrongly configured. I see two ways to look into this:
1 - Take Tango 9 (with patch). This case of memory increase due to wrongly configured event system is handled in Tango 9 but note
that this does not solve why the event system is wrongly configured but it should solve the memory increase.
2 - Stay with Tango 8 and try to understand what happens in your event system. To do so, run the TangoTest device server with
these environment variables:
Start your client (ATKPanel) and let the client run during 2 mins. Then send us the generated log file
Regards
Emmanuel
Hi,
just to add to Emmanuel’s comments. TANGO is used on tens of sites around the world most (all?) of them running various flavours of Linux. AFAIK all of the are running TANGO 8 in production. Only the ESRF is running TANGO 9 in production on a large system. The virtual box is used only as a demonstrator for tutorials. I am not sure where your memory leak is coming from but as mentioned by Emmanuel there is an issue with the event system when the events are badly configured. But are you sure you are using events? TangoTest is not configured to send events by default. You can check with atkpanel to see if it uses events.
I would strongly recommend you move to TANGO 9.1.0 patched unless you have some constraints which prevent this. If you need PyTango then you can mix TANGO 9 and TANGO 8. TANGO 9 could be your base installation and TANGO 8 only for PyTango.
Can you check to make sure you have all the patches applied? Compare your patched version with the patched version you can download here:
If you send me your email and phone number (offline) we can try to contact you directly to understand your problem better and help you more efficiently. You can send it to andy dot gotz at esrf dot fr.
Kind regards
Andy
Hello,
What are the news using Tango 9? Did you had some time to download and compile it? Do you still notice a memory increase with TangoTest using it?
About the log file you sent to Andy, I confirm that it seems to have a problem in your event configuration.
If I take the example of the TangoTest device double_scalar attribute, I see in your log file several call to register
on attr_conf event on this attribute from ATKPanel (for instance at lines 927588, 927886, 938606, 950101, 961545,…)
This is very suspicious. In order to also progress on that point, could you send us:
1 - The TANGO_HOST environment variable used on the server side
2 - The TANGO_HOST environment variable value used on ATKPanel side
3 - The result of the database device command DbGetCSDbServerList. To do, in Jive open DS “Databaseds” until you find its
device (should be sys/database/2 for a default installation), right click and select “Test device”. In the Commands tab, select the command
DbGetCSDbServerList and click “Execute”
Regards
Emmanuel
Hi Manu,
Yes. We downloaded and compiled TANGO 9(with patch). And there is still memory increase for TangoTest DS.
We are running server and client on the same machine so the TANGO_HOST entry is same for both which is:
TANGO_HOST = :10000
Command: sys/database/2/DbGetCSDbServerList
Duration: 20 msec
Output argument(s) :
array length: 1
[0] :10000
Note: our host name is combination of 10 alphanumeric characters in lower case.
As per your suggestion we also tried the same scenarios with ‘TANGO_HOST = myhost:10000’ and ‘myhost’ is defined in ‘/etc/hosts’ to the IP of the system.
After making above settings we are still observing memory increase of TangoTest DS.
Thanks and Regards,
TGMRT
Hi,
Could you try to restart your database server with a TANGO_HOST environment variable set to the SAME value than the one
used for ATKPanel and your TangoTest device server?
Regards
Emmanuel
Hi Manu,
Thanks for your help. As you suggested we made the changes of host-name. And restarted the Tango Database. Now we can still observe memory increase but its not as rapid as it was before.
Same behavior is observed with TANGO 9 installation.
Thanks and Regards,
TGMRT