Panic-The Alba Alarm System

srubio · January 26, 2016, 8:24am

At ALBA we have several ways to get logs from PyAlarm:

Using the SNAP database, but as it is not multihost you can’t use it.
Reading the PastAlarms attribute of the PyAlarm devices. It contains all the alarms triggered since the device was restarted. It should work for you, as taurus can get attributes from other tango hosts .
Generate an Alarm log file; using the LogFile property of the PyAlarm server you can save all PyAlarm actions to a file. As it is file-based you won’t need to care about tango hosts; but it is very hard to maintain in the long term.
Using a mail archive. We have some internal email addresses (e.g. alarms@vacuum) that record every single mail sent to them (just like the tango mailing list archives). Adding this address to all alarms allow us to have a full-log of all messages sent. The advantage of this method is that it keeps record of both the message and the receivers in CC. All logs are centralized in the mail service, so it is much easier to maintain.

If you want my advice, I would use a mix of the 2nd (for taurus gui’s) and 4th method (for long-term logs).

vhardion · February 3, 2016, 10:05pm

Hi,

We are also using the Alba Alarm system at MAX IV.
For the long term storage of the alarm we use an elastic search database + kibana for the UI. Our user are quite happy about kibana although this is not integrated into any taurus gui (for that we are using Panic).
This has the advantage to easily define query and to see the events over a time.

my 20cts,
Vincent

jforsberg · February 5, 2016, 10:44am

Just to expand on Vincent’s reply:

we’ve added a tiny feature to PyAlarm which pushes each alarm event as a JSON document to a simple “logger” device (using a command), which in turn stores the event in elasticsearch. The historical data can then be viewed through the kibana web UI, where users can do various filtering and also set up specific views. So far it has been pretty solid, with very low maintenance.

I’m attaching a kibana screenshot from our controlroom. The UI is a bit strange but powerful once you get used to it. However, the main benefit is that we’re not developing it ourselves

Caveat: we’re currently using ES 1.X and kibana 3, but the current version of ES is 2.X and kibana 3 is no longer compatible. Kibana 4 is a complete rewrite and works quite differently, with an even more confusing UI. We’re not sure whether to migrate or how.

srubio · February 6, 2016, 4:43pm

Very nice tool, Johan

It looks very nice, and we would be very glad to integrate this tools in the main branch of PyAlarm (maybe as an optional feature like the smslib).

I’m working in the next Panic major release, so if you can send me some link and/or notes I will give it a try.

Thanks

cfischer · August 27, 2018, 11:03am

Hello,

I just started to try out PANIC. Has there been any news on what you discussed last for logging alarms?
In principle I am not so much interested in archiving, but since this SNAP is a feature of the PANIC gui, I would like to understand how I can make it work.
What I did was downloading Tango Java Archiver (I only sourced SNAp-related sql-files from db-folder)
installed MYSQL-python
installed PyTangoArchiving
I changed the properties of my PyAlarm DS to ‘UseSnap: True’, ‘CreateNewContexts: True’

Now, when I start PyAlarm, one message says: “Creating SnapAPI object …/n ERROR:snapArchiver.DbHost property not defined”
I guess that means I need to add this property to the PyAlarm DS but am unsure of what it needs to be.
In the PANIC gui clicking on “Alarm History Viewer” a dialog-window tells me that snapshots are not available and more specifically, when I click “Open Snapshot” the message is “Could not establish connection to SnapManagerDS. Please check if DS is running or if credentials are correct”.
I certainly didn’t have this DS running. Where is this one? What do I need to do here? Do I need additional packages? How do these snaps look like in the gui? I imagine that they would just look like the last-report of certain alarms, only logged so that the reports are kept.

Thanks for enlightening me and sorry if there is already an easy-to-understand documentation that I missed.

pgoryl2 · August 27, 2018, 11:46am

Hi CoraF,

It sounds like the SNAP archiving itself is not properly configured.
Have you tried to run Bensikin (https://tango-controls.readthedocs.io/en/latest/tools-and-extensions/archiving/bensikin.html?highlight=Bensikin) to check if you can manually create some contexts and then take some snapshots?
You may need to dig into SNAP documentation which is included in ArchivingRoot.zip, for how to setup SNAP.

Below are example settings from the TangoBox VM:

SnapManager class properties:

DbHost: tangobox-archive
DbName: snap
DbSchema: snap
isRac: false

And archiving/snap/snapmanager.1 device propereties:

DbHost: tangobox-archive
DbName: snap
DbPassword: snapmanager
DbSchema: snap
DbUser: snapmanager

And SnapArchiver archiving/snap/snaparchiver.01_01 properties (I use to setup both device and class):

beansFileName: beansBeamline.xml
DbHost: tangobox-archive
DbName: snap
DbPassword: snaparchiver
DbSchema: snap
DbUser: snaparchiver

Best regards,
Piotr

cfischer · August 27, 2018, 1:08pm

Thanks, Piotr!

I don’t know anything about archiving tools, so really my dumb question: How to run those three Snap device servers and how to run bensikin? If I am not blind the doc on SNAP is mostly a user guide on Bensikin and doesn’t explain how to launch it actually. Where are the executables? How to setup the database (ok, for this I would take your input for the properties)?
Thanks!

pgoryl2 · August 31, 2018, 9:55am

Hi, I didn’t manage to answer quickly and unfortunately (for this answer) I am starting week of vacations (which is fortunate for me ). Yes, there is missing documentation on how to make SNAP running. It is on pending list for documentation project.
So, to be quick, you may look on TangoBox archiving container related scripts for reference: tangobox/docker/archive at master · tango-controls/tangobox · GitHub, be careful there is (as far as I remember) a bug for beansFileName property value. The working value is as the above: beansBeamline.xml

Best regards,
Piotr

abence · October 17, 2018, 3:43pm

I’m testing PANIC 6.5.1 in tangoBox 9.2.
I create devices with same pattern like in network Machine Control at SOLEIL
the pattern is X/Y/Z.
but alarm with comprehensionList doesn’t work

the same pattern without dot work

('obj', Alarm(TC_UpperFixedValue:test/pyalarm/1:1539779848.0))
('device', 'test/pyalarm/1')
('formula', 'any( [ s > 21 for s in FIND( test/VI/TC*/Value )])')
('test', 'any( [ s > 21 for s in FIND( test/VI/TC*/Value )])')
TangoEval: TangoEval.parse_formula: Replacing FIND( test/VI/TC*/Value ) with [test/vi/tc.01/value, test/vi/tc.02/value, test/vi/tc.03/value, test/vi/tc.04/value, test/vi/tc.05/value, test/vi/tc.06/value]
[b]TangoEval: parse_variables(...): [('test/vi/tc', '', ''), ('test/vi/tc', '', ''), ('test/vi/tc', '', ''), ('test/vi/tc', '', ''), ('test/vi/tc', '', ''), ('test/vi/tc', '', '')][/b]

('obj', Alarm(one_TC_UpperFixedValue2:test/pyalarm/1:1539782149.0))
('device', 'test/pyalarm/1')
('formula', 'any( [ s > 21 for s in FIND( test/FC/TC*/Value )])')
('test', 'any( [ s > 21 for s in FIND( test/FC/TC*/Value )])')
TangoEval: TangoEval.parse_formula: Replacing FIND( test/FC/TC*/Value ) with [test/fc/tc01/value, test/fc/tc02/value, test/fc/tc03/value]
TangoEval: parse_variables(...): [('test/fc/tc01', 'value', ''), ('test/fc/tc02', 'value', ''), ('test/fc/tc03', 'value', '')]

the fomula is good ? or i make a mistake?

pgoryl2 · October 17, 2018, 9:17pm

Hi,
Unfortunately, it looks like it is a way TangoEval deals with dots in names, at this moment.
Looking through fandango code, “parse_variables” is accepting only alphanumeric (a-zA-Z0-9-_) for device name parts.

Fortunately, It seems that it is enough to extend a regex in the line:

redev = '(?P<device>(?:'+alnum+':[0-9]+/{1,2})?(?:'+'/'.join([alnum]*3)+'))'

.
To:

redev = '(?P<device>(?:'+alnum+':[0-9]+/{1,2})?(?:'+'/'.join([alnum]*3)+'(?:\.[0-9]+)?))'

in fandandgo/tango/defaults.py and tangoeval.py

I will prepare a related pull request for fandango.

srubio · October 19, 2018, 8:07am

Hi,

This same “issue” has been resolved in other fandango-related packages like SimulatorDS

I can patch this parsing of dots in device names … but my question is, Does Tango support dots in device names?

As far as I remember, some versions of Jive had serious problems writing/reading properties whenever there was dots in the names. I think it was caused by dots being wildcards on regular expressions.

Sergi

srubio · October 19, 2018, 8:18am

Hi Piotr,

Actually, the problem is another one. TangoEval should be using the regexp as defined in fandango.tango.defaults; that were already modified to allow dots on names.

I’ll push the change to develop on github so it can be tested,

Sergi

[quote=“pgoryl”]Hi,
Unfortunately, it looks like it is a way TangoEval deals with dots in names, at this moment.
Looking through fandango code, “parse_variables” is accepting only alphanumeric (a-zA-Z0-9-_) for device name parts.

Fortunately, It seems that it is enough to extend a regex in the line:

redev = '(?P<device>(?:'+alnum+':[0-9]+/{1,2})?(?:'+'/'.join([alnum]*3)+'))'

.
To:

redev = '(?P<device>(?:'+alnum+':[0-9]+/{1,2})?(?:'+'/'.join([alnum]*3)+'(?:\.[0-9]+)?))'

in fandandgo/tango/defaults.py and tangoeval.py

I will prepare a related pull request for fandango.
[/quote]

gabeille · October 19, 2018, 12:54pm

Hi Sergi,

At SOLEIL, the dots in device names are widely used. I don’t remember experiencing any issues with it, neither with jive or any other tango client.

Cheers,

Gwen.

abence · October 24, 2018, 7:22am

Thanks for your answers
Like gwen says at SOLEIL we have many devices with dot and i really want finish my test with a good conclusion to encourage IT Group to put PANIC in control room