Tango Feature Request 4: Defining a standard Tango REST API

General Development
1

Hi tangoers,

During the 29th Tango Meeting we announced the possibility to make a standard Tango REST API.
This feature request has been written in a more formal way below as a tentative for helping the discussion.
Please give any feedback about the definition of this TFR and I will update this page as soon as I can.

I will open a new topic in the forum to discuss especially about the API ;-).

Cheers,
Vincent

Code name

Scope
The scope of this feature request is focus on the definition of a REST API for Tango.
This will include the definition of the HTTP request (URL, content, …)

Context
Several web application have been develop for Tango and a REST api has been often used.
Here is a list of different existing implementations:
[*] mtango - https://bitbucket.org/hzgwpn/mtango/wiki/mTango%20REST%20API
mTango REST API allows you to do the following:

  • list all devices available through the API
    get info about device, its attributes or commands
    read/write attributes
    execute commands

You can find a very complete description on the wiki of the project

[*] Solaris Tango REST
Generic Tango web server

Please find more about Jive on Android project from Solaris:
Android app:

REST interface:

[*] MAXIV BPM application
Specific Tango web server
(TODO)

[*] Gotan - GitHub - hardion/GoTan: Thought about Control System, REST and groovy
Tango module for Gotan.
The purpose is to access the Tango world through the Gotan REST server
A document describing the api: https://www.dropbox.com/s/9nqeabizxlmj6zp/Gotan%20-%20Tango%20REST.pdf?dl=0

Problem to solve
There are many way to define an API using the architecture REST. Also there are different constraints among them the Stateless is the most important. RESTful is defining a set of rule to guarantee a compatibility between all of the Restful software. How to handle command with RESTful compatibility can be one of the topics among others.
Also Tango already defines a standard API on how to access devices, attributes, properties… That can be matched with a Tango Rest Api.

All the web client application won’t be compatible if all the different Tango REST server have a different interface.

Goal and milestones:
The goal is to collect the different experiences to define a stable first version of the standard.
First of all a proposition with different option will be released then to collect the feedback from the overall community. This will define a alpha version of the TANGO REST API
Finally a document will release before the Tango meeting the 30rd for approval from the board. This will define a beta version of the TANGO REST API.

Output:
The first version of this API is expected to be release with:

  • A complete description of the API written in a document, online manual,…
    A set of compliance test with a possible automatic test set
    (Optional)Some possible reference implementation for the only purpose of education and given as an example. Different language can be proposed.
    Release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

Benefit:
A standard API provide an interoperability between the different implementation of the web server and the client. The leverage of the standard will allow to create an active ecosystem.
The expectation is to grow the web development of Tango in a coherent way for the users.

Possible later improvement:
Performance test: Can be part of the compliance test

Edit:

  • added links for the Solaris project (Lukasz)
  • added implementation guidelines (Igor)
2

Hi Tangoers,

Please find more about Jive on Android project from Solaris:
Android app:

REST interface:

All feedback is most welcome :slight_smile:

3

Hi All,

Here I shortly highlight the ideas implemented in mTangoREST API. For more details please refer to the page.

1. Attributes

GET {API_ROOT}/device/sys/tg_test/1/double_scalar – access to the ‘double_scalar’ attr of the TangoTest device. This is simple GET request. The response looks like the following:


{
  "argout":3.14,
  "quality":"VALID",
  "timestamp":1404817953999
}

PUT request writes an attribute:

PUT {API_ROOT}/device/sys/tg_test/1/double_scalar_w=2%2E78

NOTE: that a client is responsible for encoding special characters, i.e. %2E => ‘.’ in this particular case.

Neither DELETE nor POST are not allowed.

1.1 Image attributes:

As our applications are browser oriented images are encoded into JPEG and BASE64 and injected into response:


{
  "argout":"data:/jpeg;base64,/9j/4AAQSkZJRgABAgAAAQABAAD...",
  "quality":"VALID",
  "timestamp":1404817953999
}

On the server side byte array received from a remote Tango device is wrapped with BufferedImage without copying the array and is immediately written into the response.

This approach does not support TIFF images. So the server side must ensure to convert all images into JPEG first.

2. Commands

Command execution is always a GET request, even with an argument:

GET {API_ROOT}/device/sys/tg_test/1/DevString=Hello%20World%20!!!

And this was a major question when I first started to design the API. GET was chosen just for simplicity – one can execute commands directly from the browser’s address bar.

Only GET is supported.

3. Events

GET {API_ROOT}/device/sys/tg_test/1/long_scalar_w.change

Events implementation is based on the Comet specification, i.e. server blocks request till it gets a notification from the remote Tango device. This approach may utilize all available connections in heavily events oriented applications. Therefore client and server must implement multiplexing/demultiplexing of events’ requests. Plus it might be useful to have a timeout so that if server does not get a notification within the specified period of time - it releases the request and responses with an old value. In this case client may resend the request.

Other supported event types are: periodic, user, archive.

Only GET is supported.

4. Family/domain/device browsing

As Tango hierarchy perfectly matches REST’s idea of a resource tree this case is straight forward:

GET {API_ROOT}/devices

lists all the devices defined in the Tango DB attached to this mTango host:


{
  "argout":["dserver/DataBaseds/2",
                  "dserver/Starter/local-starter",
                  "dserver/TangoAccessControl/1",
                  "dserver/TangoTest/test",
                  "dserver/WebCam/local",
                  "local/webcam/0",
                  "sys/access_control/1",
                  "sys/database/2",
                  "sys/tg_test/1",
                  "tango/admin/local-starter"],
  "timestamp":1432556335002}

GET {API_ROOT}/device/

lists all available domains in the Tango DB:


{
  "argout":["dserver","local","sys","tango"],
  "timestamp":1432556231252
}

GET {API_ROOT}/device/{domain}

lists all families in the Tango DB

GET {API_ROOT}/device/{domain}/{family}

lists all members of the family in the Tango DB.

GET {API_ROOT}/device/{domain}/{family}/{member}

displays status of the device:


{
  "state":"RUNNING",
  "status":"The device is in RUNNING state."
}

Only GET is supported.

4. Attributes’, commands’ infos

GET {API_ROOT}/device/{domain}/{family}/{member}/attributes

lists all attributes of the device.

GET {API_ROOT}/device/{domain}/{family}/{member}/{attribute}/info

displays info of the attribute.

Same for the commands:

GET {API_ROOT}/device/{domain}/{family}/{member}/commands

GET {API_ROOT}/device/{domain}/{family}/{member}/{command}/info

4. Caching

Caching in terms of HTTP response header is not supported. But mTangoREST server performs a binary caching of the response for 200 ms.

This was a quick overview of the REST API implemented in mTangoREST API. For more details please refer to the page.

Apart from API specification I think it is important to release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

And do we really need so much time?

In other words currently we will only get a beta version in an year. I believe this task can be accomplished much faster: in three month release alpha version, then perform a beta before next Tango meeting so board can agree on Release Candidate 1.

Cheers,

Igor.

4

[quote=“Ingvord”]Apart from API specification I think it is important to release some implementation guidelines, i.e. caching and optimizations advises. For instance, implementation must be clever enough to combine several requests, i.e. several clients read the same attribute. This is implemented in mTangoREST server.

And do we really need so much time?

In other words currently we will only get a beta version in an year. I believe this task can be accomplished much faster: in three month release alpha version, then perform a beta before next Tango meeting so board can agree on Release Candidate 1. [/quote]

Thanks for your feedback.
Great idea for the implementation guidelines. I will add that to the first post.
Effectively we can have a stable version before the next Tango meeting. An approval from the community can be done online and just have a official presentation to the board.:sunglasses:

/Vincent

5

Hi,

Today I have looked through the Solaris REST project. Here is a small overview of it:

Request:


GET http://localhost:8080/restfultango/rest/localhost:10000/Device/false

This gives a full device list stored in the db:


{
  "numberOfDevices":8,
  "device0":"DEVELOPMENT\/WEBCAM\/0", 
  "device1":"DSERVER\/DATABASEDS\/2",
  "device2":"DSERVER\/TANGOACCESSCONTROL\/1",
  "device3":"DSERVER\/TANGOTEST\/TEST",
  "device4":"DSERVER\/WEBCAM\/DEVELOPMENT",
  "device5":"SYS\/ACCESS_CONTROL\/1",
  "device6":"SYS\/DATABASE\/2",
  "device7":"SYS\/TG_TEST\/1",
  "connectionStatus":"OK",
  "Operation time":0
}

Attribute write/read:

Attribute write request:


PUT restfultango/rest/localhost:10000/Device/sys/tg_test/1/write_attribute/double_scalar_w/3.14

Response:


{"connectionStatus":"OK"}

Read attribute request:


restfultango/rest/localhost:10000/Device/sys/tg_test/1/read_attribute/double_scalar_w

Response:


{
  "devName":"sys\/tg_test\/1",
  "attName":"double_scalar_w",
  "attValue":"measure date: 14\/06\/2015 15:00:30 + 137ms\nquality: VALID\nRead:\t3.14\n",
  "connectionStatus":"OK"
}

API also provides other interesting features:

/ping_device. NOTE from now on = tango/rest/{host}:{port}/Device/{domain}/{family}/{name}

/get_device_info

Attributes:
/get_attribute_list;/plot_attribute/{attr_name} [Reads attribute and returns its values as array or set of arrays depending if attribute is of type SPECTRUM or IMAGE.]; PUT /read_attributes

Tango proxy related:
/set_source;/get_source;/set_timeout_milis
/{value}

Commands:
/command_list_query;PUT /command_inout/{command_name}/{argin};/black_box/{number_of_commands};PUT /extract_plot_data/{command_name}

Dealing with properties:
/get_property_list;/get_property/{prop_name};PUT /put_property/{prop_name}/{prop_val}

Interesting ideas of this API:

Provided access to properties (not implemented in mTangoREST)

Provided access to proxy control like source, timeout (not implemented in mTangoREST)

Arguments passed as a part of URL, i.e. /write_attribute/{attr_name}/{argin} (in mTangoREST arguments are passed via ‘=’, i.e. PUT /{attr_or_command_name}={argin})

My thoughts on this API:

First of all this is not a REST API meaning that it does not follow REST specification. For instance, a tree structure of the resources ain’t satisfied, i.e. instead of /attributes we have /get_attribute_list. API overflowed with redundant case specific commands, like ‘plot_attribute’, ‘put_property’, ‘command_inout’ and so on.

Return value: "measure date: 14/06/2015 15:00:30 + 137ms\nquality: VALID\nRead:\t3.14\n” seems very strange to me saying the least. This is a string client must parse! Other issue is the date – it should be simple timestamp, because it is much more convenient for the client then to do something with it.

Errors handling:
API does not return anything more than “Attribute double_xxx not read. Unable to connect with device sys/tg_test/1” [NOTE: there is no such attribute ‘double_xxx’], i.e. in case of any error raised by a tango proxy on the serer side a client will get “Unable to connect”

Tango related features:

API exports any Tango data base to the outside world. This is very questionable approach, hence very interesting if not ‘Device’ anchor in between, i.e. GET http://localhost:8080/tango/rest/localhost:10000/sys/tg_test/1 seems much more elegant in this case. Another issue with this anchor is that it is capitalized, i.e. server won’t match anything if one asks for …/localhost:10000/device/sys/…

Tango hierarchy is not supported: GET restfultango/rest/localhost:10000/device/sys/tg_test returns 404

Commands execution always requires argin, even if it is a command(void). Moreover in case of void it requires special argin=DevVoidArgument

No events.

No TangoAccessControl integration.

Conclusions

API definitely provides some interesting ideas could be brought into Tango REST API specification. Like access to properties and control of the Tango proxy.

P.S. next week I will look through GOTAN REST API. Meanwhile I will appreciate if other involved people look through mTangoREST, GOTAN and Solaris’s RESTful Tango so we can start to discuss common things and how to combine them into a single Tango REST API

6

I would like to rise here several questions for further discussion:

  1. should API export Tango database? Or API must be limited with only one DB?

  2. Mapping Tango hierarchy to REST API. Strictly following REST specification we will end up with something like this:


GET http://localhost:8080/tango/rest/domains/sys/families/tg_test/members/1/attributes/double_scalar

Which is quite weired. To avoid this weariness I suggest to replace ‘resources’ anchor with actual resource:


GET http://localhost:8080/tango/rest/sys/tg_test/1/double_scalar

So that each resource implements has many relationship, i.e.


GET http://localhost:8080/tango/rest/sys

Returns all families of the sys-domain.

  1. Should command be a GET or PUT request?

  2. Passing arguments: there are three possibilities I see so far: in request’s body; as a part of URL; or as a parameter of URL (double_scalar?argin=3.14). So which one should we choose?

7

Unfortunately I was not able to full-scale test the GoTan project. As the only request it servers is

GET http://localhost:8080/gotan/objects

This one returns an array of devices defined in the Tango DB:


["dserver/DataBaseds/2","dserver/Starter/local-starter","dserver/TangoAccessControl/1","dserver/TangoTest/test","dserver/WebCam/local","local/webcam/0","sys/access_control/1","sys/database/2","sys/tg_test/1","tango/admin/local-starter"]

Any other GET request to the server returns 405 (Method is not allowed). And POST an attribute, e.g. POST http://localhost:8080/gotan/sys/tg_test/1/double_scalar crashes the server:


11:52:38.958 [ClientMessageReceptor2] INFO  jacorb.orb.iiop - Client-side TCP transport to 192.168.2.100:2179 closed.
11:52:38.958 [ClientMessageReceptor2] DEBUG jacorb.giop.conn - ClientGIOPConnection to 192.168.2.100:2179 (4d0f2471): will wait until connected
2015-06-20	12:33:04	127.0.0.1	-	-	8080	PUT	/gotan/sys/tg_test/1/double_scalar		405	487	4	4	http://localhost:8080	Apache-HttpClient/4.3.2 (java 1.5)	-
2015-06-20	12:33:33	127.0.0.1	-	-	8080	GET	/gotan/sys/tg_test/1/double_scalar		405	487	0	2	http://localhost:8080	Apache-HttpClient/4.3.2 (java 1.5)	-
Stopping the internal [HTTP/1.1] server

But OK, since it is only a proof of the concept and API is very well described in this document (https://www.dropbox.com/s/9nqeabizxlmj6zp/Gotan%20-%20Tango%20REST.pdf?dl=0) Its implementation can be omitted.

The only thing from my point of view worth to stress is that API is a little bit messy in terms of using http methods. For instance, attribute write must be PUT not POST, as it does not create anything new, just writes a new value, i.e. updates an attribute. Read attribute must be GET and so on.

Anyway next week I will prepare a draft document that defines REST API for Tango.

8

REST API specification proposal ver. 0.1

Any input is welcome!

9

Hi,

It has been useful for me to look at this since I am involved in a project where we are going to use a rest api as well. It looks a well though api, I just have a few comments/questions:

[*]Device Commands: In the forum it is specified that commands are still to be decided if they are GET or PUT requests (with a first choice for GET), however in the proposal api it is being used POST. I like the idea of being able to test easily directly through the browser, however I feel more inclined to think a command as a PUT request since you are acting on a device, i.e., triggering the execution of whatever actions the command has.

[*] When talking about partial answers, the GET request ~hangs from the device only? The following requests would still be valid?

GET /mtango/rest/device/sys/tg_test/1/ampli?fields=argout
GET /mtango/rest/device/sys/tg_test/1/ampli/info?fields=unit, format
GET /mtango/rest/device/sys/tg_test/1/info?fields=classname

It migth be easier to filter in this way.

[*]Managing tango database data, such as get db info, export devices, etc. Therefore

GET /mtango/rest/database/info  (answer: number of devices exported, etc.)
PUT/GET /mtango/rest/database/commands/export?argin=device_name

[*]Properties: when writing new values for several properties, any property not specified will be deleted. I feel it is a bit risky and prone to property losses. I would suggest to leave the rest of the properties unchanged, a safer approach.

I hope it these comments help you somehow, cheers.

Mikel

10

Hi Mikel,

Thanks for your input!

I agree - PUT seems a little bit more natural, as it updates the “state” of the device, i.e. executes a command.

There will be a generic filter implementation for any response, i.e. one can always request only that portion of the response which is actually needed. In terms of implementation this will be a servlet filter on the server side (at least I think to implement it in this way).

Tango Database is just another Tango device. You can access it using the same API.

I agree that this approach is quite misleading and counterintuitive. I can not image any scenario where it can be useful.

In fact in the next edition of the proposal I will remove PATCH request at all. Apart from introducing unnecessary complexity there is also a problem - it might not be implemented everywhere. Like in raw Java HTTP library.

11

Hi all,

I’m looking at the proposal for REST API posted by Ingvord. First of all, nice work, overall it looks great! I have a few comments off the top of my head.

  • Regarding the CamelCase vs. snake_case question, I think it would just cause confusion to automatically try and convert between them. However in order to be consistent with Tango, the API should definitely be case insensitive for all Tango names. What case should be returned by the API I suppose must remain undefined (i.e. determined by the server).
  • I suppose the “/{domain}/{family}/{member}…” paths should always be prefixed by something (e.g. "/devices/{domain}/… otherwise there could be some confusion (hypothetically there could be a domain called “devices”…). But maybe that was implied.
  • Would it make sense to also add the server tree (like Jive) in a similar way, e.g. "/servers/MyServer/1/MyClass/some/device/name/..? I guess it’s not as useful from a pure user perspective, but in order to allow full database configuration abilities it might be practical. But perhaps in that case, we should separate the “device” and “server” API specifications so that one does not have to wait for the other. I have a feeling that an API that allows e.g. creating servers and so on would get complicated…
  • I agree that the PATCH operation is probably more confusing than useful. I do like the ability to read/write several attributes in one call.
  • In the response for reading an attribute, how about using “value” instead of “argout”? Anyway we also need to be able to get the write value (“w_value”). These are the member names used in the Tango DeviceAttribute class so it seems consistent to keep them. I assume the choice to go with “argout” was made so that attribute and command results are consistent? I’m not sure what’s more important here; consistency across the API, or with Tango…

I’m playing around with a simple python implementation, with the intention to see if I run into any corner cases or other questions. I’ll be back in that case :slight_smile:

12

Hi Johfor,

Thank you very much for the input!

I tend to move this to a particular client implementation, i.e. Feature Request #6 - Tango Web Application (aka Jive in browser via REST API). As you correctly mentioned it does not give a lot to the end users, but increases API and implementations significantly.

BTW what do mean under full database configuration abilities? From my vision one has them just operating on the databaseds, am I wrong? For instance,

PUT http://host/tango_host_10000/devices/sys/databaseds/2/DbRenameServer?input=[TangoTest/test,TangoTest/development]

Anyway, we can add it later.

Agree with value.

What for do you need w_value? In my philosophy GET or PUT to device’s attribute must always return the latest actual value of the attribute (PUT sometimes may return different value from what user has sent if there is a race).

This is awesome. In the best case we can announce at least two implementations on the next Tango meeting. Keep me informed.

13

Hi All,

Here is a new version of Tango REST API specification - Release Candidate 1 aka RC1.

There will be some minor changes but as for now I want to freeze the specification to start implement it.

Hopefully this one will make it to Executive Committee next Tango meeting, i.e. no major issues will occur.

14

Hi,

I agree the w_value will be useful to have. Because the set+read values are closely related. Otherwise you have to cache the write value but you might have concurrency problems ensuring they refer to the same value in time. Is there a way the w_value can be supported?

You are right about the database being a device which can use the same api. The same is true for the admin device in each server.

Good news that coding on the first implementation is soon going to start.

Kind regards

Andy

15

I do not know about set+read values. While dig in into it, could you please give me an example how it is used?

16

Regarding DB operations: I guess you are right, all operations are available through the database device commands. But this is sometimes pretty cumbersome, in part due to the limitations in the TANGO command arguments (expressing everything through lists of strings). So a more “friendly” API would probably be useful, e.g. where you can put many properties in one call, etc, but it can certainly wait. Perhaps it’d make sense to call the API under discussion the “client” API, and later perhaps define a “database” API? This split would be in line with the PyTango API at least.

Regarding read/write values: For example, since a TANGO device is (usually) backed by some hardware which may have any kind of behaviour, such as being slow to react to changes, it can be useful to be able to tell if the last written value differs from the read value from the hardware. Also, if the set value is changed by someone else, it’s good to be able to tell that it’s not just the read value that is changing, which could be the case if the hardware is overloaded, etc. Anyway, would there be an issue with just adding a “w_value” field to the results for writeable attributes?

I’m looking at the proposed RC1 API and I think it looks pretty complete, awesome job! I’ll be back if/when I have some concrete comments.

17
  • How about moving hyperlinks out of the “_meta” field into a separate “links" field? I’ve seen this used elsewhere and it would make it easier to distinguish what’s what. Also, what’s the reason for the prefixing of "” to the keys in these fields? There should be no risk of name collision there.
  • Including the in all links seems unnecessary, it should be enough to assume the API entry point. At least it could be optional.
  • For consistency, perhaps the DevCommandInfo stuff (in_type, …) about commands could be put in an “info” field just like for attributes?
  • I’m a bit unclear on how the filters work; do they apply recursively to all fields?

All in all it looks great, but I haven’t gotten to trying to implement it yet.

18

_meta refers to the implementation related stuff. Eventually it contains only links at present. So, yes, maybe we can rename it to _links.

Prefix is to stress the fact that these info is not related to Tango.

Yes, it is implementation’s choice to put links relative to the entry point as well as to put an absolute path. Client should work fine in both cases.

Ok.

Yes. Filter scans the whole response and keeps (or removes) specified fields only.

19

Just my two cents:

In the restfull API, I have noticed the followed missing features compared to the C++ API.
This does not mean that they have to be implemented but in case you did not know they exist:

  • It is not possible to ping a device (ping() method on a DeviceProxy)
  • It is not possible to retrieve the device black-box
  • The source parameter is not handled meaning you rely on the default one which is CACHE_DEVICE
  • Is it possible to do a “set_attribute_config()”?
  • What about the history related calls (read_attribute_history() and command_inout_history())
  • What about the write_read_attribute() and write_read_attributes() calls (if it follows a RESTfull api)
  • What about the new pipe related calls (too new may be)

For my understanding, if I want to read several attributes in one call, what do I have to give to the GET verb?

Regards

Emmanuel

20

REST becomes popular :slight_smile: Good…

In case a device is not there client will get standard failure response. If ping does what I think, i.e. checks that Tango server is alive – you simply do not need it.

Could you please provide a usage scenario for this?

I think is must be configured on the Web server side via special admin interface. As 99,9% of applications will just get the values and display them this should be fine?!

This I do not know – what are the usage scenarios?

Write_read = write + read, i.e. finish write then read? Well any PUT request to an attribute(s) is write_read, i.e. synchronous, unless client specifies async=true

Well, yes… I took Tango 8 when started to work on this spec.

This is a tricky question. It is not intended to read multiple attributes at once. This is due to built-in HTTP cache. Reading only one value allows setting Last-Modified and Expires response headers. And I think this is more important than multiply reading in this case. Web server still can multiplex several requests into one and send read_attributes to a remote Tango server.