Internet Draft K. H. Wolf Document: draft-wolf-http-select-00.txt University of Ulm Category: Informational June 1999 Experiences with HTTP-SELECT and Asynchronous Updates Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract Many Internet services need asynchronous notifications. At the same time there is an increasing number of protocols which use HTTP or HTTP-alike request-response transactions. This document proposes an extension to application protocols which enables asynchronous notifications from server to client through client initiated request-response transactions. The extension applies the Unix select() pattern to TCP connections of HTTP-alike protocols. Hence it is called HTTP-SELECT. Structure of the Document The document starts by introducing the reason for this extension in general and in the specific case where the mechanism has been developed. It then gives implementation details, abstract syntax, and examples. Table of Contents Wolf Informational - Expires December 1999 1 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 1. Reason .............................................................2 1.1. Notifications vs. Request-Response .............................2 1.2. The Virtual Presence Case ......................................3 2. Implementation Details .............................................3 2.1. Introduction ...................................................3 2.2. Nameless Subscription Sets .....................................4 2.3. Typical Request Sequence .......................................4 3. Extension Syntax ...................................................5 3.1. Basic Definitions ..............................................5 3.2. Messages .......................................................6 3.3. Example ........................................................7 3.4. Optmizations ...................................................9 3.5. Traffic .......................................................10 4. Security Considerations ...........................................10 5. References ........................................................10 6. Acknowledgments ...................................................11 7. Author's Addresses ................................................11 1. Reason 1.1. Notifications vs. Request-Response An increasing number of Internet services use HTTP as transport protocol, an HTTP-alike protocol, or another request-response based protocol. They have in common that TCP connections are established by the client to send the request from the client to the server. The response uses the same TCP connection from server to client. In the world of protected intranets and firewalls this is a requirement of many services. An increasing number of Internet services needs asynchronous notifications. Notifications from client to server are easily embedded into a client initiated request-response protocol. But asynchronous notifications from a server to clients do not fit well into such a protocol. Services which want to keep a strict request-response architecture have two options to implement server-client asynchronous notifications: - Notification as requests from server to client with a TCP connection established from the server. This is not feasible for services where servers are outside of the firewall. - Notification as a response to a (long time) outstanding request. This is compatible with firewalls. It is not firewall friendly, though not firewall hostile. However, this is the way chosen by this proposal. Wolf Informational - Expires December 1999 2 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 A summary of such services/protocols which need asynchronous notifications is available in [2]. 1.2. The Virtual Presence Case The development was done while working on an Internet Presence protocol in combination with the Web (Virtual Presence). Internet Presence basically provides users with information about other users in a dynamic fashion. Peers subscribe for each other's online status and get notifications when the status changes, i.e. when a peer "goes online". Virtual Presence augments the user detail with information on the user's virtual locations (VPP [3]). Virtual Presence is a wide area, HTTP-related service. Hence, it must cope with the existing infrastructure of HTTP clients, servers, proxies, and firewalls. The developers decided to encapsulate VPP requests into HTTP to tunnel VPP through HTTP/1.1. VPP requests use HTTP-GET. The data is URL-encoded unless the request carries body data. In this case HTTP-POST is used for the body data. The encapsulation is strict HTTP/1.1 with request pipelining. VPP clients subscribe for asynchronous notifications. The server sends notifications when the value of objects changes. But as mentioned above, connection establishment from the server to the client is not an option. The chosen HTTP encapsulation of VPP uses long living TCP connections from client to server with outstanding (unanswered) requests. The protocol uses a Unix select-like mechanism. 2. Implementation Details 2.1. Introduction The relevant methods of the protocol are SUBSCRIBE, SELECT, and GET. - The SUBSCRIBE method requests asynchronous notifications from the server for a single object maintained by the server. - The SELECT response carries the notifications. - GET requests the object's contents. Typically, a client sends a number of SUBSCRIBE requests followed by SELECT. Then the client waits for the SELECT response. The SELECT method works similar to the select() system call. The request blocks the connection and keeps the connection open until a timeout is reached or an event happens. The server responds with a set of events or an empty set. The client may then request the contents of these objects using the GET method. Wolf Informational - Expires December 1999 3 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 As in the case of Unix select(), the client creates a set of object identifiers for the SELECT request. The set is created by SUBSCRIBE requests which carry the name of the subscription set (set-id). The SELECT request contains the same name. It contains the set-id of the subscriptions for which it is supposed to return events. All events from subscriptions of a set are returned by a single SELECT response. 2.2. Nameless Subscription Sets The set-id may be omitted for nameless sets on a single TCP connection. Both SUBSCRIBE and SELECT requests MUST use the same TCP connection in this case. Subscriptions of nameless sets are lost (unsubscribed automatically) when the connection closes. Nameless sets do NOT work with proxies and/or HTTP requests made by HTTP clients implementations (Web browser, Java URLConnection), since this software may terminate the TCP connection at any time. They do work with socket connections. Implementations are available in C++, Java, and Delphi. 2.3. Typical Request Sequence Assumed a client wants notifications of contents changes of two objects (object1, object2). The client creates a subscription set (set1). It sends the subscriptions, a SELECT request, and waits for the response. If the SELECT response returns an empty set of events then it restarts the SELECT request. If the server returns an event, then the client examines the event. In the case shown here it just retrieves the changed contents of the object. C: client, S: server C: SUBSCRIBE request (object1, set1) S: SUBSCRIBE response (OK) C: SUBSCRIBE request (object2, set1) S: SUBSCRIBE response (OK) C: SELECT request (timeout, set1) <--------+ | ...client waits for response... | | S: SELECT response ( (object1, CHANGED) ) | C: GET request (object1) | S: GET response (object1-data) | | | +-------------- restart SELECT -------------+ If the client wants to cancel the subscriptions, it just sends an UNSUBSCRIBE request. The server first returns the SELECT response and then the UNSUBSCRIBE response (HTTP-pipelining). Wolf Informational - Expires December 1999 4 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 C: UNSUBSCRIBE request (object1, set1) S: SELECT response () S: UNSUBSCRIBE response (OK) C: UNSUBSCRIBE request (object2, set1) S: UNSUBSCRIBE response (OK) The best practice for client implementations is to dedicate a thread of execution to maintain an unanswered SELECT request. - If the server responds, then the client evaluates the events. It might fetch the updated objects and finally issues another SELECT request. - If the server terminates the SELECT transaction due to a timeout then the client restarts the SELECT. - The same happens, if intermediate proxies cut the connection after some time. - Clients can issue other requests at any time, even on the SELECT- connection. In this case the server terminates the SELECT, responds to the other request, and the client automatically restarts the SELECT. 3. Extension Syntax The following syntax specification uses the augmented Backus-Naur Form (BNF) as described in RFC-2234 [4]. This is not a complete application protocol. It is a protocol neutral specification to add the SELECT mechanism to HTTP-alike application protocols. 3.1. Basic Definitions Requests consist of - object, - method, - request specific attributes. Responses consist of - responsecode, - attributes, - optional additional data. http-select-method = SUBSCRIBE | UNSUBSCRIBE | NOTIFY | GET | PUT http-select-time = HTTP-date | delta-seconds http-select-object = *(VCHAR) http-select-reference = *(VCHAR) http-select-eventtype = UPDATED | DELETED | SUBSCRIBED | UNSUBSCRIBED http-select-responsecode = OK | NOT_FOUND | BAD_REQUEST Wolf Informational - Expires December 1999 5 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 | ACCESS_DENIED | SERVER_ERROR http-select-notifymode = EVENT | DATA 3.2. Messages 3.2.1. SUBSCRIBE SUBSCRIBE request: object = http-select-object ; REQUIRED method = SUBSCRIBE ; REQUIRED Attributes: sub-id = http-select-reference ; REQUIRED (subscription id) timeout = http-select-time ; OPTIONAL (lifetime) eventtype = http-select-eventtype ; OPTIONAL (default: UPDATED) set-id = http-select-reference ; OPTIONAL (named set) notifymode = http-select-notifymode ; OPTIONAL (default: EVENT) SUBSCRIBE response: responsecode = http-select-code ; REQUIRED Attributes: timeout = http-select-time ; OPTIONAL (if changed) Additional-data: (none) 3.2.2. GET GET request: object = http-select-object ; REQUIRED method = GET ; REQUIRED Attributes: (none) GET response: responsecode = http-select-code ; REQUIRED Attributes: (none) Additional-data: additional-data = http-data ; OPTIONAL (in case of OK) http-data = 3.2.3. SELECT SELECT request: object = "sys" ; REQUIRED (dummy object) method = SELECT ; REQUIRED Attributes: timeout = http-select-time ; OPTIONAL Wolf Informational - Expires December 1999 6 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 set-id = http-select-reference ; OPTIONAL (named set) SELECT response: responsecode = http-select-code ; REQUIRED Attributes: (none) Additional-data: additional-data = http-select-data ; REQUIRED http-select-data = *[ "" "" http-select-eventtype "" "" http-select-object "" [ "" http-data "" ] "" ] The SELECT response carries events on multiple objects. If the client requests instantaneous delivery for UPDATE events (SUBSCRIBE attribute mode=data and attribute eventtype=UPDATED), then the event contains the object data. The encapsulation of multiple events with optional data into a single response is depends on the application protocol. VPP is based on ASCII object data, so XML with object data between tags is appropriate. Control characters are URL-encoded according to chapter 2.2. of [5]. Other protocols might prefer MIME-multipart encoding to deliver multiple objects in a single SELECT response. 3.2.4. UNSUBSCRIBE UNSUBSCRIBE request: object = http-select-object ; REQUIRED method = UNSUBSCRIBE ; REQUIRED Attributes: sub-id = http-select- reference ; REQUIRED (subscription id) eventtype = http-select-eventtype ; OPTIONAL (default: UPDATED) UNSUBSCRIBE response: responsecode = http-select-code ; REQUIRED Attributes: (none) Additional-data: (none) 3.3. Example The VPP protocol is encapsulated into HTTP. Requests are completely URL-encoded as query parameters (called HTTP-encapsulated). Other protocols might extend HTTP directly with new header fields (called HTTP-alike). Messages will be shown here in both variants. Wolf Informational - Expires December 1999 7 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 A leading "|" denotes a message line. Lines are separated by CRLF. "+" denotes a continued line. 3.3.1. SUBSCRIBE Subscription request HTTP-encapsulated (adapted from VPP): | GET /vpp&object=http://vpp.cobrow.com/user1/status + &method=SUBSCRIBE + &sub-id=subscription123 + &eventtype=UPDATED + &set-id=set456 + ¬ifymode=event HTTP/1.1 | Subscription request HTTP-alike: | SUBSCRIBE /user1/status | Host: vpp.cobrow.com:677 | X-Select-sub-id: subscription123 | X-Select-eventtype: UPDATED | X-Select-set-id: set456 | X-Select-notifymode: event | 3.3.2. SELECT SELECT request HTTP-encapsulated (adapted from VPP): | GET /vpp&object=sys + &method=SELECT + &set-id= set456 + &timeout=86400 HTTP/1.1 | SELECT request HTTP-alike: | SELECT sys | X-Select-set-id: set456 | X-Select- timeout: 86400 The response is delayed until the timeout is reached or one of the subscriptions of the set creates a notification. SELECT response HTTP-encapsulated (adapted from VPP): | HTTP/1.1 200 OK | Content-type: text/xml | | 200 2.0 | | updated | http://vpp.cobrow.com/user1/status | | Wolf Informational - Expires December 1999 8 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 | deleted | http://vpp.cobrow.com/user1/message | A client will then usually request the updated object: | GET /user1/status HTTP/1.1 | Host: vpp.cobrow.com:677 | and restart the SELECT. 3.4. Optmizations 3.4.1. Subscription Clustering Multiple subscriptions could be clustered and sent in a single SUBSCRIBE request. This reduces the traffic, but increases the complexity. Whether this optimization is necessary depends on the application and the traffic pattern. In the current VPP implementation, clients typically issue 4 subscriptions. Subscription clustering is not necessary since the data volume of objects is much larger than the data volume of subscription requests. 3.4.2. Subscription as Extended GET In many cases clients will need not only updates to the contents of an object, but also the initial value of an object. In this case the client will request the contents (GET) and will then SUBSCRIBE to the object. In order to avoid the gap between GET and SUBSCRIBE/SELECT in which events could be lost. Programmers tend to send the sequence SUBSCRIBE, SELECT, GET. However, since the server must keep the response order (HTTP/1.1) it terminates the SELECT to return the GET response. The client then automatically resends the SELECT. Actually the sequence becomes: SUBSCRIBE, SELECT, GET, SELECT. This gets even worse with more objects. If clients always combine SUBSCRIBE with GET, then SUBSCRIBE can be extended to cover the GET functionality, i.e. to return the initial contents of the object. In this case the typical sequence becomes just: SUBSCRIBE, SELECT. 3.4.3. Instantaneous Notifications The basic mechanism always requires the client to fetch (GET) the contents after receiving the notification. The notification and the transfer of the object have been separated for flexibility reasons. But the application may require a quick delivery of the updated Wolf Informational - Expires December 1999 9 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 contents without the additional round-trip of SELECT response and GET request. For this reason the SELECT response (the notification) may carry the object's contents. Whether instantaneous delivery is desired by the client is specified in the subscription. The client adds an attribute (notifymode=DATA) to the SUBSCRIBE request to request instantaneous delivery for a single object. 3.5. Traffic The SELECT method with simple GET proved to be very useful and flexible. In usual scenarios the traffic generated by timeouts (empty responses and re-request) is negligible compared to the amount of object data Both, named subscription sets in multi-threaded clients and nameless sets with HTTP-pipelining on a single connection have been implemented. We suggest to use this method for other notification based protocols under similar conditions. 4. Security Considerations This document proposes an extension to application protocols. The proposed subscription mechanism is an extension to simple GET-access. Both methods involved, SUBSCRIBE and SELECT, need the same protection as HTTP-GET. Instantaneous delivery of object data by the SELECT response provides access to object data, if the name of the subscription set is disclosed. It is recommended to restrict the SELECT response to simple event notification and leave the transfer of object data to GET requests. 5. References 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 2 Rohit Khare, Adam Rifkin , "Scenarios for an Internet-Scale Event Notification Service (ISENS)", INTERNET-DRAFT, , Expires May 1999, November 17, 1998. 3 K. H. Wolf, " VPP: Virtual Presence Protocol", INTERNET-DRAFT, Expires: July 5, 1999, . 4 Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, Internet Mail Consortium and Demon Internet Ltd., November 1997. Wolf Informational - Expires December 1999 10 INTERNET-DRAFT Experiences with HTTP-SELECT June 1999 5 [URL] Berners-Lee T., Masinter L., McCahill M., "Uniform Resource Locators (URL)", RFC 1738, Dec 1994. 6. Acknowledgments Thanks to members of the my working group and the members of the CoBrow projects (EU Telematics for Research RE 1003/RE 4003) who helped to develop the model and implemented it. 7. Author's Addresses Klaus H. Wolf Distributed Systems Dept. University of Ulm 89069 Ulm Germany Phone: +49 (731) 502 4145 Email: wolf@informatik.uni-ulm.de Wolf Informational - Expires December 1999 11