haptics Media Type: The 'haptics' Top-Level Media Type

Internet Engineering Task Force (IETF)                   Y. K. Muthusamy
Request for Comments: 9695
Category: Standards Track                                     C. Ullrich
ISSN: 2070-1721                                               March 2025


                   The 'haptics' Top-Level Media Type

Abstract

   This memo registers and documents the 'haptics' top-level media type,
   under which subtypes for representation formats for haptics may be
   registered.  This document also serves as a registration for a set of
   subtypes, which are representative of some existing subtypes already
   in use.

Status of This Memo

   This is an Internet Standards Track document.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Further information on
   Internet Standards is available in Section 2 of RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   https://www.rfc-editor.org/info/rfc9695.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Revised BSD License text as described in Section 4.e of the
   Trust Legal Provisions and are provided without warranty as described
   in the Revised BSD License.

Table of Contents

   1.  Introduction
   2.  Background and Justification
     2.1.  MPEG ISOBMFF
     2.2.  Haptic Sub-Modalities
     2.3.  Another Human Sense
     2.4.  Commercial Uptake
     2.5.  Haptic Data Formats in Use
     2.6.  Haptic Subtypes (Envisioned Standards)
     2.7.  'application' Top-Level Type Not Suitable
   3.  Security Considerations
   4.  IANA Considerations
     4.1.  Definition and Encoding
     4.2.  Registration Procedure
     4.3.  Subtype Registrations
       4.3.1.  IVS Haptics Type
       4.3.2.  HJIF Haptics Type
       4.3.3.  HMPG Haptics Type
   5.  Normative References
   6.  Informative References
   Authors' Addresses

1.  Introduction

   The term 'haptics' refers to the generation of touch-related
   sensations in a device or interface.  Haptics is widely used in
   consumer devices in order to provide touch-based feedback to users.
   The most common use of haptics is in mobile devices, where it is used
   to provide feedback to users interacting with the touchscreen, e.g.,
   typing on a virtual keyboard.  Haptic technologies are unlike audio
   and visual enabling technologies in the sense that they require some
   form of actuation in order to create a tactile sensation.  For mobile
   phones and game controllers, these actuators are typically small
   vibrating motors.  For large touchscreens in vehicles, these
   actuators can be specialized piezoelectric materials.  Haptic
   capabilities are found in nearly every modern smartphone, game, and
   virtual reality controller, making these devices an ideal target for
   enhanced media experiences.

   Internet Media Types [RFC6838] are used to label content carried over
   Internet protocols.  This document defines a new top-level type,
   'haptics', according to [RFC9694].  This top-level type indicates
   that the content specifies haptic data.  Under this top-level type,
   different representation formats of haptics may be registered.

2.  Background and Justification

   Haptic signals provide an additional layer of entertainment and
   sensory immersion for the user, when combined with audio and video
   signals.  Haptic tracks, in separate files, can be combined with
   audio/video files and played back in sync to provide an overall
   immersive media experience (audio, visual, tactile) for the user.
   More recently, haptic tracks embedded in standard file formats, such
   as ISOBMFF (ISO Base Media File Format), enable playback of the
   haptic signals over one or more actuators, simultaneously with audio
   and video playback [ISOBMFF-IS].  Haptic signals are also part of
   media streams that use RTP, such as those for streaming games, XR,
   and wearables.

2.1.  MPEG ISOBMFF

   Historically, there has not been a registration of formats for
   haptics.  However, haptics was proposed as a first-order media type
   (at the same level as audio and video) in ISOBMFF in April 2020.  The
   proposal has since progressed to International Standard, and was
   published in January 2022 [ISOBMFF-IS].  Haptics is officially part
   of the ISO/IEC 14496-12 (ISOBMFF) standard, 7th Edition.  Given this
   development, a strong case can be made for haptics to be added to the
   list of top-level media types recognized by the IETF.

   We envision the following designations for haptics in mp4 files, once
   the top-level type 'haptics' is registered:

   *  'haptics/mp4' - mp4 files with just haptic tracks and no audio or
      video in them (e.g., streaming games, haptics files for haptic
      vests, belts, gloves, etc.)

   *  'video/mp4' - mp4 files with video, audio, and haptics (to ensure
      consistency with existing mp4 files with video content)

   *  'audio/mp4' - mp4 files with audio and haptics (to ensure
      consistency with existing mp4 files with audio content without any
      video)

2.2.  Haptic Sub-Modalities

   There are multiple sub-modalities of haptics:

   *  Vibrotactile (touch, vibration)

   *  Kinesthetic (force feedback)

   *  Surface (surface friction)

   *  Spatial, non-contact (ultrasound)

   *  Thermal (temperature)

   Therefore, designating 'haptics' as a top-level media type enables
   the definition of data formats pertaining to these sub-modalities in
   a more streamlined manner.  This would not be possible if 'haptics'
   were to be placed under other top-level types like 'audio', 'video',
   or 'application'.

2.3.  Another Human Sense

   The top-level media type 'audio' pertains to the human sense of
   hearing; the top-level media type 'video' pertains to the human sense
   of seeing; so it only makes sense for the (equally important) human
   sense of touch to be represented by another top-level media type
   'haptics'.  Placing 'haptics' under 'audio' or 'video' is not
   reflective of the kinds of files or use cases that would need haptics
   but have nothing whatsoever to do with audio or video.

2.4.  Commercial Uptake

   Haptics is rapidly becoming a standard feature of consumer electronic
   devices.  For example:

   *  iPhone (231+ million units sold in 2023 [iPhone-Sales]): built-in
      support for haptic-encoded data

   *  Android (1.27+ billion units sold in 2023 [Android-Sales]
      [GlobalSmartPhone-Sales]): API support of haptic buffers

   *  W3C (HTML vibration API [W3C-Vibration]): Optionally supported in
      mobile web browsers.  W3C has also defined vibration extensions
      for gamepads [W3C-Gamepad]

   *  Game consoles (45+ million units sold in 2023
      [GameConsole-Sales]): MS Xbox, Sony PlayStation, Nintendo Switch,
      etc.

   *  XR devices (30+ million units sold in 2023 [XRDevice-Sales]):
      OpenXR haptic API

   Haptic media is expected to be commonly exchanged between these
   devices.  Since they represent the majority of consumer electronics
   (CE) devices, a strong case can be made for 'haptics' as a top-level
   media type.

2.5.  Haptic Data Formats in Use

   There are multiple instances of existing haptic data formats that
   will live as sub-types under the proposed 'haptics' top-level media
   type.  While these subtypes have *not* been standardized (yet), the
   prevalence of these haptic data formats in a large number of devices
   around the world, pre-dating the standardization of haptic tracks in
   ISOBMFF, provides a compelling argument for 'haptics' to be
   designated as a top-level media type:

   *  'ahap': The AHAP haptic data format [AHAP] is currently the
      standard encoding on all iOS devices + iOS connected game
      peripherals.  The format has seen usage and adoption beyond Apple
      devices as well, with decoders available for Android and other XR
      systems.

   *  'ogg': Google has introduced a proprietary extension to the OGG
      format in the latest version of Android 11.  This encoding enables
      haptic media to be stored in OGG files.

   *  'ivs': The IVS haptic data format [MPEG-Haptics-Encoder] is in
      use:

      -  In mobile phones from LG Electronics (specifically, the models
         V30, V40, and the newest V50) that are sold worldwide

      -  In gaming phones from ASUS (specifically, models ROG, ROG Phone
         II, ROG Phone 3) that are sold worldwide

   *  'hapt': The HAPT haptic data format is currently a vendor-specific
      format that is in use:

      -  In mobile haptic advertising (for W3C devices)

      -  The following Japanese game developers use the HAPT format as
         part of Immersion's TouchSense SDK:

         o  KLAB: https://www.klab.com/en/

         o  Craft&Meister: https://www.crafts-meister.co.jp/pc/
            company_en.html

      -  Tencent is using the TouchSense SDK for their popular social
         media application QQ and live streaming application NOW:
         "Immersion Announces Tencent Licenses TouchSense Technology to
         Deliver Next-Generation Interactive Experiences" (https://web.a
         rchive.org/web/20241201160955/https://www.businesswire.com/news
         /home/20171026006443/en/Immersion-Announces-Tencent-Licenses-
         TouchSense%C2%AE-Technology-Deliver)

   Given the widespread use of these subtypes, it makes sense for
   'haptics' to be a top-level media type.

2.6.  Haptic Subtypes (Envisioned Standards)

   The MPEG ISOBMFF proposal included an informative annex of known
   haptic coding formats with proposed FourCCs (four-character codes)
   for them.  The first two, hmpg and hjif, have been registered per
   this document and have already been standardized in ISO-IEC 23090-31:
   Haptics coding [MPEG-Haptics-Coding].  As for the remaining codes,
   once the format names have been finalized and standardized, these
   types should also be registered as subtypes of the 'haptics' top-
   level media type:

   *  'hmpg': the MPEG-I haptics streamable binary coding format
      described in ISO/IEC 23090-31: Haptics coding
      [MPEG-Haptics-Coding]

   *  'hjif': the MPEG-I haptics JSON-based interchange format described
      in ISO/IEC 23090-31: Haptics coding [MPEG-Haptics-Coding]

   *  IEEE 1918.1.1 vibrotactile coding standard [IEEE-191811] being
      developed under the IEEE Tactile Internet initiative as part of
      the 5G Ultra-Reliable Low-Latency Communication (URLLC) profile.
      Format name is yet to be finalized.

   *  Enumerated effects haptic coding format (based on MIDI).  Format
      name is yet to be finalized.

   *  Audio-to-vibe haptic coding format (automatic audio-to-vibration
      conversion algorithms).  Format name is yet to be finalized.

2.7.  'application' Top-Level Type Not Suitable

   From the above arguments, it is clear that haptics does not really
   belong under any other media type.  To reiterate, there are three
   main reasons why the 'haptics' media type does not fit under the
   'application' top-level type:

   *  haptics connects to a sensory system, touch/motion, directly, and
      is more specific than the abstract 'application' type, and

   *  'application' has historically been used for applications, i.e.,
      code, which means it is viewed and treated with great care for
      security.  'haptics' is not code, just as 'audio' and 'video' are
      not code either.

   *  haptics is a property of a media stream, it is not an application
      under any normal definition.  As such, it should be its own type.

3.  Security Considerations

   Haptics are interpreted data structures that represent collections of
   different media rendering instructions intended to be decoded and
   rendered on target device hardware.  Haptic data can be represented
   as collections of signal data and/or descriptive text in XML/JSON or
   a similar format.  Signal data is typically not executed by endpoint
   processors and represents minimal security risk.  Descriptive text is
   typically parsed and represented in memory using standard XML data
   structures.  This data is utilized to construct one or more signals
   that are sent to the endpoint device hardware.

   Because of the media/rendering nature of the data path for haptic-
   coded data, the security profile of haptic data is expected to be
   largely consistent with the security profile of visual and audio
   media data.

   As with any synthesized media data (audio, video, and haptics), there
   is a security risk associated with execution of commands based on the
   descriptive encoding either through its inherent extensibility or
   through the insertion of arbitrary executable data in the descriptive
   format itself.  Indeed, media rendering systems are normally
   implemented with a mix of user and kernel space execution since these
   media must ultimately make their way to a hardware system.  In
   theory, malicious instructions present in descriptive haptic media
   have the potential to execute arbitrary code in kernel space,
   effectively bypassing system permissions structures and/or execution
   sandboxes.

   Haptics, audio, and video media have widespread use and careful
   attention should be paid by operating system and device driver
   implementors to ensure that synthesis and rendering signal paths do
   not provide attack surfaces for malicious payloads.

   Thermal haptic devices (that provide a sensation of heat) and
   kinesthetic haptic devices (that provide force feedback) could
   potentially injure users if the heat or force, respectively, are not
   properly controlled or inadvertently exceed safety levels.
   Implementors need to ensure that adequate measures are taken to
   prevent such scenarios.

   These security considerations apply to the subtype registrations
   described in this document as well as all future haptics
   registrations.

4.  IANA Considerations

   IANA has registered 'haptics' in the "Top-Level Media Types" registry
   defined in [RFC9694] and registered several subtypes.  IANA has also
   added 'haptics' as an alternative value of "Type Name" in the media
   types registration form [Media-Type-Reg].

4.1.  Definition and Encoding

   'haptics' is the primary media content type that indicates the
   content identified by it requires a certain haptics subsystem such as
   low-level haptics APIs, which in turn will require hardware
   capabilities such as one or more actuators to render the haptics
   media.  The 'haptics' media type does not provide any specific
   information about the underlying data format and how the haptics
   information should be interpreted -- the subtypes defined within a
   'haptics' tree name the specific haptic formats.  Unrecognized
   subtypes of 'haptics' should be treated as 'application/octet-
   stream'.  Implementations may still pass unrecognized subtypes to the
   haptics subsystem and associated rendering hardware.

4.2.  Registration Procedure

   New haptics formats should be requested using the Application for a
   Media Type online form [Media-Type-Reg].  [RFC6838] should be
   consulted on registration procedures.  In particular, the haptics
   specification should preferably be freely available.

   Note that new subtypes may define parameters.  If an implementation
   does not recognize a parameter sub-value in the comma-separated list,
   it should ignore the sub-value and continue processing the other sub-
   values in the list.

4.3.  Subtype Registrations

   In this section, the initial entries under the top-level 'haptics'
   media type are specified.  They also serve as examples for future
   registrations.

4.3.1.  IVS Haptics Type

   Type name:  haptics

   Subtype name:  ivs

   Required parameters:  N/A

   Optional parameters:  N/A

   Encoding considerations:  8bit if UTF-8; binary if UTF-16 or UTF-32

   Interoperability considerations:  The IVS format is a device-
      independent haptic effect coding based on the XML format.  It is
      designed to enable interoperability between distinct physical
      endpoints.  Not all devices may be able to render all effects
      present in an IVS file.

   Security considerations:  See Section 3 of RFC 9695.

   Published specification:  ISO/IEC JTC 1/SC 29/WG 2 N0072 "Encoder
      Input Format for MPEG Haptics" [MPEG-Haptics-Encoder].

   Applications that use this media type:  All applications that are
      able to create, edit, or display haptic media content.

   Additional information:
      File extension(s):  Haptic file extensions used for IVS files:
         .ivs
      Macintosh file type code(s):  (no code specified)
      Macintosh Universal Type Identifier code:  N/A
      Fragment Identifier:  N/A
      Deprecated Alias:  N/A

   Person & email address to contact for further information:
      Yeshwant Muthusamy (yeshwant@yeshvik.com)

   Change controller:  ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and
      Haptic Coding)

4.3.2.  HJIF Haptics Type

   Type name:  haptics

   Subtype name:  hjif

   Required parameters:  N/A

   Optional parameters:  N/A

   Encoding considerations:  8bit if UTF-8; binary if UTF-16 or UTF-32

   Interoperability considerations:  The HJIF format is a human-readable
      haptic effect coding based on the JSON format.  It is designed as
      an interchange format for temporal and spatial haptic effects.
      The haptic effects may target specific parts of the human body and
      may be associated with a reference device description allowing
      haptic rendering software to adapt the effects to available
      hardware.

   Security considerations:  See Section 3 of RFC 9695.

   Published specification:  ISO/IEC 23090-31: Haptics coding
      [MPEG-Haptics-Coding].

   Applications that use this media type:  All applications that are
      able to create, edit, or display haptic media content.

   Additional information:
      File extension(s):  Haptic file extensions used for HJIF files:
         .hjif
      Macintosh file type code(s):  (no code specified)
      Macintosh Universal Type Identifier code:  N/A
      Fragment Identifier:  N/A
      Deprecated Alias:  N/A

   Person & email address to contact for further information:
      Yeshwant Muthusamy (yeshwant@yeshvik.com)

   Change controller:  ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and
      Haptic Coding)

4.3.3.  HMPG Haptics Type

   Type name:  haptics

   Subtype name:  hmpg

   Required parameters:  N/A

   Optional parameters:  N/A

   Encoding considerations:  binary

   Interoperability considerations:  The HMPG format is a streamable
      binary haptic effect coding.  It is designed to enable efficient
      coding of temporal and spatial haptic effects.  The haptic effects
      may target specific parts of the human body and may be associated
      with a reference device description allowing haptic rendering
      software to adapt the effects to available hardware.

   Security considerations:  See Section 3 of RFC 9695.

   Published specification:  ISO/IEC 23090-31: Haptics coding
      [MPEG-Haptics-Coding].

   Applications that use this media type:  All applications that are
      able to create, edit, or display haptic media content.

   Additional information:
      File extension(s):  Haptic file extensions used for HMPG files:
         .hmpg
      Macintosh file type code(s):  (no code specified)
      Macintosh Universal Type Identifier code:  N/A
      Fragment Identifier:  N/A
      Deprecated Alias:  N/A

   Person & email address to contact for further information:
      Yeshwant Muthusamy (yeshwant@yeshvik.com)

   Change controller:  ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and
      Haptic Coding)

5.  Normative References

   [RFC6838]  Freed, N., Klensin, J., and T. Hansen, "Media Type
              Specifications and Registration Procedures", BCP 13,
              RFC 6838, DOI 10.17487/RFC6838, January 2013,
              <https://www.rfc-editor.org/info/rfc6838>.

   [RFC9694]  Dürst, M.J., "Guidelines for the Definition of New Top-
              Level Media Types", BCP 13, RFC 9694,
              DOI 10.17487/RFC9694, March 2025,
              <https://www.rfc-editor.org/info/rfc9694>.

6.  Informative References

   [AHAP]     Apple Inc., "Representing haptic patterns in AHAP files",
              Apple Developer Documentation,
              <https://developer.apple.com/documentation/corehaptics/
              representing_haptic_patterns_in_ahap_files>.

   [Android-Sales]
              Counterpoint, "Global Smartphone Sales Share by Operating
              System", <https://www.counterpointresearch.com/insights/
              global-smartphone-os-market-share/>.

   [GameConsole-Sales]
              D'Angelo, W., "PS5 vs Xbox Series X|S vs Switch 2023
              Worldwide Sales Comparison Charts Through December -
              Sales", VGChartz.com,
              <https://www.vgchartz.com/article/459672/ps5-vs-xbox-
              series-xs-vs-switch-2023-worldwide-sales-comparison-
              charts-through-december/>.

   [GlobalSmartPhone-Sales]
              sellcell, "How many mobile phones are sold each year?",
              <https://www.sellcell.com/how-many-mobile-phones-are-sold-
              each-year/>.

   [IEEE-191811]
              IEEE, "IEEE Standard for Haptic Codecs for the Tactile
              Internet", IEEE Std 1918.1.1-2024,
              DOI 10.1109/IEEESTD.2024.10555007, June 2024,
              <https://ieeexplore.ieee.org/document/10555007>.

   [iPhone-Sales]
              Backlinko, "iPhone Users and Sales Stats",
              <https://backlinko.com/iphone-users>.

   [ISOBMFF-IS]
              ISO/IEC, "Information technology - Coding of audio-visual
              objects - Part 12: ISO base media file format", 7th
              Edition, ISO/IEC 14496-12:2022, January 2022,
              <https://www.iso.org/standard/83102.html>.

   [Media-Type-Reg]
              IANA, "Application for a Media Type",
              <https://www.iana.org/form/media-types>.

   [MPEG-Haptics-Coding]
              ISO/IEC, "Information Technology -- Coded representation
              of immersive media -- Part 31: Haptics coding", ISO/
              IEC 23090-31:2025, January 2025,
              <https://www.iso.org/standard/86122.html>.

   [MPEG-Haptics-Encoder]
              MPEG, "Encoder Input Format for Haptics", MPEG 134 Meeting
              Document, 15 May 2021,
              <https://www.mpegstandards.org/standards/
              Explorations/40/>.

   [W3C-Gamepad]
              W3C, "Gamepad Extensions", W3C Editor's Draft,
              <https://w3c.github.io/gamepad/extensions.html>.

   [W3C-Vibration]
              W3C, "Vibration API (Second Edition)", W3C Recommendation,
              18 October 2016,
              <https://www.w3.org/TR/2016/REC-vibration-20161018/>.
              Latest version available at
              <https://www.w3.org/TR/vibration/>

   [XRDevice-Sales]
              Binarybox Studios, "Virtual reality market size 2023",
              LinkedIn, <https://www.linkedin.com/pulse/virtual-reality-
              market-size-2023-binarybox-studios-dfbif/>.

Authors' Addresses

   Yeshwant K. Muthusamy
   600 Longwood Drive
   Allen, TX 75013
   United States of America
   Phone: +1 469-854-9836
   Email: yeshwant@yeshvik.com


   Chris Ullrich
   311 Court Ave
   Ventura, CA 93003
   United States of America
   Phone: +1 805-320-0774
   Email: chrisullrich@gmail.com