Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • D darshan
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 72
    • Issues 72
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • darshan
  • darshan
  • Issues
  • #232

Closed
Open
Created Aug 08, 2017 by Glenn K. Lockwood@glockDeveloper

add support for DataWarp accounting API

Recent versions of Cray Linux include a DataWarp telemetry API that provides information analogous to what the Lustre module provides. Specifically, dw_get_stripe_configuration(3) returns

  • int stripe_size
  • int stripe_width
  • int starting_index

And the DataWarp servers to which starting_index refers can be divined from /proc/mounts or other kernel interfaces.

Furthermore, CLE 6.0UP04 now includes dw_get_accounting_data_json(3) which returns aggregate server-side telemetry. While not broken out into a per-file basis, the data types returned may be of interest at some point. An example output is:

{
  "realm": {
    "namespaces": [
      {
        "stage_bytes_written": 0,
        "namespace_id": 378,
        "stripe_width": 2,
        "file_size_limit": 0,
        "stage_bytes_read": 0,
        "stripe_size": 8388608,
        "files_create_threshold": 0,
        "substripe_width": 12,
        "substripe_size": 8388608,
        "files_created": 0,
        "num_data_created": 0,
        "bytes_read": 0,
        "max_offset_written": 0,
        "bytes_written": 0,
        "max_offset_read": 0
      }
    ],
    "fragments": [
      {
        "fs_capacity": 0,
        "files_created": 0,
        "capacity_used": 0,
        "capacity_max": 0,
        "max_window_write": 86400,
        "write_high_water": 0,
        "server_hostname": "c0-0c0s6n2",
        "write_moving_avg": 0,
        "write_limit": 1730066513920,
        "bytes_written": 0,
        "bytes_read": 0
      },
      {
        "fs_capacity": 172920012800,
        "files_created": 0,
        "capacity_used": 34369536,
        "capacity_max": 34369536,
        "max_window_write": 86400,
        "write_high_water": 0,
        "server_hostname": "c0-0c0s6n1",
        "write_moving_avg": 0,
        "write_limit": 1730066513920,
        "bytes_written": 0,
        "bytes_read": 0
      }
    ],
    "realm_id": 378,
    "namespace_count": 1,
    "server_count": 2
  }
}

These sorts of data may be best collected by some other tool (just as we decided client-side file system counters should be).

Assignee
Assign to
Time tracking