Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • S ssg
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 18
    • Issues 18
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • sds
  • ssg
  • Issues
  • #22

Closed
Open
Created Aug 12, 2020 by Rob Latham@roblOwner

too many observers can crash ssg

If one starts up a lot of ssg members on one node like this:

mpiexec -np 32 ./tests/ssg-launch-group -s 360 -f group.ssg sockets mpi &

and tries to observe that group with a small number of processes, things are ok:

./ssg-observe-group sockets build/group.ssg

If I try to observe with 64 processes, I get some errors:

SWIM dping ack recv error -- group 15324806640328145610 not found
SWIM dping req recv error -- group 15324806640328145610 not found
# HG -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-2.0.0rc1-qeng5ccan7pe4mgpwopu4cpaw6ftfcbz/spack-src/src/mercury_core.c:3748
 # HG_Core_registered_data(): Could not find RPC ID in function map
# HG -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-2.0.0rc1-qeng5ccan7pe4mgpwopu4cpaw6ftfcbz/spack-src/src/mercury.c:1368
 # HG_Registered_data(): Could not get registered data

On slack @ssnyder mentioned the 'recv error' messages are spam but they sure seem to indicate something bad is about to happen.

Assignee
Assign to
Time tracking