Commit 5f0b9cbc authored by Michael Salim's avatar Michael Salim

SQLite insert benchmarks; TIMEOUT set ridiculously high!

parent 7dc5227f
......@@ -14,3 +14,4 @@ docs/_static/*
default_balsamdb
testdb
nohup.out
......@@ -8,8 +8,10 @@ import zmq
from django.db.utils import OperationalError
from concurrency.exceptions import RecordModifiedError
REQ_TIMEOUT = 30000 # 30 seconds
REQ_RETRY = 4
# These are ridiculously high to benchmark
# Should be more like 5-10 sec, 3-4 retry
REQ_TIMEOUT = 60000 # 60 seconds
REQ_RETRY = 56
class Client:
......
......@@ -20,7 +20,7 @@ from concurrency.exceptions import RecordModifiedError
logger = logging.getLogger('balsam.django_config.sqlite_server')
SERVER_PERIOD = 1000
SERVER_PERIOD = 2000
TERM_LINGER = 3 # wait 3 sec after final save() to exit
terminate = False
......
# BENCHMARK: test_concurrent_mpi_insert
# Host: thetamom1
# COBALT_PARTCORES: 64
# COBALT_ENDTIME: 1517592363
# COBALT_QUEUE: debug-cache-quad
# COBALT_PARTNAME: 3828,3835-3837
# COBALT_STARTTIME: 1517591763
# COBALT_JOBID: 182446
# COBALT_PARTSIZE: 4
# COBALT_JOBSIZE: 4
# COBALT_PROJECT: datascience
# Each rank simultaneously calls dag.add_job (num_ranks simultaneous insertions)
# num_nodes ranks_per_node num_ranks total_time_sec
# --------------------------------------------------------------
1 32 32 29.490
2 32 64 33.620
4 32 128 35.900
# BENCHMARK: test_concurrent_mpi_insert
# Host: thetamom3
# COBALT_PARTCORES: 64
# COBALT_ENDTIME: 1517596142
# COBALT_QUEUE: default
# COBALT_PARTNAME: 70-75,310-319,630,664-665,667-669,770-779,860-869,880-889,1040-1049,1262,1264-1269,1310-1319,1330-1337,1460-1469,1684-1689,1910-1925,2020-2029,2113,2117,2324,2411,2460-2461,2692,2694,2698,2750,2752-2759,2908,2970-2977,3020-3021,3025,3027,3035,3095,3124-3129,3141,3201,3273,3276,3283,3285-3289,3341,3402-3405,3407,3442-3445,3447-3449,3491,3610-3611,3669,3722,3729,3948-3949,3970,3973,3976-3977,3983,3988,4014,4050,4052-4053,4055-4056,4059,4089,4120-4125,4127-4128,4210-4212,4214,4216-4218,4230-4239,4302-4303,4306,4313,4327,4359,4361,4367,4553,4555,4557,4587,4590-4592,4595-4597
# COBALT_STARTTIME: 1517593442
# COBALT_JOBID: 182461
# COBALT_PARTSIZE: 256
# COBALT_JOBSIZE: 256
# COBALT_PROJECT: datascience
# Each rank simultaneously calls dag.add_job (num_ranks simultaneous insertions)
# At 64 nodes/2048 ranks, we made it all the way to 2046 successful saves, very painfully (server crashed and restarted many times)
# Two saves got lost; hence failed at 2048 concurrent saves. Reaches peak of about 9 insert per second
# num_nodes ranks_per_node num_ranks total_time_sec
# --------------------------------------------------------------
1 32 32 29.020
2 32 64 28.930
4 32 128 35.160
8 32 256 43.350
16 32 512 64.740
32 32 1024 114.590
Jobid: 182461
qsub --mode script -A datascience -n 256 -q default -t 45 -M msalim@anl.gov run_bench_insert.sh
Fri Feb 02 17:43:35 2018 +0000 (UTC) submitted with cwd set to: /gpfs/mira-home/msalim/hpc-edge-service/tests/benchmarks/scripts
jobid 182461 submitted from terminal /dev/pts/31
2018-02-02 17:44:28,169 INFO: Starting Resource_Prologue
2018-02-02 17:44:28,171 DEBUG: no enabling attribute found
2018-02-02 17:44:28,172 INFO: Resource_Prologue finished with exit code 0
Fri Feb 02 17:44:47 2018 +0000 (UTC)
Fri Feb 02 17:44:47 2018 +0000 (UTC) Command: '/gpfs/mira-home/msalim/hpc-edge-service/tests/benchmarks/scripts/run_bench_insert.sh'
Fri Feb 02 17:44:47 2018 +0000 (UTC)
Fri Feb 02 17:44:47 2018 +0000 (UTC) Environment:
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_PARTCORES=64
Fri Feb 02 17:44:47 2018 +0000 (UTC) SHELL=/bin/bash
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_PROJECT=datascience
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_PARTNAME=70-75,310-319,630,664-665,667-669,770-779,860-869,880-889,1040-1049,1262,1264-1269,1310-1319,1330-1337,1460-1469,1684-1689,1910-1925,2020-2029,2113,2117,2324,2411,2460-2461,2692,2694,2698,2750,2752-2759,2908,2970-2977,3020-3021,3025,3027,3035,3095,3124-3129,3141,3201,3273,3276,3283,3285-3289,3341,3402-3405,3407,3442-3445,3447-3449,3491,3610-3611,3669,3722,3729,3948-3949,3970,3973,3976-3977,3983,3988,4014,4050,4052-4053,4055-4056,4059,4089,4120-4125,4127-4128,4210-4212,4214,4216-4218,4230-4239,4302-4303,4306,4313,4327,4359,4361,4367,4553,4555,4557,4587,4590-4592,4595-4597
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_JOBID=182461
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_STARTTIME=1517593442
Fri Feb 02 17:44:47 2018 +0000 (UTC) LOGNAME=msalim
Fri Feb 02 17:44:47 2018 +0000 (UTC) USER=msalim
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_ENDTIME=1517596142
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_PARTSIZE=256
Fri Feb 02 17:44:47 2018 +0000 (UTC) HOME=/home/msalim
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_JOBSIZE=256
Fri Feb 02 17:44:47 2018 +0000 (UTC) COBALT_QUEUE=default
Fri Feb 02 17:44:47 2018 +0000 (UTC)
Fri Feb 02 17:44:47 2018 +0000 (UTC) Info: stdin received from /dev/null
Fri Feb 02 17:44:47 2018 +0000 (UTC) Info: stdout sent to /gpfs/mira-home/msalim/hpc-edge-service/tests/benchmarks/scripts/182461.output
Fri Feb 02 17:44:47 2018 +0000 (UTC) Info: stderr sent to /gpfs/mira-home/msalim/hpc-edge-service/tests/benchmarks/scripts/182461.error
Fri Feb 02 17:44:47 2018 +0000 (UTC)
Fri Feb 02 18:00:04 2018 +0000 (UTC) Info: task completed normally with an exit code of 0; initiating job cleanup and removal
This diff is collapsed.
{"db_type": "sqlite3", "address": "tcp://thetalogin4:44379"}
\ No newline at end of file
#!/bin/bash -x
#COBALT -A datascience
#COBALT -n 4
#COBALT -q debug-cache-quad
#COBALT -t 10
#COBALT -n 256
#COBALT -q default
#COBALT -t 45
#COBALT -M msalim@anl.gov
#COBALT --cwd ~/hpc-edge-service
source ~/.bash_profile
source activate balsam
cat testdb/dbwriter_address
rm testdb/log/*.log
cat ~/hpc-edge-service/testdb/dbwriter_address
rm ~/hpc-edge-service/testdb/log/*.log
export BALSAM_DB_PATH=~/hpc-edge-service/testdb
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment