Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
D
darshan
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
74
Issues
74
List
Boards
Labels
Milestones
Merge Requests
9
Merge Requests
9
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
darshan
darshan
Commits
0448ed3c
Commit
0448ed3c
authored
Mar 25, 2015
by
Shane Snyder
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
code cleanup + posix read/write wrappers
parent
165a27fc
Changes
9
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
9 changed files
with
533 additions
and
158 deletions
+533
-158
darshan-posix-log-format.h
darshan-posix-log-format.h
+100
-91
darshan-runtime/configure
darshan-runtime/configure
+2
-2
darshan-runtime/configure.in
darshan-runtime/configure.in
+2
-2
darshan-runtime/darshan-config.in
darshan-runtime/darshan-config.in
+0
-1
darshan-runtime/lib/darshan-core.c
darshan-runtime/lib/darshan-core.c
+2
-2
darshan-runtime/lib/darshan-posix.c
darshan-runtime/lib/darshan-posix.c
+385
-41
darshan-util/darshan-posix-logutils.c
darshan-util/darshan-posix-logutils.c
+2
-2
darshan-util/darshan-posix-parser.c
darshan-util/darshan-posix-parser.c
+40
-4
doc/darshan-modularization-design-notes.txt
doc/darshan-modularization-design-notes.txt
+0
-13
No files found.
darshan-posix-log-format.h
View file @
0448ed3c
...
...
@@ -8,121 +8,130 @@
#include "darshan-log-format.h"
/* TODO: DARSHAN_* OR CP_* */
enum
darshan_posix_indices
{
CP_POSIX_READS
,
/* count of posix reads */
CP_POSIX_WRITES
,
/* count of posix writes */
CP_POSIX_OPENS
,
/* count of posix opens */
CP_POSIX_SEEKS
,
/* count of posix seeks */
CP_POSIX_STATS
,
/* count of posix stat/lstat/fstats */
CP_POSIX_MMAPS
,
/* count of posix mmaps */
CP_POSIX_FREADS
,
CP_POSIX_FWRITES
,
CP_POSIX_FOPENS
,
CP_POSIX_FSEEKS
,
CP_POSIX_FSYNCS
,
CP_POSIX_FDSYNCS
,
CP_MODE
,
/* mode of file */
CP_BYTES_READ
,
/* total bytes read */
CP_BYTES_WRITTEN
,
/* total bytes written */
CP_MAX_BYTE_READ
,
/* highest offset byte read */
CP_MAX_BYTE_WRITTEN
,
/* highest offset byte written */
CP_CONSEC_READS
,
/* count of consecutive reads */
CP_CONSEC_WRITES
,
/* count of consecutive writes */
CP_SEQ_READS
,
/* count of sequential reads */
CP_SEQ_WRITES
,
/* count of sequential writes */
CP_RW_SWITCHES
,
/* number of times switched between read and write */
CP_MEM_NOT_ALIGNED
,
/* count of accesses not mem aligned */
CP_MEM_ALIGNMENT
,
/* mem alignment in bytes */
CP_FILE_NOT_ALIGNED
,
/* count of accesses not file aligned */
CP_FILE_ALIGNMENT
,
/* file alignment in bytes */
CP_MAX_READ_TIME_SIZE
,
CP_MAX_WRITE_TIME_SIZE
,
POSIX_OPENS
,
/* count of posix opens */
POSIX_READS
,
/* count of posix reads */
POSIX_WRITES
,
/* count of posix writes */
#if 0
POSIX_SEEKS, /* count of posix seeks */
POSIX_STATS, /* count of posix stat/lstat/fstats */
POSIX_MMAPS, /* count of posix mmaps */
#endif
POSIX_FOPENS
,
POSIX_FREADS
,
POSIX_FWRITES
,
#if 0
POSIX_FSEEKS,
POSIX_FSYNCS,
POSIX_FDSYNCS,
#endif
POSIX_MODE
,
/* mode of file */
#if 0
POSIX_BYTES_READ, /* total bytes read */
POSIX_BYTES_WRITTEN, /* total bytes written */
POSIX_MAX_BYTE_READ, /* highest offset byte read */
POSIX_MAX_BYTE_WRITTEN, /* highest offset byte written */
CONSEC_READS, /* count of consecutive reads */
CONSEC_WRITES, /* count of consecutive writes */
SEQ_READS, /* count of sequential reads */
SEQ_WRITES, /* count of sequential writes */
RW_SWITCHES, /* number of times switched between read and write */
MEM_NOT_ALIGNED, /* count of accesses not mem aligned */
MEM_ALIGNMENT, /* mem alignment in bytes */
FILE_NOT_ALIGNED, /* count of accesses not file aligned */
FILE_ALIGNMENT, /* file alignment in bytes */
#endif
POSIX_MAX_READ_TIME_SIZE
,
POSIX_MAX_WRITE_TIME_SIZE
,
#if 0
/* buckets */
CP_
SIZE_READ_0_100
,
/* count of posix read size ranges */
CP_
SIZE_READ_100_1K
,
CP_
SIZE_READ_1K_10K
,
CP_
SIZE_READ_10K_100K
,
CP_
SIZE_READ_100K_1M
,
CP_
SIZE_READ_1M_4M
,
CP_
SIZE_READ_4M_10M
,
CP_
SIZE_READ_10M_100M
,
CP_
SIZE_READ_100M_1G
,
CP_
SIZE_READ_1G_PLUS
,
SIZE_READ_0_100, /* count of posix read size ranges */
SIZE_READ_100_1K,
SIZE_READ_1K_10K,
SIZE_READ_10K_100K,
SIZE_READ_100K_1M,
SIZE_READ_1M_4M,
SIZE_READ_4M_10M,
SIZE_READ_10M_100M,
SIZE_READ_100M_1G,
SIZE_READ_1G_PLUS,
/* buckets */
CP_
SIZE_WRITE_0_100
,
/* count of posix write size ranges */
CP_
SIZE_WRITE_100_1K
,
CP_
SIZE_WRITE_1K_10K
,
CP_
SIZE_WRITE_10K_100K
,
CP_
SIZE_WRITE_100K_1M
,
CP_
SIZE_WRITE_1M_4M
,
CP_
SIZE_WRITE_4M_10M
,
CP_
SIZE_WRITE_10M_100M
,
CP_
SIZE_WRITE_100M_1G
,
CP_
SIZE_WRITE_1G_PLUS
,
SIZE_WRITE_0_100, /* count of posix write size ranges */
SIZE_WRITE_100_1K,
SIZE_WRITE_1K_10K,
SIZE_WRITE_10K_100K,
SIZE_WRITE_100K_1M,
SIZE_WRITE_1M_4M,
SIZE_WRITE_4M_10M,
SIZE_WRITE_10M_100M,
SIZE_WRITE_100M_1G,
SIZE_WRITE_1G_PLUS,
/* counters */
CP_STRIDE1_STRIDE
,
/* the four most frequently appearing strides */
CP_STRIDE2_STRIDE
,
CP_STRIDE3_STRIDE
,
CP_STRIDE4_STRIDE
,
CP_STRIDE1_COUNT
,
/* count of each of the most frequent strides */
CP_STRIDE2_COUNT
,
CP_STRIDE3_COUNT
,
CP_STRIDE4_COUNT
,
CP_ACCESS1_ACCESS
,
/* the four most frequently appearing access sizes */
CP_ACCESS2_ACCESS
,
CP_ACCESS3_ACCESS
,
CP_ACCESS4_ACCESS
,
CP_ACCESS1_COUNT
,
/* count of each of the most frequent access sizes */
CP_ACCESS2_COUNT
,
CP_ACCESS3_COUNT
,
CP_ACCESS4_COUNT
,
CP_DEVICE
,
/* device id reported by stat */
CP_SIZE_AT_OPEN
,
CP_FASTEST_RANK
,
CP_FASTEST_RANK_BYTES
,
CP_SLOWEST_RANK
,
CP_SLOWEST_RANK_BYTES
,
STRIDE1_STRIDE, /* the four most frequently appearing strides */
STRIDE2_STRIDE,
STRIDE3_STRIDE,
STRIDE4_STRIDE,
STRIDE1_COUNT, /* count of each of the most frequent strides */
STRIDE2_COUNT,
STRIDE3_COUNT,
STRIDE4_COUNT,
ACCESS1_ACCESS, /* the four most frequently appearing access sizes */
ACCESS2_ACCESS,
ACCESS3_ACCESS,
ACCESS4_ACCESS,
ACCESS1_COUNT, /* count of each of the most frequent access sizes */
ACCESS2_COUNT,
ACCESS3_COUNT,
ACCESS4_COUNT,
DEVICE, /* device id reported by stat */
SIZE_AT_OPEN,
FASTEST_RANK,
FASTEST_RANK_BYTES,
SLOWEST_RANK,
SLOWEST_RANK_BYTES,
#endif
CP
_NUM_INDICES
,
POSIX
_NUM_INDICES
,
};
/* floating point statistics */
enum
darshan_
f_posix
_indices
enum
darshan_
posix_f
_indices
{
/* NOTE: adjust cp_normalize_timestamps() function if any TIMESTAMPS are
* added or modified in this list
*/
CP_F_OPEN_TIMESTAMP
=
0
,
/* timestamp of first open */
CP_F_READ_START_TIMESTAMP
,
/* timestamp of first read */
CP_F_WRITE_START_TIMESTAMP
,
/* timestamp of first write */
CP_F_CLOSE_TIMESTAMP
,
/* timestamp of last close */
CP_F_READ_END_TIMESTAMP
,
/* timestamp of last read */
CP_F_WRITE_END_TIMESTAMP
,
/* timestamp of last write */
CP_F_POSIX_READ_TIME
,
/* cumulative posix read time */
CP_F_POSIX_WRITE_TIME
,
/* cumulative posix write time */
CP_F_POSIX_META_TIME
,
/* cumulative posix meta time */
CP_F_MAX_READ_TIME
,
CP_F_MAX_WRITE_TIME
,
POSIX_F_OPEN_TIMESTAMP
=
0
,
/* timestamp of first open */
POSIX_F_READ_START_TIMESTAMP
,
/* timestamp of first read */
POSIX_F_WRITE_START_TIMESTAMP
,
/* timestamp of first write */
POSIX_F_READ_END_TIMESTAMP
,
/* timestamp of last read */
POSIX_F_WRITE_END_TIMESTAMP
,
/* timestamp of last write */
POSIX_F_CLOSE_TIMESTAMP
,
/* timestamp of last close */
POSIX_F_READ_TIME
,
/* cumulative posix read time */
POSIX_F_WRITE_TIME
,
/* cumulative posix write time */
POSIX_F_META_TIME
,
/* cumulative posix meta time */
POSIX_F_MAX_READ_TIME
,
POSIX_F_MAX_WRITE_TIME
,
#if 0
/* Total I/O and meta time consumed by fastest and slowest ranks,
* reported in either MPI or POSIX time depending on how the file
* was accessed.
*/
CP_F_FASTEST_RANK_TIME
,
CP_F_SLOWEST_RANK_TIME
,
CP_F_VARIANCE_RANK_TIME
,
CP_F_VARIANCE_RANK_BYTES
,
F_FASTEST_RANK_TIME,
F_SLOWEST_RANK_TIME,
F_VARIANCE_RANK_TIME,
F_VARIANCE_RANK_BYTES,
#endif
CP
_F_NUM_INDICES
,
POSIX
_F_NUM_INDICES
,
};
struct
darshan_posix_file
{
darshan_record_id
f_id
;
int64_t
rank
;
int64_t
counters
[
CP
_NUM_INDICES
];
double
fcounters
[
CP
_F_NUM_INDICES
];
int64_t
counters
[
POSIX
_NUM_INDICES
];
double
fcounters
[
POSIX
_F_NUM_INDICES
];
};
#endif
/* __DARSHAN_POSIX_LOG_FORMAT_H */
darshan-runtime/configure
View file @
0448ed3c
...
...
@@ -4079,10 +4079,10 @@ fi
done
# libc functions wrapped by darshan
#CP_WRAPPERS="-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,write,-wrap,open,-wrap,creat,-wrap,creat64,-wrap,open64,-wrap,close,-wrap,read,-wrap,lseek,-wrap,lseek64,-wrap,pread,-wrap,pwrite,-wrap,readv,-wrap,writev,-wrap,__xstat,-wrap,__lxstat,-wrap,__fxstat,-wrap,__xstat64,-wrap,__lxstat64,-wrap,__fxstat64,-wrap,mmap,-wrap,mmap64,-wrap,fopen,-wrap,fclose,-wrap,fread,-wrap,fwrite,-wrap,fseek,-wrap,fopen64,-wrap,pread64,-wrap,pwrite64,-wrap,fsync,-wrap,fdatasync,-wrap,ncmpi_create,-wrap,ncmpi_open,-wrap,ncmpi_close,-wrap,H5Fcreate,-wrap,H5Fopen,-wrap,H5Fclose,-wrap,aio_write,-wrap,aio_write64,-wrap,aio_read,-wrap,aio_read64,-wrap,lio_listio,-wrap,lio_listio64,-wrap,aio_return,-wrap,aio_return64,-wrap,mkstemp,-wrap,mkostemp,-wrap,mkstemps,-wrap,mkostemps"
CP_WRAPPERS
=
"-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,open,-wrap,open64,-wrap,close"
# libc functions wrapped by darshan
CP_WRAPPERS
=
"-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,open,-wrap,open64,-wrap,creat,-wrap,creat64,-wrap,read,-wrap,write,-wrap,pread,-wrap,pwrite,-wrap,pread64,-wrap,pwrite64,-wrap,readv,-wrap,writev,-wrap,close"
# We need to know the value of the $libdir and $bindir variables so that
# we can reference the correct path in the darshan compiler wrappers.
...
...
darshan-runtime/configure.in
View file @
0448ed3c
...
...
@@ -187,10 +187,10 @@ CFLAGS="$old_cflags"
AC_CHECK_HEADERS(mntent.h sys/mount.h)
# libc functions wrapped by darshan
#CP_WRAPPERS="-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,write,-wrap,open,-wrap,creat,-wrap,creat64,-wrap,open64,-wrap,close,-wrap,read,-wrap,lseek,-wrap,lseek64,-wrap,pread,-wrap,pwrite,-wrap,readv,-wrap,writev,-wrap,__xstat,-wrap,__lxstat,-wrap,__fxstat,-wrap,__xstat64,-wrap,__lxstat64,-wrap,__fxstat64,-wrap,mmap,-wrap,mmap64,-wrap,fopen,-wrap,fclose,-wrap,fread,-wrap,fwrite,-wrap,fseek,-wrap,fopen64,-wrap,pread64,-wrap,pwrite64,-wrap,fsync,-wrap,fdatasync,-wrap,ncmpi_create,-wrap,ncmpi_open,-wrap,ncmpi_close,-wrap,H5Fcreate,-wrap,H5Fopen,-wrap,H5Fclose,-wrap,aio_write,-wrap,aio_write64,-wrap,aio_read,-wrap,aio_read64,-wrap,lio_listio,-wrap,lio_listio64,-wrap,aio_return,-wrap,aio_return64,-wrap,mkstemp,-wrap,mkostemp,-wrap,mkstemps,-wrap,mkostemps"
CP_WRAPPERS="-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,open,-wrap,open64,-wrap,close"
# libc functions wrapped by darshan
CP_WRAPPERS="-Wl,-u,MPI_Init,-u,MPI_Wtime,-wrap,open,-wrap,open64,-wrap,creat,-wrap,creat64,-wrap,read,-wrap,write,-wrap,pread,-wrap,pwrite,-wrap,pread64,-wrap,pwrite64,-wrap,readv,-wrap,writev,-wrap,close"
# We need to know the value of the $libdir and $bindir variables so that
# we can reference the correct path in the darshan compiler wrappers.
...
...
darshan-runtime/darshan-config.in
View file @
0448ed3c
...
...
@@ -11,7 +11,6 @@ DARSHAN_LD_FLAGS="@LDFLAGS@"
# dependencies on PnetCDF and HDF5 symbols (if the app used a library which
# in turn used one of those HLLs).
# TODO: extra -lmpi in POST_LD_FLAGS -- link ordering issue
PRE_LD_FLAGS
=
"-L
$DARSHAN_LIB_PATH
$DARSHAN_LD_FLAGS
-ldarshan -lz
$CP_WRAPPERS
"
POST_LD_FLAGS
=
"-L
$DARSHAN_LIB_PATH
-ldarshan -lz -lrt -lpthread"
...
...
darshan-runtime/lib/darshan-core.c
View file @
0448ed3c
...
...
@@ -27,8 +27,8 @@
#include "uthash.h"
#include "darshan-core.h"
/* TODO is __progname_full needed here */
extern
char
*
__progname
;
extern
char
*
__progname_full
;
/* internal variable delcarations */
static
struct
darshan_core_runtime
*
darshan_core
=
NULL
;
...
...
@@ -193,7 +193,7 @@ static void darshan_core_initialize(int argc, char **argv)
if
(
argc
==
0
)
{
chars_left
=
DARSHAN_EXE_LEN
-
strlen
(
darshan_core
->
exe
);
strncat
(
darshan_core
->
exe
,
__progname
,
chars_left
);
strncat
(
darshan_core
->
exe
,
__progname
_full
,
chars_left
);
chars_left
=
DARSHAN_EXE_LEN
-
strlen
(
darshan_core
->
exe
);
strncat
(
darshan_core
->
exe
,
" <unknown args>"
,
chars_left
);
}
...
...
darshan-runtime/lib/darshan-posix.c
View file @
0448ed3c
This diff is collapsed.
Click to expand it.
darshan-util/darshan-posix-logutils.c
View file @
0448ed3c
...
...
@@ -37,9 +37,9 @@ int darshan_log_get_posix_file(darshan_fd fd, struct darshan_posix_file *file)
/* swap bytes if necessary */
DARSHAN_BSWAP64
(
&
file
->
f_id
);
DARSHAN_BSWAP64
(
&
file
->
rank
);
for
(
i
=
0
;
i
<
CP
_NUM_INDICES
;
i
++
)
for
(
i
=
0
;
i
<
POSIX
_NUM_INDICES
;
i
++
)
DARSHAN_BSWAP64
(
&
file
->
counters
[
i
]);
for
(
i
=
0
;
i
<
CP
_F_NUM_INDICES
;
i
++
)
for
(
i
=
0
;
i
<
POSIX
_F_NUM_INDICES
;
i
++
)
DARSHAN_BSWAP64
(
&
file
->
fcounters
[
i
]);
}
}
...
...
darshan-util/darshan-posix-parser.c
View file @
0448ed3c
...
...
@@ -153,7 +153,7 @@ int main(int argc, char **argv)
}
if
(
ret
==
0
)
{
printf
(
"# no files opened."
);
printf
(
"# no files opened.
\n
"
);
darshan_log_close
(
fd
);
return
(
0
);
}
...
...
@@ -168,9 +168,45 @@ int main(int argc, char **argv)
printf
(
"
\t
Record %d: id=%"
PRIu64
" (path=%s, rank=%"
PRId64
")
\n
"
,
i
,
next_file
.
f_id
,
ref
->
rec
.
name
,
next_file
.
rank
);
printf
(
"
\t\t
POSIX_OPENS:
\t
%"
PRIu64
"
\n\t\t
F_OPEN_TIMESTAMP:
\t
%lf
\n\t\t
F_CLOSE_TIMESTAMP:
\t
%lf
\n
"
,
next_file
.
counters
[
CP_POSIX_OPENS
],
next_file
.
fcounters
[
CP_F_OPEN_TIMESTAMP
],
next_file
.
fcounters
[
CP_F_CLOSE_TIMESTAMP
]);
printf
(
"
\t\t
POSIX_OPENS:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_READS:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_WRITES:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_FOPENS:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_FREADS:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_FWRITES:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_MODE:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_MAX_READ_TIME_SIZE:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_MAX_WRITE_TIME_SIZE:
\t
%"
PRIu64
"
\n
"
"
\t\t
POSIX_F_OPEN_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_READ_START_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_WRITE_START_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_READ_END_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_WRITE_END_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_CLOSE_TIMESTAMP:
\t
%lf
\n
"
"
\t\t
POSIX_F_READ_TIME:
\t
%lf
\n
"
"
\t\t
POSIX_F_WRITE_TIME:
\t
%lf
\n
"
"
\t\t
POSIX_F_MAX_READ_TIME:
\t
%lf
\n
"
"
\t\t
POSIX_F_MAX_WRITE_TIME:
\t
%lf
\n
"
,
next_file
.
counters
[
POSIX_OPENS
],
next_file
.
counters
[
POSIX_READS
],
next_file
.
counters
[
POSIX_WRITES
],
next_file
.
counters
[
POSIX_FOPENS
],
next_file
.
counters
[
POSIX_FREADS
],
next_file
.
counters
[
POSIX_FWRITES
],
next_file
.
counters
[
POSIX_MODE
],
next_file
.
counters
[
POSIX_MAX_READ_TIME_SIZE
],
next_file
.
counters
[
POSIX_MAX_WRITE_TIME_SIZE
],
next_file
.
fcounters
[
POSIX_F_OPEN_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_READ_START_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_WRITE_START_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_READ_END_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_WRITE_END_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_CLOSE_TIMESTAMP
],
next_file
.
fcounters
[
POSIX_F_READ_TIME
],
next_file
.
fcounters
[
POSIX_F_WRITE_TIME
],
next_file
.
fcounters
[
POSIX_F_MAX_READ_TIME
],
next_file
.
fcounters
[
POSIX_F_MAX_WRITE_TIME
]);
i
++
;
}
while
((
ret
=
darshan_log_get_posix_file
(
fd
,
&
next_file
))
==
1
);
...
...
doc/darshan-modularization-design-notes.txt
View file @
0448ed3c
...
...
@@ -110,16 +110,3 @@ API:
- strip down to basic example
- just do one or two posix counters to start, but exercise all of the
API and code organization stuff
TODO NOTES:
- why are darshan mutexes recursive?
- how do we allow modules to specify there necessary linker/wrapper flags
- configurable amount of max runtime memory
- fs mount information -- should we export this info from darshan-core?
- would the posix module be the only one to leverage this info?
- it is used to correlate filenames with mount points, but not sure what is needed where
- test fortran and c apps to make sure command line args are handled properly (see email with Phil 11/4)
- when is DARSHAN_MPI_CALL macro needed? All MPI functions, or just those intercepted by darshan?
- darshan link order causes issues when compiling mpi-io-test -- had to add extra -lmpi to resolve
- is there more coordination between core and modules for shared file reductions?
- do we change darshan file extension (.dz? .darshan?)
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment