Needed for compilation on Summit, reported by Michael Brim

- allow multiple initialization attempts

This reverts commit c4ba141e.

Breaks BMI IB.

This reverts commit 1883ac0f.

One of these patches is causing connection setup to fail with a bad file
descriptor error.

The new coming connection are handled in the function
ib_tcp_server_check_new_connections(); and this is called inside the
function ib_block_for_activity(). However the ib_block_for_activity()
is only called when the network is idle in BMI_ib_testcontext() or
BMI_ib_testunexpected().

As a result, when the server is busy serving one client process, the
other processes can't make a new connections to the server and thus
they can't transfer data to the server concurrently.

Change-Id: Ib9c28e0e5c38cf03741c2220a9a95e2c9ffbe5a1

The correctness of BMI depends on the order of two events: the
completion event of the work request for the MSG_CTS message, the
arrival of the message with type MSG_RTS_DONE.

If the MSG_RTS_DONE messages arrives first, then there would be no
request in the state of RQ_RTS_WAITING_RTS_DONE, so it can't advance
the state machine. As a result, a log message is printed and the
message is lost.

Patch from Jingwang Zhang.

Here is our use case, and we are using the BMI code in OrangeFS 2.8.6:

1.       Malloc a buffer and write something to it.

2.       Send the buffer using BMI routines.

3.       Free the buffer and goto step 1

And here we found the following problem: We found that the messages
captured using ibdump is corrupt in iteration 2. It became a mixture of
data from iteration 1 and iteration 2.

Here is some analysis we did: We noticed that when the data corruption
occurs, the buffer always point to the same virtual address. And after
checking with the BMI code, we found that the memcache code will keep
the buffer registered(to ibverbs) and use virtual address to determine
whether a registered buffer could be reused or not later.

However I think the memcache shouldn't keep the buffer registered,
because that the user might free this buffer, and when the user did free
and re-allocate the buffer, there might be a false match which might
lead to data corruption.

So at first, we tested the code with "define ENABLE_MEMCACHE 0" to
disable the memcache. And then the test passed, so it is proven that the
data corruption is caused by memcache. However, performance will be
affected if the memcache is disabled completely.

Finally we formatted the attached patch to solve the problem. It fixes
the broken code in the clauses when memcache is disabled. And it
deregister the buffer whenever its use-count drops to 0 and register it
when it is used again.

This new get info option returns a comma separated list of protocols
that bmi is configured with.

BMI supports the given protocol.

- runs stable on XT4 and XT5
- tested up to 200,000 cores with IOFSL

removed dead code and consolidated files