218
|
|
|
Raghavendra D Prabhu |
9 years ago
|
|
|
217
|
|
|
Raghavendra D Prabhu |
9 years ago
|
|
|
216
|
|
|
Raghavendra D Prabhu |
9 years ago
|
|
|
215
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
214
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
213
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
212
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
211
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
210
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
209
|
|
Bug#1285380: Donor in desynced state makes the joiner wait indefinitely
So, this is how it looks:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%5 do { end = strchr(begin, ',');
int len;
if (NULL == end) { len = str_len - (begin - str); } else { len = end - begin; }
assert (len >= 0);
int const idx = len > 0 ? /* consider empty name as "any" */ group_find_node_by_name (group, joiner_idx, begin, len, status) : /* err == -EAGAIN here means that at least one of the nodes in the * list will be available later, so don't try others. */ (err == -EAGAIN ? err : group_find_node_by_state(group, joiner_idx, status));
if (idx >= 0) return idx;
/* once we hit -EAGAIN, don't try to change error code: this means * that at least one of the nodes in the list will become available. */ if (-EAGAIN != err) err = idx;
begin = end + 1; /* skip comma */
} while (end != NULL);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Based on my tests, when wsrep_sst_donor='A1,A2,' and A1 is unavailable (non-SYNCED), A3 does SST from A2 without any issues.
*However*, if wsrep_sst_donor='A1,' and A1 is unavailable, then it keeps looping without any bounds (there are no strict bounds on number of retries):
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
do { tries++;
gcs_seqno_t seqno_l;
ret = gcs_.request_state_transfer(req->req(), req->len(), sst_donor_, &seqno_l);
if (ret < 0) { if (!retry_str(ret)) { log_error << "Requesting state transfer failed: " << ret << "(" << strerror(-ret) << ")"; } else if (1 == tries) { log_info << "Requesting state transfer failed: " << ret << "(" << strerror(-ret) << "). " << "Will keep retrying every " << sst_retry_sec_ << " second(s)"; } }
if (seqno_l != GCS_SEQNO_ILL) { /* Check that we're not running out of space in monitor. */ if (local_monitor_.would_block(seqno_l)) { long const seconds = sst_retry_sec_ * tries; log_error << "We ran out of resources, seemingly because " << "we've been unsuccessfully requesting state " << "transfer for over " << seconds << " seconds. " << "Please check that there is " << "at least one fully synced member in the group. " << "Application must be restarted."; ret = -EDEADLK; } else { // we are already holding local monitor LocalOrder lo(seqno_l); local_monitor_.self_cancel(lo); } } }
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
From log:
140309 0:01:32 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff528 after receiving -11 140309 0:01:32 [Note] WSREP: galera/src/replicator_str.cpp:send_state_request():560: Requesting state transfer failed: -11(Resource temporarily unavailable). Will keep retrying every 1 second(s) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:16001,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140309 00:01:32.391) 140309 0:01:33 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff592 after receiving -11 140309 0:01:34 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff5fc after receiving -11 140309 0:01:35 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff666 after receiving -11 140309 0:01:36 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff6d0 after receiving -11 140309 0:01:37 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff73a after receiving -11 140309 0:01:38 [Note] [Debug] WSREP: gcs/src/gcs.c:gcs_replv():1568: Freeing gcache buffer 0x7f528bfff7a4 after receiving -11
So I see that, according to current design, it looks good. The current design being if a node in wsrep_sst_donor is unavailable then don't fall back at all (to group_find_node_by_state).
However, there is a flaw in that which is that in wsrep_sst_donor there is a choice given whether to leave a dangling comma or not. The former implies that check all nodes and try the fall provider logic whereas the latter implies a strict checking.
Currently, irrespective of whether the comma exists or not, it does only strict membership checking for donor without ever falling back. In case when a dangling comma is provided by user (for precisely that reason) - "A1,A2," it should check A1, A2 and if both are unavailable then check for others (may A3 or A5 are available (in a cluster of A1,A2,A3,A4,A5) as well.
The number of retries should also be bounded, but that is for another bug.
|
Raghavendra D Prabhu |
10 years ago
|
|
|
208
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
207
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
206
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
205
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
204
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
203
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
202
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
201
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
200
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|
199
|
|
|
Raghavendra D Prabhu |
10 years ago
|
|
|