file system resource becomes inaccesible when any of the node goes down

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

file system resource becomes inaccesible when any of the node goes down

Muhammad Sharfuddin-2
SLES 11 SP 3 + online updates(pacemaker-1.1.11-0.8.11.70
openais-1.1.4-5.22.1.7)

Its a dual primary drbd cluster, which mounts a file system resource on
both the cluster nodes simultaneously(file system type is ocfs2).

Whenever any of the nodes goes down, the file system(/sharedata) become
inaccessible for exact 35 seconds on the other (surviving/online) node,
and then become available again on the online node.

Please help me understand why the node which survives or remains online
unable to access the file system resource(/sharedata) for 35 seconds ?
and how can I fix the cluster so that file system remains accessible on
the surviving node without any interruption/delay(as in my case of about
35 seconds)

By inaccessible, I meant to say that running "ls -l /sharedata" and "df
/sharedata" does not return any output and does not return the prompt
back on the online node for exact 35 seconds once the other node becomes
offline.

e.g "node1" got offline somewhere around  01:37:15, and then /sharedata
file system was inaccessible during 01:37:35 and 01:38:18 on the online
node i.e "node2".


/var/log/messages on node2, when node1 went offline:
Jul  5 01:37:26 node2 kernel: [  675.255865] drbd r0: PingAck did not
arrive in time.
Jul  5 01:37:26 node2 kernel: [  675.255886] drbd r0: peer( Primary ->
Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Jul  5 01:37:26 node2 kernel: [  675.256030] block drbd0: new current
UUID C23D1458962AD18D:A8DD404C9F563391:6A5F4A26F64BAF0B:6A5E4A26F64BAF0B
Jul  5 01:37:26 node2 kernel: [  675.256079] drbd r0: asender terminated
Jul  5 01:37:26 node2 kernel: [  675.256081] drbd r0: Terminating drbd_a_r0
Jul  5 01:37:26 node2 kernel: [  675.256306] drbd r0: Connection closed
Jul  5 01:37:26 node2 kernel: [  675.256338] drbd r0: conn(
NetworkFailure -> Unconnected )
Jul  5 01:37:26 node2 kernel: [  675.256339] drbd r0: receiver terminated
Jul  5 01:37:26 node2 kernel: [  675.256340] drbd r0: Restarting
receiver thread
Jul  5 01:37:26 node2 kernel: [  675.256341] drbd r0: receiver (re)started
Jul  5 01:37:26 node2 kernel: [  675.256344] drbd r0: conn( Unconnected
-> WFConnection )
Jul  5 01:37:29 node2 corosync[4040]:  [TOTEM ] A processor failed,
forming new configuration.
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.131)
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice:
pcmk_peer_update: Transitional membership event on ring 216: memb=1,
new=0, lost=1
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
memb: node2 739307908
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
lost: node1 739307907
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice:
plugin_handle_membership: Membership 216: quorum lost
Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice:
plugin_handle_membership: Membership 216: quorum lost
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
Jul  5 01:37:35 node2 crmd[4050]:   notice: plugin_handle_membership:
Membership 216: quorum lost
Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
plugin_handle_membership: Membership 216: quorum lost
Jul  5 01:37:35 node2 cib[4045]:   notice: plugin_handle_membership:
Membership 216: quorum lost
Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice:
crm_update_peer_state: plugin_handle_membership: Node node1[739307907] -
state is now lost (was member)
Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice:
crm_update_peer_state: plugin_handle_membership: Node node1[739307907] -
state is now lost (was member)
Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match
for shutdown action on node1
Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: crm_update_peer_state:
plugin_handle_membership: Node node1[739307907] - state is now lost (was
member)
Jul  5 01:37:35 node2 cib[4045]:   notice: crm_update_peer_state:
plugin_handle_membership: Node node1[739307907] - state is now lost (was
member)
Jul  5 01:37:35 node2 cluster-dlm[4344]: update_cluster: Processing
membership 216
Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice:
pcmk_peer_update: Stable membership event on ring 216: memb=1, new=0, lost=0
Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
Stonith/shutdown of node1 not matched
Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Skipped
active node 739307908: born-on=204, last-seen=216, this-event=216,
last-event=212
Jul  5 01:37:35 node2 ocfs2_controld[4473]: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 1, joined 0
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
MEMB: node2 739307908
Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_peer_state:
plugin_handle_membership: Node node1[739307907] - state is now lost (was
member)
Jul  5 01:37:35 node2 cluster-dlm[4344]: del_configfs_node:
del_configfs_node rmdir "/sys/kernel/config/dlm/cluster/comms/739307907"
Jul  5 01:37:35 node2 ocfs2_controld[4473]: node daemon left 739307907
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
ais_mark_unseen_peer_dead: Node node1 was not seen in the previous
transition
Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match
for shutdown action on node1
Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Removed
inactive node 739307907: born-on=212, last-seen=212, this-event=216,
last-event=212
Jul  5 01:37:35 node2 ocfs2_controld[4473]: node down 739307907
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: update_member:
Node 739307907/node1 is now: lost
Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
Stonith/shutdown of node1 not matched
Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config: dlm:controld conf 1
0 1 memb 739307908 join left 739307907
Jul  5 01:37:35 node2 ocfs2_controld[4473]: Node 739307907 has left
mountgroup 12925001B70F4A529491DFF16F8B8352
Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
send_member_notification: Sending membership update 216 to 5 children
Jul  5 01:37:35 node2 sbd: [4028]: WARN: CIB: We do NOT have quorum!
Jul  5 01:37:35 node2 sbd: [4026]: WARN: Pacemaker health check: UNHEALTHY
Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config:
dlm:ls:12925001B70F4A529491DFF16F8B8352 conf 1 0 1 memb 739307908 join
left 739307907
Jul  5 01:37:35 node2 corosync[4040]:  [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jul  5 01:37:35 node2 corosync[4040]:  [CPG   ] chosen downlist: sender
r(0) ip(172.16.241.132) ; members(old:2 left:1)
Jul  5 01:37:35 node2 cluster-dlm[4344]: node_history_fail:
12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 set
Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
12925001B70F4A529491DFF16F8B8352 add_change cg 4 remove nodeid 739307907
reason 3
Jul  5 01:37:35 node2 corosync[4040]:  [MAIN  ] Completed service
synchronization, ready to provide service.
Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
12925001B70F4A529491DFF16F8B8352 add_change cg 4 counts member 1 joined
0 remove 1 failed 1
Jul  5 01:37:35 node2 cluster-dlm[4344]: stop_kernel:
12925001B70F4A529491DFF16F8B8352 stop_kernel cg 4
Jul  5 01:37:35 node2 ocfs2_controld[4473]: Sending notification of node
739307907 for "12925001B70F4A529491DFF16F8B8352"
Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
Jul  5 01:37:35 node2 cluster-dlm[4344]: do_sysfs: write "0" to
"/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
Jul  5 01:37:35 node2 ocfs2_controld[4473]: group
"ocfs2:12925001B70F4A529491DFF16F8B8352" confchg: members 1, left 1,
joined 0
Jul  5 01:37:35 node2 kernel: [  684.452042] dlm: closing connection to
node 739307907
Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
quorum status to false (call=114)
Jul  5 01:37:35 node2 cluster-dlm[4344]: fence_node_time: Node
739307907/node1 has not been shot yet
Jul  5 01:37:35 node2 crmd[4050]:   notice: do_state_transition: State
transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=check_join_state ]
Jul  5 01:37:35 node2 cluster-dlm[4344]: check_fencing_done:
12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 wait add
1436042043 fail 1436042255 last 0
Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
quorum status to false (call=119)
Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: shutdown (0)
Jul  5 01:37:35 node2 pengine[4049]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Jul  5 01:37:35 node2 pengine[4049]:  warning: pe_fence_node: Node node1
will be fenced because the node is no longer part of the cluster
Jul  5 01:37:35 node2 pengine[4049]:  warning: determine_online_status:
Node node1 is unclean
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_demote_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_demote_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: master-p-drbd (10000)
Jul  5 01:37:35 node2 cluster-dlm[4344]: set_fs_notified:
12925001B70F4A529491DFF16F8B8352 set_fs_notified nodeid 739307907
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_demote_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Jul  5 01:37:35 node2 ocfs2_controld[4473]: message from dlmcontrol
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 ocfs2_controld[4473]: Notified for
"12925001B70F4A529491DFF16F8B8352", node 739307907, status 0
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_demote_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 ocfs2_controld[4473]: Completing notification on
"12925001B70F4A529491DFF16F8B8352" for node 739307907
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-drbd:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-dlm:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-dlm:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-o2cb:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-o2cb:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-vg1:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-vg1:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-sharefs:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
p-sharefs:1_stop_0 on node1 is unrunnable (offline)
Jul  5 01:37:35 node2 pengine[4049]:  warning: stage6: Scheduling Node
node1 for STONITH
Jul  5 01:37:35 node2 pengine[4049]:    error: rsc_expand_action:
Couldn't expand cl-lock_demote_0
Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:37:35 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:37:35 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:37:35 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:37:35 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Demote
p-drbd:1    (Master -> Stopped node1)
Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
p-dlm:1    (node1)
Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
p-o2cb:1    (node1)
Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
p-vg1:1    (node1)
Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
p-sharefs:1    (node1)
Jul  5 01:37:35 node2 pengine[4049]:  warning: process_pe_message:
Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-5.bz2
Jul  5 01:37:35 node2 crmd[4050]:   notice: do_te_invoke: Processing
graph 6 (ref=pe_calc-dc-1436042255-74) derived from
/var/lib/pacemaker/pengine/pe-warn-5.bz2
Jul  5 01:37:35 node2 crmd[4050]:   notice: te_fence_node: Executing
reboot fencing operation (79) on node1 (timeout=90000)
Jul  5 01:37:35 node2 crmd[4050]:   notice: te_rsc_command: Initiating
action 95: notify p-drbd_pre_notify_demote_0 on node2 (local)
Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: handle_request: Client
crmd.4050.3c8ccdb0 wants to fence (reboot) 'node1' with device '(any)'
Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
initiate_remote_stonith_op: Initiating remote operation reboot for
node1: 2fce4697-65d5-40d2-b5b4-5c10be4d3d5a (0)
Jul  5 01:37:35 node2 crmd[4050]:   notice: process_lrm_event: LRM
operation p-drbd_notify_0 (call=54, rc=0, cib-update=0, confirmed=true) ok
Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
can_fence_host_with_device: sbd_stonith can fence node1: dynamic-list
Jul  5 01:37:35 node2 sbd: [6197]: info: Delivery process handling
/dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382dfbd
Jul  5 01:37:35 node2 sbd: [6197]: info: Device UUID:
4e398182-4894-4f11-b84e-6c6ad4c02614
Jul  5 01:37:35 node2 sbd: [6197]: info: Writing reset to node slot node1
Jul  5 01:37:35 node2 sbd: [6197]: info: Messaging delay: 40
Jul  5 01:38:15 node2 sbd: [6197]: info: reset successfully delivered to
node1
Jul  5 01:38:15 node2 sbd: [6196]: info: Message successfully delivered.
Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: log_operation:
Operation 'reboot' [6185] (call 3 from crmd.4050) for host 'node1' with
device 'sbd_stonith' returned: 0 (OK)
Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: remote_op_done:
Operation reboot of node1 by node2 for crmd.4050@node2.2fce4697: OK
Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_callback:
Stonith operation 3/79:6:0:34e9a1ca-ebc9-4114-b5b7-26e670dc6aa0: OK (0)
Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_notify: Peer
node1 was terminated (reboot) by node2 for node2: OK
(ref=2fce4697-65d5-40d2-b5b4-5c10be4d3d5a) by client crmd.4050
Jul  5 01:38:16 node2 crmd[4050]:   notice: te_rsc_command: Initiating
action 96: notify p-drbd_post_notify_demote_0 on node2 (local)
Jul  5 01:38:16 node2 crmd[4050]:   notice: process_lrm_event: LRM
operation p-drbd_notify_0 (call=55, rc=0, cib-update=0, confirmed=true) ok
Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 6
(Complete=24, Pending=0, Fired=0, Skipped=6, Incomplete=5,
Source=/var/lib/pacemaker/pengine/pe-warn-5.bz2): Stopped
Jul  5 01:38:16 node2 pengine[4049]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Jul  5 01:38:16 node2 pengine[4049]:    error: rsc_expand_action:
Couldn't expand cl-lock_demote_0
Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:38:16 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:38:16 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:38:16 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
clone_update_actions_interleave: Triggered assert at clone.c:1260 :
first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
Jul  5 01:38:16 node2 pengine[4049]:    error:
clone_update_actions_interleave: No action found for demote in g-lock:0
(first)
Jul  5 01:38:16 node2 pengine[4049]:   notice: process_pe_message:
Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-30.bz2
Jul  5 01:38:16 node2 crmd[4050]:   notice: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1436042296-79) derived from
/var/lib/pacemaker/pengine/pe-input-30.bz2
Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 7
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Complete
Jul  5 01:38:16 node2 crmd[4050]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul  5 01:38:17 node2 cluster-dlm[4344]: fence_node_time: Node
739307907/node1 was last shot 'now'
Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 done add
1436042043 fail 1436042255 last 1436042297
Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
12925001B70F4A529491DFF16F8B8352 check_fencing done
Jul  5 01:38:17 node2 cluster-dlm[4344]: check_quorum_done:
12925001B70F4A529491DFF16F8B8352 check_quorum disabled
Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 clear
Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
12925001B70F4A529491DFF16F8B8352 check_fs done
Jul  5 01:38:17 node2 cluster-dlm[4344]: send_info:
12925001B70F4A529491DFF16F8B8352 send_start cg 4 flags 2 data2 0 counts
3 1 0 1 1
Jul  5 01:38:17 node2 cluster-dlm[4344]: receive_start:
12925001B70F4A529491DFF16F8B8352 receive_start 739307908:4 len 76
Jul  5 01:38:17 node2 cluster-dlm[4344]: match_change:
12925001B70F4A529491DFF16F8B8352 match_change 739307908:4 matches cg 4
Jul  5 01:38:17 node2 cluster-dlm[4344]: wait_messages_done:
12925001B70F4A529491DFF16F8B8352 wait_messages cg 4 got all 1
Jul  5 01:38:17 node2 cluster-dlm[4344]: start_kernel:
12925001B70F4A529491DFF16F8B8352 start_kernel cg 4 member_count 1
Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
739307907
Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
739307908
Jul  5 01:38:17 node2 cluster-dlm[4344]: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/12925001B70F4A529491DFF16F8B8352/nodes/739307907"
Jul  5 01:38:17 node2 cluster-dlm[4344]: do_sysfs: write "1" to
"/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
Jul  5 01:38:17 node2 kernel: [  726.576130]
(ocfs2rec,6152,0):ocfs2_replay_journal:1549 Recovering node 739307907
from slot 0 on device (253,0)
Jul  5 01:38:18 node2 kernel: [  727.453597]
(ocfs2rec,6152,0):ocfs2_begin_quota_recovery:407 Beginning quota
recovery in slot 0
Jul  5 01:38:18 node2 kernel: [  727.455965]
(kworker/u:28,139,0):ocfs2_finish_quota_recovery:599 Finishing quota
recovery in slot 0


cib:
primitive p-dlm ocf:pacemaker:controld \
         op monitor interval="120" timeout="30" \
         op start interval="0" timeout="90" \
         op stop interval="0" timeout="100"
primitive p-drbd ocf:linbit:drbd \
         params drbd_resource="r0" \
         op monitor interval="50" role="Master" timeout="30" \
         op monitor interval="60" role="Slave" timeout="30" \
         op start interval="0" timeout="240" \
         op stop interval="0" timeout="100"
primitive p-o2cb ocf:ocfs2:o2cb \
         op monitor interval="120" timeout="30" \
         op start interval="0" timeout="90" \
         op stop interval="0" timeout="100"
primitive p-sharefs ocf:heartbeat:Filesystem \
         params device="/dev/vg1/sharelv" directory="/sharedata"
fstype="ocfs2" \
         op monitor interval="60s" timeout="60s" \
         op start interval="0" timeout="90s" \
         op stop interval="0" timeout="90s"
primitive p-vg1 ocf:heartbeat:LVM \
         params volgrpname="vg1" \
         op monitor interval="60s" timeout="40" \
         op start interval="0" timeout="40" \
         op stop interval="0" timeout="40"
primitive sbd_stonith stonith:external/sbd \
         meta target-role="Started" \
         op monitor interval="3000" timeout="20" \
         op start interval="0" timeout="20" \
         op stop interval="0" timeout="20" \
         params
sbd_device="/dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382dfbd"
group g-lock p-dlm p-o2cb
group g-sharedata p-vg1 p-sharefs
ms ms-drbd p-drbd \
         meta master-max="2" clone-max="2" notify="true" interleave="true"
clone cl-lock g-lock \
         meta globally-unique="false" interleave="true"
clone cl-sharedata g-sharedata \
         meta globally-unique="false" interleave="true"
colocation col-drbd-lock inf: cl-lock ms-drbd:Master
colocation col-lock-sharedata inf: cl-sharedata cl-lock
order ord-drbd-lock inf: ms-drbd:promote cl-lock
order ord-lock-sharedata inf: cl-lock cl-sharedata
         stonith-timeout="90s" \
         no-quorum-policy="ignore" \
         expected-quorum-votes="2"


--
Regards,

Muhammad Sharfuddin
_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: file system resource becomes inaccesible when any of the node goes down

emmanuel segura
when a node goes down, you will see the node in unclean state, how you
see in your logs, forming new configuration(corosync) -> stonith
reboot request -> and you are using sbd and the node become offline
after thet msgwait is expired, when msgwait is expired pacemaker knows
the node is dead and than the ocfs will ok, stonith->dlm->ocfs2, if
you need to reduce the msgwait you need to be careful about others
timeouts and cluster problems.

2015-07-05 18:13 GMT+02:00 Muhammad Sharfuddin <[hidden email]>:

> SLES 11 SP 3 + online updates(pacemaker-1.1.11-0.8.11.70
> openais-1.1.4-5.22.1.7)
>
> Its a dual primary drbd cluster, which mounts a file system resource on both
> the cluster nodes simultaneously(file system type is ocfs2).
>
> Whenever any of the nodes goes down, the file system(/sharedata) become
> inaccessible for exact 35 seconds on the other (surviving/online) node, and
> then become available again on the online node.
>
> Please help me understand why the node which survives or remains online
> unable to access the file system resource(/sharedata) for 35 seconds ? and
> how can I fix the cluster so that file system remains accessible on the
> surviving node without any interruption/delay(as in my case of about 35
> seconds)
>
> By inaccessible, I meant to say that running "ls -l /sharedata" and "df
> /sharedata" does not return any output and does not return the prompt back
> on the online node for exact 35 seconds once the other node becomes offline.
>
> e.g "node1" got offline somewhere around  01:37:15, and then /sharedata file
> system was inaccessible during 01:37:35 and 01:38:18 on the online node i.e
> "node2".
>
>
> /var/log/messages on node2, when node1 went offline:
> Jul  5 01:37:26 node2 kernel: [  675.255865] drbd r0: PingAck did not arrive
> in time.
> Jul  5 01:37:26 node2 kernel: [  675.255886] drbd r0: peer( Primary ->
> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
> Jul  5 01:37:26 node2 kernel: [  675.256030] block drbd0: new current UUID
> C23D1458962AD18D:A8DD404C9F563391:6A5F4A26F64BAF0B:6A5E4A26F64BAF0B
> Jul  5 01:37:26 node2 kernel: [  675.256079] drbd r0: asender terminated
> Jul  5 01:37:26 node2 kernel: [  675.256081] drbd r0: Terminating drbd_a_r0
> Jul  5 01:37:26 node2 kernel: [  675.256306] drbd r0: Connection closed
> Jul  5 01:37:26 node2 kernel: [  675.256338] drbd r0: conn( NetworkFailure
> -> Unconnected )
> Jul  5 01:37:26 node2 kernel: [  675.256339] drbd r0: receiver terminated
> Jul  5 01:37:26 node2 kernel: [  675.256340] drbd r0: Restarting receiver
> thread
> Jul  5 01:37:26 node2 kernel: [  675.256341] drbd r0: receiver (re)started
> Jul  5 01:37:26 node2 kernel: [  675.256344] drbd r0: conn( Unconnected ->
> WFConnection )
> Jul  5 01:37:29 node2 corosync[4040]:  [TOTEM ] A processor failed, forming
> new configuration.
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.131)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice: pcmk_peer_update:
> Transitional membership event on ring 216: memb=1, new=0, lost=1
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> memb: node2 739307908
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> lost: node1 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
> Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice:
> plugin_handle_membership: Membership 216: quorum lost
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
> Jul  5 01:37:35 node2 crmd[4050]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 cib[4045]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
> Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match for
> shutdown action on node1
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cib[4045]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: update_cluster: Processing
> membership 216
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice: pcmk_peer_update:
> Stable membership event on ring 216: memb=1, new=0, lost=0
> Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
> Stonith/shutdown of node1 not matched
> Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Skipped active
> node 739307908: born-on=204, last-seen=216, this-event=216, last-event=212
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: ocfs2_controld (group
> "ocfs2:controld") confchg: members 1, left 1, joined 0
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> MEMB: node2 739307908
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: del_configfs_node:
> del_configfs_node rmdir "/sys/kernel/config/dlm/cluster/comms/739307907"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: node daemon left 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
> ais_mark_unseen_peer_dead: Node node1 was not seen in the previous
> transition
> Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match for
> shutdown action on node1
> Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Removed inactive
> node 739307907: born-on=212, last-seen=212, this-event=216, last-event=212
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: node down 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: update_member: Node
> 739307907/node1 is now: lost
> Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
> Stonith/shutdown of node1 not matched
> Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config: dlm:controld conf 1 0 1
> memb 739307908 join left 739307907
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Node 739307907 has left
> mountgroup 12925001B70F4A529491DFF16F8B8352
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
> send_member_notification: Sending membership update 216 to 5 children
> Jul  5 01:37:35 node2 sbd: [4028]: WARN: CIB: We do NOT have quorum!
> Jul  5 01:37:35 node2 sbd: [4026]: WARN: Pacemaker health check: UNHEALTHY
> Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config:
> dlm:ls:12925001B70F4A529491DFF16F8B8352 conf 1 0 1 memb 739307908 join left
> 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [TOTEM ] A processor joined or left
> the membership and a new membership was formed.
> Jul  5 01:37:35 node2 corosync[4040]:  [CPG   ] chosen downlist: sender r(0)
> ip(172.16.241.132) ; members(old:2 left:1)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: node_history_fail:
> 12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 set
> Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
> 12925001B70F4A529491DFF16F8B8352 add_change cg 4 remove nodeid 739307907
> reason 3
> Jul  5 01:37:35 node2 corosync[4040]:  [MAIN  ] Completed service
> synchronization, ready to provide service.
> Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
> 12925001B70F4A529491DFF16F8B8352 add_change cg 4 counts member 1 joined 0
> remove 1 failed 1
> Jul  5 01:37:35 node2 cluster-dlm[4344]: stop_kernel:
> 12925001B70F4A529491DFF16F8B8352 stop_kernel cg 4
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Sending notification of node
> 739307907 for "12925001B70F4A529491DFF16F8B8352"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
> Jul  5 01:37:35 node2 cluster-dlm[4344]: do_sysfs: write "0" to
> "/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: group
> "ocfs2:12925001B70F4A529491DFF16F8B8352" confchg: members 1, left 1, joined
> 0
> Jul  5 01:37:35 node2 kernel: [  684.452042] dlm: closing connection to node
> 739307907
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
> quorum status to false (call=114)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: fence_node_time: Node
> 739307907/node1 has not been shot yet
> Jul  5 01:37:35 node2 crmd[4050]:   notice: do_state_transition: State
> transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL
> origin=check_join_state ]
> Jul  5 01:37:35 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 wait add 1436042043
> fail 1436042255 last 0
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
> quorum status to false (call=119)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_local_callback: Sending
> full refresh (origin=crmd)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update: Sending
> flush op to all hosts for: shutdown (0)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jul  5 01:37:35 node2 pengine[4049]:  warning: pe_fence_node: Node node1
> will be fenced because the node is no longer part of the cluster
> Jul  5 01:37:35 node2 pengine[4049]:  warning: determine_online_status: Node
> node1 is unclean
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-p-drbd (10000)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: set_fs_notified:
> 12925001B70F4A529491DFF16F8B8352 set_fs_notified nodeid 739307907
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update: Sending
> flush op to all hosts for: probe_complete (true)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: message from dlmcontrol
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Notified for
> "12925001B70F4A529491DFF16F8B8352", node 739307907, status 0
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Completing notification on
> "12925001B70F4A529491DFF16F8B8352" for node 739307907
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-dlm:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-dlm:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-o2cb:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-o2cb:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-vg1:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-vg1:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-sharefs:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-sharefs:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: stage6: Scheduling Node node1
> for STONITH
> Jul  5 01:37:35 node2 pengine[4049]:    error: rsc_expand_action: Couldn't
> expand cl-lock_demote_0
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Demote p-drbd:1
> (Master -> Stopped node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop p-dlm:1
> (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop p-o2cb:1
> (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop p-vg1:1
> (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop p-sharefs:1
> (node1)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: process_pe_message:
> Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-5.bz2
> Jul  5 01:37:35 node2 crmd[4050]:   notice: do_te_invoke: Processing graph 6
> (ref=pe_calc-dc-1436042255-74) derived from
> /var/lib/pacemaker/pengine/pe-warn-5.bz2
> Jul  5 01:37:35 node2 crmd[4050]:   notice: te_fence_node: Executing reboot
> fencing operation (79) on node1 (timeout=90000)
> Jul  5 01:37:35 node2 crmd[4050]:   notice: te_rsc_command: Initiating
> action 95: notify p-drbd_pre_notify_demote_0 on node2 (local)
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: handle_request: Client
> crmd.4050.3c8ccdb0 wants to fence (reboot) 'node1' with device '(any)'
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
> initiate_remote_stonith_op: Initiating remote operation reboot for node1:
> 2fce4697-65d5-40d2-b5b4-5c10be4d3d5a (0)
> Jul  5 01:37:35 node2 crmd[4050]:   notice: process_lrm_event: LRM operation
> p-drbd_notify_0 (call=54, rc=0, cib-update=0, confirmed=true) ok
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
> can_fence_host_with_device: sbd_stonith can fence node1: dynamic-list
> Jul  5 01:37:35 node2 sbd: [6197]: info: Delivery process handling
> /dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382dfbd
> Jul  5 01:37:35 node2 sbd: [6197]: info: Device UUID:
> 4e398182-4894-4f11-b84e-6c6ad4c02614
> Jul  5 01:37:35 node2 sbd: [6197]: info: Writing reset to node slot node1
> Jul  5 01:37:35 node2 sbd: [6197]: info: Messaging delay: 40
> Jul  5 01:38:15 node2 sbd: [6197]: info: reset successfully delivered to
> node1
> Jul  5 01:38:15 node2 sbd: [6196]: info: Message successfully delivered.
> Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: log_operation: Operation
> 'reboot' [6185] (call 3 from crmd.4050) for host 'node1' with device
> 'sbd_stonith' returned: 0 (OK)
> Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: remote_op_done: Operation
> reboot of node1 by node2 for crmd.4050@node2.2fce4697: OK
> Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_callback:
> Stonith operation 3/79:6:0:34e9a1ca-ebc9-4114-b5b7-26e670dc6aa0: OK (0)
> Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_notify: Peer
> node1 was terminated (reboot) by node2 for node2: OK
> (ref=2fce4697-65d5-40d2-b5b4-5c10be4d3d5a) by client crmd.4050
> Jul  5 01:38:16 node2 crmd[4050]:   notice: te_rsc_command: Initiating
> action 96: notify p-drbd_post_notify_demote_0 on node2 (local)
> Jul  5 01:38:16 node2 crmd[4050]:   notice: process_lrm_event: LRM operation
> p-drbd_notify_0 (call=55, rc=0, cib-update=0, confirmed=true) ok
> Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 6
> (Complete=24, Pending=0, Fired=0, Skipped=6, Incomplete=5,
> Source=/var/lib/pacemaker/pengine/pe-warn-5.bz2): Stopped
> Jul  5 01:38:16 node2 pengine[4049]:   notice: unpack_config: On loss of CCM
> Quorum: Ignore
> Jul  5 01:38:16 node2 pengine[4049]:    error: rsc_expand_action: Couldn't
> expand cl-lock_demote_0
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:   notice: process_pe_message:
> Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-30.bz2
> Jul  5 01:38:16 node2 crmd[4050]:   notice: do_te_invoke: Processing graph 7
> (ref=pe_calc-dc-1436042296-79) derived from
> /var/lib/pacemaker/pengine/pe-input-30.bz2
> Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 7
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Complete
> Jul  5 01:38:16 node2 crmd[4050]:   notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jul  5 01:38:17 node2 cluster-dlm[4344]: fence_node_time: Node
> 739307907/node1 was last shot 'now'
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 done add 1436042043
> fail 1436042255 last 1436042297
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing done
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_quorum_done:
> 12925001B70F4A529491DFF16F8B8352 check_quorum disabled
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
> 12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 clear
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
> 12925001B70F4A529491DFF16F8B8352 check_fs done
> Jul  5 01:38:17 node2 cluster-dlm[4344]: send_info:
> 12925001B70F4A529491DFF16F8B8352 send_start cg 4 flags 2 data2 0 counts 3 1
> 0 1 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: receive_start:
> 12925001B70F4A529491DFF16F8B8352 receive_start 739307908:4 len 76
> Jul  5 01:38:17 node2 cluster-dlm[4344]: match_change:
> 12925001B70F4A529491DFF16F8B8352 match_change 739307908:4 matches cg 4
> Jul  5 01:38:17 node2 cluster-dlm[4344]: wait_messages_done:
> 12925001B70F4A529491DFF16F8B8352 wait_messages cg 4 got all 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: start_kernel:
> 12925001B70F4A529491DFF16F8B8352 start_kernel cg 4 member_count 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
> 739307907
> Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
> 739307908
> Jul  5 01:38:17 node2 cluster-dlm[4344]: set_configfs_members: set_members
> rmdir
> "/sys/kernel/config/dlm/cluster/spaces/12925001B70F4A529491DFF16F8B8352/nodes/739307907"
> Jul  5 01:38:17 node2 cluster-dlm[4344]: do_sysfs: write "1" to
> "/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
> Jul  5 01:38:17 node2 kernel: [  726.576130]
> (ocfs2rec,6152,0):ocfs2_replay_journal:1549 Recovering node 739307907 from
> slot 0 on device (253,0)
> Jul  5 01:38:18 node2 kernel: [  727.453597]
> (ocfs2rec,6152,0):ocfs2_begin_quota_recovery:407 Beginning quota recovery in
> slot 0
> Jul  5 01:38:18 node2 kernel: [  727.455965]
> (kworker/u:28,139,0):ocfs2_finish_quota_recovery:599 Finishing quota
> recovery in slot 0
>
>
> cib:
> primitive p-dlm ocf:pacemaker:controld \
>         op monitor interval="120" timeout="30" \
>         op start interval="0" timeout="90" \
>         op stop interval="0" timeout="100"
> primitive p-drbd ocf:linbit:drbd \
>         params drbd_resource="r0" \
>         op monitor interval="50" role="Master" timeout="30" \
>         op monitor interval="60" role="Slave" timeout="30" \
>         op start interval="0" timeout="240" \
>         op stop interval="0" timeout="100"
> primitive p-o2cb ocf:ocfs2:o2cb \
>         op monitor interval="120" timeout="30" \
>         op start interval="0" timeout="90" \
>         op stop interval="0" timeout="100"
> primitive p-sharefs ocf:heartbeat:Filesystem \
>         params device="/dev/vg1/sharelv" directory="/sharedata"
> fstype="ocfs2" \
>         op monitor interval="60s" timeout="60s" \
>         op start interval="0" timeout="90s" \
>         op stop interval="0" timeout="90s"
> primitive p-vg1 ocf:heartbeat:LVM \
>         params volgrpname="vg1" \
>         op monitor interval="60s" timeout="40" \
>         op start interval="0" timeout="40" \
>         op stop interval="0" timeout="40"
> primitive sbd_stonith stonith:external/sbd \
>         meta target-role="Started" \
>         op monitor interval="3000" timeout="20" \
>         op start interval="0" timeout="20" \
>         op stop interval="0" timeout="20" \
>         params
> sbd_device="/dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382dfbd"
> group g-lock p-dlm p-o2cb
> group g-sharedata p-vg1 p-sharefs
> ms ms-drbd p-drbd \
>         meta master-max="2" clone-max="2" notify="true" interleave="true"
> clone cl-lock g-lock \
>         meta globally-unique="false" interleave="true"
> clone cl-sharedata g-sharedata \
>         meta globally-unique="false" interleave="true"
> colocation col-drbd-lock inf: cl-lock ms-drbd:Master
> colocation col-lock-sharedata inf: cl-sharedata cl-lock
> order ord-drbd-lock inf: ms-drbd:promote cl-lock
> order ord-lock-sharedata inf: cl-lock cl-sharedata
>         stonith-timeout="90s" \
>         no-quorum-policy="ignore" \
>         expected-quorum-votes="2"
>
>
> --
> Regards,
>
> Muhammad Sharfuddin
> _______________________________________________
> Linux-HA mailing list is closing down.
> Please subscribe to [hidden email] instead.
> http://clusterlabs.org/mailman/listinfo/users
> _______________________________________________
> [hidden email]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha



--
  .~.
  /V\
 //  \\
/(   )\
^`~'^
_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Antw: file system resource becomes inaccesible when any of the node goes down

Ulrich Windl
In reply to this post by Muhammad Sharfuddin-2
>>> Muhammad Sharfuddin <[hidden email]> schrieb am 05.07.2015 um 18:13 in
Nachricht <[hidden email]>:
> SLES 11 SP 3 + online updates(pacemaker-1.1.11-0.8.11.70
> openais-1.1.4-5.22.1.7)
>
> Its a dual primary drbd cluster, which mounts a file system resource on
> both the cluster nodes simultaneously(file system type is ocfs2).
>
> Whenever any of the nodes goes down, the file system(/sharedata) become
> inaccessible for exact 35 seconds on the other (surviving/online) node,
> and then become available again on the online node.

I'm no expert on DRBD, but I guess DRBD has to resolve the split brain situation: Obviously both nodes cannot continue to be primary if the connection between the nodes is down. When one node is down, DRDB first sees a connection that is down. If the timeout were tiny, you might have a lot of false "connection interrupted" alerts.

>
> Please help me understand why the node which survives or remains online
> unable to access the file system resource(/sharedata) for 35 seconds ?
> and how can I fix the cluster so that file system remains accessible on
> the surviving node without any interruption/delay(as in my case of about
> 35 seconds)

For true continuous access I guess you need true shared disks, or a clean DRBD shutdown on the other node.

Regards,
Ulrich

>
> By inaccessible, I meant to say that running "ls -l /sharedata" and "df
> /sharedata" does not return any output and does not return the prompt
> back on the online node for exact 35 seconds once the other node becomes
> offline.
>
> e.g "node1" got offline somewhere around  01:37:15, and then /sharedata
> file system was inaccessible during 01:37:35 and 01:38:18 on the online
> node i.e "node2".
>
>
> /var/log/messages on node2, when node1 went offline:
> Jul  5 01:37:26 node2 kernel: [  675.255865] drbd r0: PingAck did not
> arrive in time.
> Jul  5 01:37:26 node2 kernel: [  675.255886] drbd r0: peer( Primary ->
> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
> Jul  5 01:37:26 node2 kernel: [  675.256030] block drbd0: new current
> UUID C23D1458962AD18D:A8DD404C9F563391:6A5F4A26F64BAF0B:6A5E4A26F64BAF0B
> Jul  5 01:37:26 node2 kernel: [  675.256079] drbd r0: asender terminated
> Jul  5 01:37:26 node2 kernel: [  675.256081] drbd r0: Terminating drbd_a_r0
> Jul  5 01:37:26 node2 kernel: [  675.256306] drbd r0: Connection closed
> Jul  5 01:37:26 node2 kernel: [  675.256338] drbd r0: conn(
> NetworkFailure -> Unconnected )
> Jul  5 01:37:26 node2 kernel: [  675.256339] drbd r0: receiver terminated
> Jul  5 01:37:26 node2 kernel: [  675.256340] drbd r0: Restarting
> receiver thread
> Jul  5 01:37:26 node2 kernel: [  675.256341] drbd r0: receiver (re)started
> Jul  5 01:37:26 node2 kernel: [  675.256344] drbd r0: conn( Unconnected
> -> WFConnection )
> Jul  5 01:37:29 node2 corosync[4040]:  [TOTEM ] A processor failed,
> forming new configuration.
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.131)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice:
> pcmk_peer_update: Transitional membership event on ring 216: memb=1,
> new=0, lost=1
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> memb: node2 739307908
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> lost: node1 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] CLM CONFIGURATION CHANGE
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] New Configuration:
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ]     r(0) ip(172.16.241.132)
> Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice:
> plugin_handle_membership: Membership 216: quorum lost
> Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice:
> plugin_handle_membership: Membership 216: quorum lost
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Left:
> Jul  5 01:37:35 node2 crmd[4050]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
> plugin_handle_membership: Membership 216: quorum lost
> Jul  5 01:37:35 node2 cib[4045]:   notice: plugin_handle_membership:
> Membership 216: quorum lost
> Jul  5 01:37:35 node2 cluster-dlm[4344]:   notice:
> crm_update_peer_state: plugin_handle_membership: Node node1[739307907] -
> state is now lost (was member)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]:   notice:
> crm_update_peer_state: plugin_handle_membership: Node node1[739307907] -
> state is now lost (was member)
> Jul  5 01:37:35 node2 corosync[4040]:  [CLM   ] Members Joined:
> Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match
> for shutdown action on node1
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cib[4045]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: update_cluster: Processing
> membership 216
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] notice:
> pcmk_peer_update: Stable membership event on ring 216: memb=1, new=0, lost=0
> Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
> Stonith/shutdown of node1 not matched
> Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Skipped
> active node 739307908: born-on=204, last-seen=216, this-event=216,
> last-event=212
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: ocfs2_controld (group
> "ocfs2:controld") confchg: members 1, left 1, joined 0
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: pcmk_peer_update:
> MEMB: node2 739307908
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_peer_state:
> plugin_handle_membership: Node node1[739307907] - state is now lost (was
> member)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: del_configfs_node:
> del_configfs_node rmdir "/sys/kernel/config/dlm/cluster/comms/739307907"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: node daemon left 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
> ais_mark_unseen_peer_dead: Node node1 was not seen in the previous
> transition
> Jul  5 01:37:35 node2 crmd[4050]:  warning: match_down_event: No match
> for shutdown action on node1
> Jul  5 01:37:35 node2 cluster-dlm[4344]: dlm_process_node: Removed
> inactive node 739307907: born-on=212, last-seen=212, this-event=216,
> last-event=212
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: node down 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info: update_member:
> Node 739307907/node1 is now: lost
> Jul  5 01:37:35 node2 crmd[4050]:   notice: peer_update_callback:
> Stonith/shutdown of node1 not matched
> Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config: dlm:controld conf 1
> 0 1 memb 739307908 join left 739307907
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Node 739307907 has left
> mountgroup 12925001B70F4A529491DFF16F8B8352
> Jul  5 01:37:35 node2 corosync[4040]:  [pcmk  ] info:
> send_member_notification: Sending membership update 216 to 5 children
> Jul  5 01:37:35 node2 sbd: [4028]: WARN: CIB: We do NOT have quorum!
> Jul  5 01:37:35 node2 sbd: [4026]: WARN: Pacemaker health check: UNHEALTHY
> Jul  5 01:37:35 node2 cluster-dlm[4344]: log_config:
> dlm:ls:12925001B70F4A529491DFF16F8B8352 conf 1 0 1 memb 739307908 join
> left 739307907
> Jul  5 01:37:35 node2 corosync[4040]:  [TOTEM ] A processor joined or
> left the membership and a new membership was formed.
> Jul  5 01:37:35 node2 corosync[4040]:  [CPG   ] chosen downlist: sender
> r(0) ip(172.16.241.132) ; members(old:2 left:1)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: node_history_fail:
> 12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 set
> Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
> 12925001B70F4A529491DFF16F8B8352 add_change cg 4 remove nodeid 739307907
> reason 3
> Jul  5 01:37:35 node2 corosync[4040]:  [MAIN  ] Completed service
> synchronization, ready to provide service.
> Jul  5 01:37:35 node2 cluster-dlm[4344]: add_change:
> 12925001B70F4A529491DFF16F8B8352 add_change cg 4 counts member 1 joined
> 0 remove 1 failed 1
> Jul  5 01:37:35 node2 cluster-dlm[4344]: stop_kernel:
> 12925001B70F4A529491DFF16F8B8352 stop_kernel cg 4
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Sending notification of node
> 739307907 for "12925001B70F4A529491DFF16F8B8352"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: confchg called
> Jul  5 01:37:35 node2 cluster-dlm[4344]: do_sysfs: write "0" to
> "/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: group
> "ocfs2:12925001B70F4A529491DFF16F8B8352" confchg: members 1, left 1,
> joined 0
> Jul  5 01:37:35 node2 kernel: [  684.452042] dlm: closing connection to
> node 739307907
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
> quorum status to false (call=114)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: fence_node_time: Node
> 739307907/node1 has not been shot yet
> Jul  5 01:37:35 node2 crmd[4050]:   notice: do_state_transition: State
> transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Jul  5 01:37:35 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 wait add
> 1436042043 fail 1436042255 last 0
> Jul  5 01:37:35 node2 crmd[4050]:   notice: crm_update_quorum: Updating
> quorum status to false (call=119)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_local_callback:
> Sending full refresh (origin=crmd)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
> Sending flush op to all hosts for: shutdown (0)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jul  5 01:37:35 node2 pengine[4049]:  warning: pe_fence_node: Node node1
> will be fenced because the node is no longer part of the cluster
> Jul  5 01:37:35 node2 pengine[4049]:  warning: determine_online_status:
> Node node1 is unclean
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-p-drbd (10000)
> Jul  5 01:37:35 node2 cluster-dlm[4344]: set_fs_notified:
> 12925001B70F4A529491DFF16F8B8352 set_fs_notified nodeid 739307907
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 attrd[4048]:   notice: attrd_trigger_update:
> Sending flush op to all hosts for: probe_complete (true)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: message from dlmcontrol
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Notified for
> "12925001B70F4A529491DFF16F8B8352", node 739307907, status 0
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_demote_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 ocfs2_controld[4473]: Completing notification on
> "12925001B70F4A529491DFF16F8B8352" for node 739307907
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-drbd:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-dlm:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-dlm:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-o2cb:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-o2cb:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-vg1:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-vg1:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-sharefs:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: custom_action: Action
> p-sharefs:1_stop_0 on node1 is unrunnable (offline)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: stage6: Scheduling Node
> node1 for STONITH
> Jul  5 01:37:35 node2 pengine[4049]:    error: rsc_expand_action:
> Couldn't expand cl-lock_demote_0
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:37:35 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Demote
> p-drbd:1    (Master -> Stopped node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
> p-dlm:1    (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
> p-o2cb:1    (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
> p-vg1:1    (node1)
> Jul  5 01:37:35 node2 pengine[4049]:   notice: LogActions: Stop
> p-sharefs:1    (node1)
> Jul  5 01:37:35 node2 pengine[4049]:  warning: process_pe_message:
> Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-5.bz2
> Jul  5 01:37:35 node2 crmd[4050]:   notice: do_te_invoke: Processing
> graph 6 (ref=pe_calc-dc-1436042255-74) derived from
> /var/lib/pacemaker/pengine/pe-warn-5.bz2
> Jul  5 01:37:35 node2 crmd[4050]:   notice: te_fence_node: Executing
> reboot fencing operation (79) on node1 (timeout=90000)
> Jul  5 01:37:35 node2 crmd[4050]:   notice: te_rsc_command: Initiating
> action 95: notify p-drbd_pre_notify_demote_0 on node2 (local)
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice: handle_request: Client
> crmd.4050.3c8ccdb0 wants to fence (reboot) 'node1' with device '(any)'
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
> initiate_remote_stonith_op: Initiating remote operation reboot for
> node1: 2fce4697-65d5-40d2-b5b4-5c10be4d3d5a (0)
> Jul  5 01:37:35 node2 crmd[4050]:   notice: process_lrm_event: LRM
> operation p-drbd_notify_0 (call=54, rc=0, cib-update=0, confirmed=true) ok
> Jul  5 01:37:35 node2 stonith-ng[4046]:   notice:
> can_fence_host_with_device: sbd_stonith can fence node1: dynamic-list
> Jul  5 01:37:35 node2 sbd: [6197]: info: Delivery process handling
> /dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382dfbd
> Jul  5 01:37:35 node2 sbd: [6197]: info: Device UUID:
> 4e398182-4894-4f11-b84e-6c6ad4c02614
> Jul  5 01:37:35 node2 sbd: [6197]: info: Writing reset to node slot node1
> Jul  5 01:37:35 node2 sbd: [6197]: info: Messaging delay: 40
> Jul  5 01:38:15 node2 sbd: [6197]: info: reset successfully delivered to
> node1
> Jul  5 01:38:15 node2 sbd: [6196]: info: Message successfully delivered.
> Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: log_operation:
> Operation 'reboot' [6185] (call 3 from crmd.4050) for host 'node1' with
> device 'sbd_stonith' returned: 0 (OK)
> Jul  5 01:38:16 node2 stonith-ng[4046]:   notice: remote_op_done:
> Operation reboot of node1 by node2 for crmd.4050@node2.2fce4697: OK
> Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_callback:
> Stonith operation 3/79:6:0:34e9a1ca-ebc9-4114-b5b7-26e670dc6aa0: OK (0)
> Jul  5 01:38:16 node2 crmd[4050]:   notice: tengine_stonith_notify: Peer
> node1 was terminated (reboot) by node2 for node2: OK
> (ref=2fce4697-65d5-40d2-b5b4-5c10be4d3d5a) by client crmd.4050
> Jul  5 01:38:16 node2 crmd[4050]:   notice: te_rsc_command: Initiating
> action 96: notify p-drbd_post_notify_demote_0 on node2 (local)
> Jul  5 01:38:16 node2 crmd[4050]:   notice: process_lrm_event: LRM
> operation p-drbd_notify_0 (call=55, rc=0, cib-update=0, confirmed=true) ok
> Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 6
> (Complete=24, Pending=0, Fired=0, Skipped=6, Incomplete=5,
> Source=/var/lib/pacemaker/pengine/pe-warn-5.bz2): Stopped
> Jul  5 01:38:16 node2 pengine[4049]:   notice: unpack_config: On loss of
> CCM Quorum: Ignore
> Jul  5 01:38:16 node2 pengine[4049]:    error: rsc_expand_action:
> Couldn't expand cl-lock_demote_0
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:    error: crm_abort:
> clone_update_actions_interleave: Triggered assert at clone.c:1260 :
> first_action != NULL || is_set(first_child->flags, pe_rsc_orphan)
> Jul  5 01:38:16 node2 pengine[4049]:    error:
> clone_update_actions_interleave: No action found for demote in g-lock:0
> (first)
> Jul  5 01:38:16 node2 pengine[4049]:   notice: process_pe_message:
> Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-30.bz2
> Jul  5 01:38:16 node2 crmd[4050]:   notice: do_te_invoke: Processing
> graph 7 (ref=pe_calc-dc-1436042296-79) derived from
> /var/lib/pacemaker/pengine/pe-input-30.bz2
> Jul  5 01:38:16 node2 crmd[4050]:   notice: run_graph: Transition 7
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Complete
> Jul  5 01:38:16 node2 crmd[4050]:   notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jul  5 01:38:17 node2 cluster-dlm[4344]: fence_node_time: Node
> 739307907/node1 was last shot 'now'
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing 739307907 done add
> 1436042043 fail 1436042255 last 1436042297
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fencing_done:
> 12925001B70F4A529491DFF16F8B8352 check_fencing done
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_quorum_done:
> 12925001B70F4A529491DFF16F8B8352 check_quorum disabled
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
> 12925001B70F4A529491DFF16F8B8352 check_fs nodeid 739307907 clear
> Jul  5 01:38:17 node2 cluster-dlm[4344]: check_fs_done:
> 12925001B70F4A529491DFF16F8B8352 check_fs done
> Jul  5 01:38:17 node2 cluster-dlm[4344]: send_info:
> 12925001B70F4A529491DFF16F8B8352 send_start cg 4 flags 2 data2 0 counts
> 3 1 0 1 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: receive_start:
> 12925001B70F4A529491DFF16F8B8352 receive_start 739307908:4 len 76
> Jul  5 01:38:17 node2 cluster-dlm[4344]: match_change:
> 12925001B70F4A529491DFF16F8B8352 match_change 739307908:4 matches cg 4
> Jul  5 01:38:17 node2 cluster-dlm[4344]: wait_messages_done:
> 12925001B70F4A529491DFF16F8B8352 wait_messages cg 4 got all 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: start_kernel:
> 12925001B70F4A529491DFF16F8B8352 start_kernel cg 4 member_count 1
> Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
> 739307907
> Jul  5 01:38:17 node2 cluster-dlm[4344]: update_dir_members: dir_member
> 739307908
> Jul  5 01:38:17 node2 cluster-dlm[4344]: set_configfs_members:
> set_members rmdir
> "/sys/kernel/config/dlm/cluster/spaces/12925001B70F4A529491DFF16F8B8352/node
> s/739307907"
> Jul  5 01:38:17 node2 cluster-dlm[4344]: do_sysfs: write "1" to
> "/sys/kernel/dlm/12925001B70F4A529491DFF16F8B8352/control"
> Jul  5 01:38:17 node2 kernel: [  726.576130]
> (ocfs2rec,6152,0):ocfs2_replay_journal:1549 Recovering node 739307907
> from slot 0 on device (253,0)
> Jul  5 01:38:18 node2 kernel: [  727.453597]
> (ocfs2rec,6152,0):ocfs2_begin_quota_recovery:407 Beginning quota
> recovery in slot 0
> Jul  5 01:38:18 node2 kernel: [  727.455965]
> (kworker/u:28,139,0):ocfs2_finish_quota_recovery:599 Finishing quota
> recovery in slot 0
>
>
> cib:
> primitive p-dlm ocf:pacemaker:controld \
>          op monitor interval="120" timeout="30" \
>          op start interval="0" timeout="90" \
>          op stop interval="0" timeout="100"
> primitive p-drbd ocf:linbit:drbd \
>          params drbd_resource="r0" \
>          op monitor interval="50" role="Master" timeout="30" \
>          op monitor interval="60" role="Slave" timeout="30" \
>          op start interval="0" timeout="240" \
>          op stop interval="0" timeout="100"
> primitive p-o2cb ocf:ocfs2:o2cb \
>          op monitor interval="120" timeout="30" \
>          op start interval="0" timeout="90" \
>          op stop interval="0" timeout="100"
> primitive p-sharefs ocf:heartbeat:Filesystem \
>          params device="/dev/vg1/sharelv" directory="/sharedata"
> fstype="ocfs2" \
>          op monitor interval="60s" timeout="60s" \
>          op start interval="0" timeout="90s" \
>          op stop interval="0" timeout="90s"
> primitive p-vg1 ocf:heartbeat:LVM \
>          params volgrpname="vg1" \
>          op monitor interval="60s" timeout="40" \
>          op start interval="0" timeout="40" \
>          op stop interval="0" timeout="40"
> primitive sbd_stonith stonith:external/sbd \
>          meta target-role="Started" \
>          op monitor interval="3000" timeout="20" \
>          op start interval="0" timeout="20" \
>          op stop interval="0" timeout="20" \
>          params
> sbd_device="/dev/disk/by-id/scsi-1494554000000000035c71550e83586b70dfc77c3b382
> dfbd"
> group g-lock p-dlm p-o2cb
> group g-sharedata p-vg1 p-sharefs
> ms ms-drbd p-drbd \
>          meta master-max="2" clone-max="2" notify="true" interleave="true"
> clone cl-lock g-lock \
>          meta globally-unique="false" interleave="true"
> clone cl-sharedata g-sharedata \
>          meta globally-unique="false" interleave="true"
> colocation col-drbd-lock inf: cl-lock ms-drbd:Master
> colocation col-lock-sharedata inf: cl-sharedata cl-lock
> order ord-drbd-lock inf: ms-drbd:promote cl-lock
> order ord-lock-sharedata inf: cl-lock cl-sharedata
>          stonith-timeout="90s" \
>          no-quorum-policy="ignore" \
>          expected-quorum-votes="2"
>
>
> --
> Regards,
>
> Muhammad Sharfuddin
> _______________________________________________
> Linux-HA mailing list is closing down.
> Please subscribe to [hidden email] instead.
> http://clusterlabs.org/mailman/listinfo/users 
> _______________________________________________
> [hidden email]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha 



_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: file system resource becomes inaccesible when any of the node goes down

Dejan Muhamedagic
In reply to this post by Muhammad Sharfuddin-2
Hi,

On Sun, Jul 05, 2015 at 09:13:56PM +0500, Muhammad Sharfuddin wrote:

> SLES 11 SP 3 + online updates(pacemaker-1.1.11-0.8.11.70
> openais-1.1.4-5.22.1.7)
>
> Its a dual primary drbd cluster, which mounts a file system resource
> on both the cluster nodes simultaneously(file system type is ocfs2).
>
> Whenever any of the nodes goes down, the file system(/sharedata)
> become inaccessible for exact 35 seconds on the other
> (surviving/online) node, and then become available again on the
> online node.
>
> Please help me understand why the node which survives or remains
> online unable to access the file system resource(/sharedata) for 35
> seconds ? and how can I fix the cluster so that file system remains
> accessible on the surviving node without any interruption/delay(as
> in my case of about 35 seconds)
>
> By inaccessible, I meant to say that running "ls -l /sharedata" and
> "df /sharedata" does not return any output and does not return the
> prompt back on the online node for exact 35 seconds once the other
> node becomes offline.
>
> e.g "node1" got offline somewhere around  01:37:15, and then
> /sharedata file system was inaccessible during 01:37:35 and 01:38:18
> on the online node i.e "node2".

Before the failing node gets fenced you won't be able to use the
ocfs2 filesystem. In this case, the fencing operation takes 40
seconds:

> [...]
> Jul  5 01:37:35 node2 sbd: [6197]: info: Writing reset to node slot node1
> Jul  5 01:37:35 node2 sbd: [6197]: info: Messaging delay: 40
> Jul  5 01:38:15 node2 sbd: [6197]: info: reset successfully
> delivered to node1
> Jul  5 01:38:15 node2 sbd: [6196]: info: Message successfully delivered.
> [...]

You may want to reduce that sbd timeout.

Thanks,

Dejan
_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: file system resource becomes inaccesible when any of the node goes down

Muhammad Sharfuddin-2
On 07/06/2015 02:50 PM, Dejan Muhamedagic wrote:

> Hi,
>
> On Sun, Jul 05, 2015 at 09:13:56PM +0500, Muhammad Sharfuddin wrote:
>> SLES 11 SP 3 + online updates(pacemaker-1.1.11-0.8.11.70
>> openais-1.1.4-5.22.1.7)
>>
>> Its a dual primary drbd cluster, which mounts a file system resource
>> on both the cluster nodes simultaneously(file system type is ocfs2).
>>
>> Whenever any of the nodes goes down, the file system(/sharedata)
>> become inaccessible for exact 35 seconds on the other
>> (surviving/online) node, and then become available again on the
>> online node.
>>
>> Please help me understand why the node which survives or remains
>> online unable to access the file system resource(/sharedata) for 35
>> seconds ? and how can I fix the cluster so that file system remains
>> accessible on the surviving node without any interruption/delay(as
>> in my case of about 35 seconds)
>>
>> By inaccessible, I meant to say that running "ls -l /sharedata" and
>> "df /sharedata" does not return any output and does not return the
>> prompt back on the online node for exact 35 seconds once the other
>> node becomes offline.
>>
>> e.g "node1" got offline somewhere around  01:37:15, and then
>> /sharedata file system was inaccessible during 01:37:35 and 01:38:18
>> on the online node i.e "node2".
> Before the failing node gets fenced you won't be able to use the
> ocfs2 filesystem. In this case, the fencing operation takes 40
> seconds:
so its expected.
>> [...]
>> Jul  5 01:37:35 node2 sbd: [6197]: info: Writing reset to node slot node1
>> Jul  5 01:37:35 node2 sbd: [6197]: info: Messaging delay: 40
>> Jul  5 01:38:15 node2 sbd: [6197]: info: reset successfully
>> delivered to node1
>> Jul  5 01:38:15 node2 sbd: [6196]: info: Message successfully delivered.
>> [...]
> You may want to reduce that sbd timeout.
Ok, so reducing the sbd timeout(or msgwait) would provide the
uninterrupted access to the ocfs2 file system on the surviving/online node ?
or would it just minimize the downtime ?

>
> Thanks,
>
> Dejan
> _______________________________________________
> Linux-HA mailing list is closing down.
> Please subscribe to [hidden email] instead.
> http://clusterlabs.org/mailman/listinfo/users
> _______________________________________________
> [hidden email]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>
--
Regards,

Muhammad Sharfuddin
_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Antw: Re: file system resource becomes inaccesible when any of the node goes down

Ulrich Windl
>>> Muhammad Sharfuddin <[hidden email]> schrieb am 06.07.2015 um 12:14 in
Nachricht <[hidden email]>:
[...]
> Ok, so reducing the sbd timeout(or msgwait) would provide the
> uninterrupted access to the ocfs2 file system on the surviving/online node ?
> or would it just minimize the downtime ?

It will reduce the time between "writing the reset message for a node" and "the cluster believes the node is down". So you can guess what happens if you set it to some very short time like 1 second...

Regards,
Ulrich



_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: Antw: Re: file system resource becomes inaccesible when any of the node goes down

Muhammad Sharfuddin-2
On 07/07/2015 12:14 PM, Ulrich Windl wrote:

>>>> Muhammad Sharfuddin <[hidden email]> schrieb am 06.07.2015 um 12:14 in
> Nachricht <[hidden email]>:
> [...]
>> Ok, so reducing the sbd timeout(or msgwait) would provide the
>> uninterrupted access to the ocfs2 file system on the surviving/online node ?
>> or would it just minimize the downtime ?
> It will reduce the time between "writing the reset message for a node" and "the cluster believes the node is down". So you can guess what happens if you set it to some very short time like 1 second...
>
> Regards,
> Ulrich
>
now msgwait timeout is set to 10s and a delay/inaccessibility of 15
seconds was observed. If a service(App, DB, file server) is installed
and running from the ocfs2 file system via the surviving/online node, then
wouldn't that service get crashed or become offline due to the
inaccessibility of the file system(event though its ocfs2) while a
member node goes down ?


If cluster is configured to run the two independent services, and starts
one on node1 and ther on node2, while both the service shared the same
file system, /sharedata(ocfs2),  then in case of a failure of one node,
the other/online wont be able to
keep running the particular service because the file system holding the
binaries/configuration/service is not available for around at least 15
seconds.

I don't understand the advantage of Ocfs2 file system in such a setup.


--
Regards,

Muhammad Sharfuddin
<http://www.nds.com.pk>
_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Antw: Re: Antw: Re: file system resource becomes inaccesible when any of the node goes down

Ulrich Windl
>>> Muhammad Sharfuddin <[hidden email]> schrieb am 07.07.2015 um 11:15 in
Nachricht <[hidden email]>:

[...]
> I don't understand the advantage of Ocfs2 file system in such a setup.

The advantage depends on the alternatives: If two nodes both want to access the same filesystem, you can use OCFS2, NFS, or CIFS (list not complete). If only one node can access a filesystem, you could try any journaled filesystem (a fsck is needed after a node crash).

If you use NFS or CIFS, you shift the problem to another machine.

If you use a local filesystem, you need recovery, mount, and start of your application on a standby node.

With OCFS2 you'll have to wait for a few seconds before your application can continue.

Note that a usual SCSI disk with read-errors can also delay I/O significantly while retrying I/O.

Regards,
Ulrich


_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: Antw: Re: file system resource becomes inaccesible when any of the node goes down

Lars Marowsky-Bree-3
In reply to this post by Muhammad Sharfuddin-2
On 2015-07-07T14:15:14, Muhammad Sharfuddin <[hidden email]> wrote:

> now msgwait timeout is set to 10s and a delay/inaccessibility of 15 seconds
> was observed. If a service(App, DB, file server) is installed and running
> from the ocfs2 file system via the surviving/online node, then
> wouldn't that service get crashed or become offline due to the
> inaccessibility of the file system(event though its ocfs2) while a member
> node goes down ?

You're seeing a trade-off of using OCFS2. The semantics of the file
system require all accessing nodes to be very closely synchronized (that
is not optional), and that requires the access to the fs to be paused
during recovery. (See the CAP theorem.)

The apps don't crash, they are simply blocked. (To them it looks like
slow IO.)

The same is true for DRBD in active/active mode; the block device is
tightly synchronized, and this requires both nodes to be up, or cleanly
reported as down.

> If cluster is configured to run the two independent services, and starts one
> on node1 and ther on node2, while both the service shared the same file
> system, /sharedata(ocfs2),  then in case of a failure of one node, the
> other/online wont be able to
> keep running the particular service because the file system holding the
> binaries/configuration/service is not available for around at least 15
> seconds.
>
> I don't understand the advantage of Ocfs2 file system in such a setup.

If that's your setup, indeed, you're not getting any advantages. OCFS2
makes sense if you have services that indeed need access to the same
file system and directory structure.

If you have two independent services, or even services that are
essentially node local, you're much better off using independent,
separate file system mounts with XFS or extX.



Regards,
    Lars

--
Architect Storage/HA
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: Antw: Re: Antw: Re: file system resource becomes inaccesible when any of the node goes down

Lars Marowsky-Bree-3
In reply to this post by Ulrich Windl
On 2015-07-07T12:23:44, Ulrich Windl <[hidden email]> wrote:

> The advantage depends on the alternatives: If two nodes both want to access the same filesystem, you can use OCFS2, NFS, or CIFS (list not complete). If only one node can access a filesystem, you could try any journaled filesystem (a fsck is needed after a node crash).

A journaled file system does not require a fsck after a crash.

> If you use NFS or CIFS, you shift the problem to another machine.
>
> If you use a local filesystem, you need recovery, mount, and start of your application on a standby node.
>
> With OCFS2 you'll have to wait for a few seconds before your application can continue.

The recovery happens in the background with OCFS2 as well; the fs
replays the failed node's journal in the background. The actual time
saved by avoiding the "mount" is negligible.



Regards,
    Lars

--
Architect Storage/HA
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
Reply | Threaded
Open this post in threaded view
|

Re: Antw: Re: Antw: Re: file system resource becomes inaccesible when any of the node goes down

Uemit Arslan

Hi Lars,

bist Du Dir hier sicher?
 
" After a crash the filesystem can very quickly be brought back on-line using the journal log reducing what could take
minutes using fsck to seconds with the added advantage that there is considerably less chance of data loss or
corruption."



Gruß
Ümit
>>> Lars Marowsky-Bree <[hidden email]> 7/9/2015 10:57 AM >>>
A journaled file system does not require a fsck after a crash.

_______________________________________________
Linux-HA mailing list is closing down.
Please subscribe to [hidden email] instead.
http://clusterlabs.org/mailman/listinfo/users
_______________________________________________
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha