Unable to start any node of pgsql Master/Slave Cluster

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to start any node of pgsql Master/Slave Cluster

Oliver Weichhold
I'm currently a bit struggling with setting up a PostgreSQL 9.3 HA Master/Slave Cluster using CentOS 7 (Corosync 2 and Pacemaker 1.1.10).

Please note that I've deliberately stopped node2 currently in order to keep the scenario simpler (I hope).

After starting the cluster on node1, crm_mon shows the following:

Stack: corosync
Current DC: node1 (1) - partition WITHOUT quorum
Version: 1.1.10-32.el7_0-368c726
2 Nodes configured
4 Resources configured

Online: [ node1 ]
OFFLINE: [ node2 ]

Full list of resources:

 Master/Slave Set: pgsql_master_slave [pgsql]
     Stopped: [ node1 node2 ]
 Resource Group: master-group
     pgsql_vip_rep      (ocf::heartbeat:IPaddr2):       Stopped
     pgsql_forward_listen_port  (ocf::heartbeat:portforward):   Stopped

Node Attributes:
* Node node1:
    + master-pgsql                      : -INFINITY
    + pgsql-status                      : STOP

Migration summary:
* Node node1:
   pgsql: migration-threshold=1 fail-count=1000000 last-failure='Thu Sep 18 11:39:06 2014'

Failed actions:
    pgsql_start_0 on node1 'unknown error' (1): call=15, status=Timed Out, last-rc-change='Thu Sep 18 11:38:05 2014', qu
eued=60028ms, exec=0ms

After running the following commands:

rm -f /var/lib/pgsql/9.3/data/recovery.conf
rm -f /var/lib/pgsql/9.3/data/ra_tmp/PGSQL.lock

I've verified that postgres can be started manually using:

systemctl start postgresql-9.3

Which is not the point, of course. But I wanted to at least make sure that the pgsql configuration is not totally hosed.

I then tried the following on node1 (node2 is still switched off as mentioned before):

rm -f /var/lib/pgsql/9.3/data/recovery.conf
rm -f /var/lib/pgsql/9.3/data/ra_tmp/PGSQL.lock
crm_attribute -l reboot -N $(uname -n) -n "pgsql-data-status" -v "LATEST"
crm_attribute -l reboot -N $(uname -n) -n "master-pgsql" -v "1000"
pcs resource cleanup pgsql
pcs resource cleanup pgsql_master_slave
pcs resource cleanup master-group

Which has the effect to briefly change crm_mon output to this:

Online: [ node1 ]
OFFLINE: [ node2 ]

Full list of resources:

 Master/Slave Set: pgsql_master_slave [pgsql]
     Stopped: [ node1 node2 ]
 Resource Group: master-group
     pgsql_vip_rep      (ocf::heartbeat:IPaddr2):       Stopped
     pgsql_forward_listen_port  (ocf::heartbeat:portforward):   Stopped

Node Attributes:
* Node node1:
    + master-pgsql                      : 1000
    + pgsql-data-status                 : LATEST
    + pgsql-status                      : STOP

Migration summary:
* Node node1:

As soon as the last resource cleanup is done, the situation reverts to the first "picture". I'm really running out of ideas here. Any suggestions?
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Oliver Weichhold
CIB Dump:

<cib admin_epoch="50" epoch="73" num_updates="7" validate-with="pacemaker-1.2" cib-last-written="Thu Sep 18 11:50:17 2014" update-origin="node1" update-client="crmd" crm_feature_set="3.0.7" have-quorum="0" dc-uuid="1">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
        <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-32.el7_0-368c726"/>
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
        <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1411055417"/>
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="1" uname="node1"/>
      <node id="2" uname="node2"/>
    </nodes>
    <resources>
      <master id="pgsql_master_slave">
        <primitive class="ocf" id="pgsql" provider="heartbeat" type="pgsql">
          <instance_attributes id="pgsql-instance_attributes">
            <nvpair id="pgsql-instance_attributes-pgctl" name="pgctl" value="/usr/pgsql-9.3/bin/pg_ctl"/>
            <nvpair id="pgsql-instance_attributes-psql" name="psql" value="/usr/pgsql-9.3/bin/psql"/>
            <nvpair id="pgsql-instance_attributes-pgdata" name="pgdata" value="/var/lib/pgsql/9.3/data/"/>
            <nvpair id="pgsql-instance_attributes-rep_mode" name="rep_mode" value="async"/>
            <nvpair id="pgsql-instance_attributes-node_list" name="node_list" value="node1 node2"/>
            <nvpair id="pgsql-instance_attributes-tmpdir" name="tmpdir" value="/var/lib/pgsql/9.3/data/ra_tmp"/>
            <nvpair id="pgsql-instance_attributes-restore_command" name="restore_command" value="cp /var/lib/pgsql/9.3/data/pg_archive/%f %p"/>
            <nvpair id="pgsql-instance_attributes-primary_conninfo_opt" name="primary_conninfo_opt" value="keepalives_idle=60 keepalives_interval=5 keepalives_count=5"/>
            <nvpair id="pgsql-instance_attributes-master_ip" name="master_ip" value="10.1.1.7"/>
            <nvpair id="pgsql-instance_attributes-restart_on_promote" name="restart_on_promote" value="true"/>
          </instance_attributes>
          <operations>
            <op id="pgsql-start-interval-0s" interval="0s" name="start" on-fail="restart" timeout="60s"/>
            <op id="pgsql-monitor-interval-4s" interval="4s" name="monitor" on-fail="restart" timeout="60s"/>
            <op id="pgsql-monitor-interval-3s" interval="3s" name="monitor" on-fail="restart" role="Master" timeout="60s"/>
            <op id="pgsql-promote-interval-0s" interval="0s" name="promote" on-fail="restart" timeout="60s"/>
            <op id="pgsql-demote-interval-0s" interval="0s" name="demote" on-fail="stop" timeout="60s"/>
            <op id="pgsql-stop-interval-0s" interval="0s" name="stop" on-fail="block" timeout="60s"/>
          </operations>
        </primitive>
        <meta_attributes id="pgsql_master_slave-meta_attributes">
          <nvpair id="pgsql_master_slave-meta_attributes-master-max" name="master-max" value="1"/>
          <nvpair id="pgsql_master_slave-meta_attributes-master-node-max" name="master-node-max" value="1"/>
          <nvpair id="pgsql_master_slave-meta_attributes-clone-max" name="clone-max" value="2"/>
          <nvpair id="pgsql_master_slave-meta_attributes-clone-node-max" name="clone-node-max" value="1"/>
          <nvpair id="pgsql_master_slave-meta_attributes-notify" name="notify" value="true"/>
        </meta_attributes>
      </master>
      <group id="master-group">
        <primitive class="ocf" id="pgsql_vip_rep" provider="heartbeat" type="IPaddr2">
          <instance_attributes id="pgsql_vip_rep-instance_attributes">
            <nvpair id="pgsql_vip_rep-instance_attributes-ip" name="ip" value="10.1.1.7"/>
            <nvpair id="pgsql_vip_rep-instance_attributes-nic" name="nic" value="tun0"/>
            <nvpair id="pgsql_vip_rep-instance_attributes-cidr_netmask" name="cidr_netmask" value="24"/>
          </instance_attributes>
          <operations>
            <op id="pgsql_vip_rep-start-interval-0s" interval="0s" name="start" on-fail="stop" timeout="60s"/>
            <op id="pgsql_vip_rep-monitor-interval-10s" interval="10s" name="monitor" on-fail="restart" timeout="60s"/>
            <op id="pgsql_vip_rep-stop-interval-0s" interval="0s" name="stop" on-fail="ignore" timeout="60s"/>
          </operations>
          <meta_attributes id="pgsql_vip_rep-meta_attributes">
            <nvpair id="pgsql_vip_rep-meta_attributes-migration-threshold" name="migration-threshold" value="0"/>
          </meta_attributes>
        </primitive>
        <primitive class="ocf" id="pgsql_forward_listen_port" provider="heartbeat" type="portforward">
          <instance_attributes id="pgsql_forward_listen_port-instance_attributes">
            <nvpair id="pgsql_forward_listen_port-instance_attributes-srcport" name="srcport" value="5433"/>
            <nvpair id="pgsql_forward_listen_port-instance_attributes-dstport" name="dstport" value="5432"/>
            <nvpair id="pgsql_forward_listen_port-instance_attributes-protocol" name="protocol" value="tcp"/>
          </instance_attributes>
          <operations>
            <op id="pgsql_forward_listen_port-start-timeout-20" interval="0s" name="start" timeout="20"/>
            <op id="pgsql_forward_listen_port-stop-timeout-20" interval="0s" name="stop" timeout="20"/>
            <op id="pgsql_forward_listen_port-monitor-interval-5s" interval="5s" name="monitor"/>
          </operations>
        </primitive>
      </group>
    </resources>
    <constraints>
      <rsc_colocation id="colocation-master-group-pgsql_master_slave-INFINITY" rsc="master-group" rsc-role="Started" score="INFINITY" with-rsc="pgsql_master_slave" with-rsc-role="Master"/>
      <rsc_order first="pgsql_master_slave" first-action="promote" id="order-pgsql_master_slave-master-group-INFINITY" score="INFINITY" symmetrical="false" then="master-group" then-action="start"/>
      <rsc_order first="pgsql_master_slave" first-action="demote" id="order-pgsql_master_slave-master-group-0" score="0" symmetrical="false" then="master-group" then-action="stop"/>
    </constraints>
    <rsc_defaults>
      <meta_attributes id="rsc_defaults-options">
        <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="INFINITY"/>
        <nvpair id="rsc_defaults-options-migration-threshold" name="migration-threshold" value="1"/>
      </meta_attributes>
    </rsc_defaults>
  </configuration>
  <status>
    <node_state id="1" uname="node1" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
      <lrm id="1">
        <lrm_resources>
          <lrm_resource id="pgsql_forward_listen_port" type="portforward" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="pgsql_forward_listen_port_last_0" operation_key="pgsql_forward_listen_port_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="6:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" transition-magic="0:7;6:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" call-id="78" rc-code="7" op-status="0" interval="0" last-run="1411055417" last-rc-change="1411055417" exec-time="137" queue-time="0" op-digest="8efe4e9ed1afd487e598578eef7352df"/>
          </lrm_resource>
          <lrm_resource id="pgsql_vip_rep" type="IPaddr2" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="pgsql_vip_rep_last_0" operation_key="pgsql_vip_rep_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="5:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" transition-magic="0:7;5:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" call-id="74" rc-code="7" op-status="0" interval="0" last-run="1411055417" last-rc-change="1411055417" exec-time="326" queue-time="0" op-digest="9d672e70b2c9726a4cb571becf2b04b4"/>
          </lrm_resource>
          <lrm_resource id="pgsql" type="pgsql" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="pgsql_last_failure_0" operation_key="pgsql_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="4:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" transition-magic="0:1;4:13:7:2b32876f-fe95-470b-b770-9c34a79944e9" call-id="70" rc-code="1" op-status="0" interval="0" last-run="1411055417" last-rc-change="1411055417" exec-time="526" queue-time="0" op-digest="489ca265de0844a902bea5974ac07875"/>
            <lrm_rsc_op id="pgsql_last_0" operation_key="pgsql_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="1:14:0:2b32876f-fe95-470b-b770-9c34a79944e9" transition-magic="0:0;1:14:0:2b32876f-fe95-470b-b770-9c34a79944e9" call-id="80" rc-code="0" op-status="0" interval="0" last-run="1411055418" last-rc-change="1411055418" exec-time="3580" queue-time="0" op-digest="489ca265de0844a902bea5974ac07875"/>
          </lrm_resource>
        </lrm_resources>
      </lrm>
      <transient_attributes id="1">
        <instance_attributes id="status-1">
          <nvpair id="status-1-probe_complete" name="probe_complete" value="true"/>
          <nvpair id="status-1-pgsql-status" name="pgsql-status" value="STOP"/>
          <nvpair id="status-1-master-pgsql" name="master-pgsql" value="-INFINITY"/>
          <nvpair id="status-1-last-failure-pgsql" name="last-failure-pgsql" value="1411055416"/>
          <nvpair id="status-1-pgsql-data-status" name="pgsql-data-status" value="LATEST"/>
          <nvpair id="status-1-fail-count-pgsql" name="fail-count-pgsql" value="INFINITY"/>
        </instance_attributes>
      </transient_attributes>
    </node_state>
  </status>
</cib>
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Oliver Weichhold
In reply to this post by Oliver Weichhold
I've monitored the cluster log and the running processes while the script containing

rm -f /var/lib/pgsql/9.3/data/recovery.conf
rm -f /var/lib/pgsql/9.3/data/ra_tmp/PGSQL.lock
crm_attribute -l reboot -N $(uname -n) -n "pgsql-data-status" -v "LATEST"
crm_attribute -l reboot -N $(uname -n) -n "master-pgsql" -v "1000"
pcs resource cleanup pgsql
pcs resource cleanup pgsql_master_slave
pcs resource cleanup master-group

is running and interestingly the resource agent places a recovery.conf into the the data directory which seems to put PostgreSQL into recovery mode as indicated by this:

postgres 61166  1.0  2.1 336488 14664 ?        S    13:35   0:00 /usr/pgsql-9.3/bin/postgres -D /var/lib/pgsql/9.3/data
postgres 61192  0.0  0.1 190072  1296 ?        Ss   13:35   0:00 postgres: logger process
postgres 61193  0.0  0.3 336608  2076 ?        Ss   13:35   0:00 postgres: startup process   recovering 0000000100000000
postgres 61208  0.0  0.2 336488  1684 ?        Ss   13:35   0:00 postgres: checkpointer process
postgres 61209  0.0  0.2 336488  1692 ?        Ss   13:35   0:00 postgres: writer process
postgres 61210  0.0  0.4 343332  2916 ?        Ss   13:35   0:00 postgres: wal receiver process

which in turn seems to cause a timeout in the resource agent waiting for PostgreSQL to start:

pgsql(pgsql)[61018]:    2014/09/18_13:36:05 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:05 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:06 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:06 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:07 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:07 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:09 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:09 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:10 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:10 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:11 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:11 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:12 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:12 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:13 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:13 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:14 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:14 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:16 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:16 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:17 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:17 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:18 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:18 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:19 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:19 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:20 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:20 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:21 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:21 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:22 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:22 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:24 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:24 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:25 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:25 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:26 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:26 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:27 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:27 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:28 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:28 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:29 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:29 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:30 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:31 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:32 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:32 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:33 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:33 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:34 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:34 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:35 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:35 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:36 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:36 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:37 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:37 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:39 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:39 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:40 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:40 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
pgsql(pgsql)[61018]:    2014/09/18_13:36:41 WARNING: PostgreSQL template1 isn't running
pgsql(pgsql)[61018]:    2014/09/18_13:36:41 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
Sep 18 13:36:41 [51045] node1       lrmd:  warning: child_timeout_callback:     pgsql_start_0 process (PID 61018) timed out
Sep 18 13:36:41 [51045] node1       lrmd:  warning: operation_finished:         pgsql_start_0:61018 - timed out after 60000ms
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:   notice: operation_finished:         pgsql_start_0:61018:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:41 [51045] node1       lrmd:     info: log_finished:       finished - rsc:pgsql action:start call_id:160 pid:61018 exit-code:1 exec-time:60037ms queue-time:0ms
Sep 18 13:36:41 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql' not found (2 active resources)
Sep 18 13:36:41 [51048] node1       crmd:    error: process_lrm_event:  LRM operation pgsql_start_0 (160) Timed Out (timeout=60000ms)
Sep 18 13:36:41 [51048] node1       crmd:  warning: do_update_resource:         Resource pgsql no longer exists in the lrmd
Sep 18 13:36:41 [51048] node1       crmd:  warning: status_from_rc:     Action 6 (pgsql_start_0) on node1 failed (target: 0 vs. rc: 1): Error
Sep 18 13:36:41 [51048] node1       crmd:  warning: update_failcount:   Updating failcount for pgsql on node1 after failed start: rc=1 (update=INFINITY, time=1411061801)
Sep 18 13:36:41 [51048] node1       crmd:     info: abort_transition_graph:     match_graph_event:313 - Triggered transition abort (complete=0, node=node1, tag=lrm_rsc_op, id=pgsql_last_failure_0, magic=2:1;6:40:0:2b32876f-fe95-470b-b770-9c34a79944e9) : Event failed
Sep 18 13:36:41 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_start_0 (6) confirmed on node1 (rc=4)
Sep 18 13:36:41 [51048] node1       crmd:  warning: update_failcount:   Updating failcount for pgsql on node1 after failed start: rc=1 (update=INFINITY, time=1411061801)
Sep 18 13:36:41 [51048] node1       crmd:     info: process_graph_event:        Detected action (40.6) pgsql_start_0.160=unknown error: failed
Sep 18 13:36:41 [51048] node1       crmd:   notice: process_lrm_event:  node1-pgsql_start_0:160 [ psql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\npsql: FATAL:
Sep 18 13:36:41 [51048] node1       crmd:     info: process_lrm_event:  Deletion of resource 'pgsql' complete after pgsql_start_0
Sep 18 13:36:41 [51048] node1       crmd:     info: notify_deleted:     Notifying f943b2e1-e9bd-435f-a8c6-31d28b7ba2af on node1 that pgsql was deleted
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_trigger_update:       Sending flush op to all hosts for: fail-count-pgsql (INFINITY)
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql']: OK (rc=0, origin=local/crmd/208, version=50.81.9)
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-pgsql']: No such device or address (rc=-6, origin=local/attrd/105, version=50.81.9)
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_perform_update:       Sent update 107: fail-count-pgsql=INFINITY
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_trigger_update:       Sending flush op to all hosts for: last-failure-pgsql (1411061801)
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-pgsql']: OK (rc=0, origin=local/attrd/108, version=50.81.10)
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_perform_update:       Sent update 109: last-failure-pgsql=1411061801
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_trigger_update:       Sending flush op to all hosts for: fail-count-pgsql (INFINITY)
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-pgsql']: OK (rc=0, origin=local/attrd/110, version=50.81.11)
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_perform_update:       Sent update 111: fail-count-pgsql=INFINITY
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_trigger_update:       Sending flush op to all hosts for: last-failure-pgsql (1411061801)
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-pgsql']: OK (rc=0, origin=local/attrd/112, version=50.81.11)
Sep 18 13:36:41 [51046] node1      attrd:   notice: attrd_perform_update:       Sent update 113: last-failure-pgsql=1411061801
Sep 18 13:36:41 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:258 - Triggered transition abort (complete=0, node=node1, tag=lrm_rsc_op, id=pgsql_last_0, magic=0:7;4:39:7:2b32876f-fe95-470b-b770-9c34a79944e9, cib=50.81.9) : Resource op removal
Sep 18 13:36:41 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:172 - Triggered transition abort (complete=0, node=node1, tag=nvpair, id=status-1-fail-count-pgsql, name=fail-count-pgsql, value=INFINITY, magic=NA, cib=50.81.10) : Transient attribute: update
Sep 18 13:36:41 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:172 - Triggered transition abort (complete=0, node=node1, tag=nvpair, id=status-1-last-failure-pgsql, name=last-failure-pgsql, value=1411061801, magic=NA, cib=50.81.11) : Transient attribute: update
Sep 18 13:36:41 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 46: notify pgsql_post_notify_start_0 on node1 (local)
Sep 18 13:36:41 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql' not found (2 active resources)
Sep 18 13:36:41 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql:0' not found (2 active resources)
Sep 18 13:36:41 [51045] node1       lrmd:     info: process_lrmd_rsc_register:  Added 'pgsql' to the rsc list (3 active resources)
Sep 18 13:36:41 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=46:40:0:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_notify_0
Sep 18 13:36:41 [51045] node1       lrmd:     info: log_execute:        executing - rsc:pgsql action:notify call_id:167
Sep 18 13:36:41 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql']: OK (rc=0, origin=local/crmd/211, version=50.82.1)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql:0' not found (3 active resources)
Sep 18 13:36:42 [51048] node1       crmd:     info: notify_deleted:     Notifying f4bfc05a-c58c-481b-ab96-0155c3f4f372 on node1 that pgsql:0 was deleted
Sep 18 13:36:42 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql:0']: OK (rc=0, origin=local/crmd/213, version=50.82.1)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql:1' not found (3 active resources)
Sep 18 13:36:42 [51048] node1       crmd:     info: notify_deleted:     Notifying f4bfc05a-c58c-481b-ab96-0155c3f4f372 on node1 that pgsql:1 was deleted
Sep 18 13:36:42 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql:1']: OK (rc=0, origin=local/crmd/216, version=50.83.1)
Sep 18 13:36:42 [51048] node1       crmd:     info: delete_resource:    Removing resource pgsql_vip_rep for ae8b028f-ff93-4228-afa2-08647279ef05 (internal) on node1
Sep 18 13:36:42 [51048] node1       crmd:     info: notify_deleted:     Notifying ae8b028f-ff93-4228-afa2-08647279ef05 on node1 that pgsql_vip_rep was deleted
Sep 18 13:36:42 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql_vip_rep']: OK (rc=0, origin=local/crmd/220, version=50.83.2)
Sep 18 13:36:42 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:258 - Triggered transition abort (complete=0, node=node1, tag=lrm_rsc_op, id=pgsql_vip_rep_last_0, magic=0:7;4:40:7:2b32876f-fe95-470b-b770-9c34a79944e9, cib=50.83.2) : Resource op removal
Sep 18 13:36:42 [51048] node1       crmd:     info: delete_resource:    Removing resource pgsql_forward_listen_port for ae8b028f-ff93-4228-afa2-08647279ef05 (internal) on node1
Sep 18 13:36:42 [51048] node1       crmd:     info: notify_deleted:     Notifying ae8b028f-ff93-4228-afa2-08647279ef05 on node1 that pgsql_forward_listen_port was deleted
Sep 18 13:36:42 [51043] node1        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='node1']//lrm_resource[@id='pgsql_forward_listen_port']: OK (rc=0, origin=local/crmd/223, version=50.83.3)
Sep 18 13:36:42 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:258 - Triggered transition abort (complete=0, node=node1, tag=lrm_rsc_op, id=pgsql_forward_listen_port_last_0, magic=0:7;5:40:7:2b32876f-fe95-470b-b770-9c34a79944e9, cib=50.83.3) : Resource op removal
Sep 18 13:36:42 [51045] node1       lrmd:     info: log_finished:       finished - rsc:pgsql action:notify call_id:167 pid:62615 exit-code:0 exec-time:783ms queue-time:0ms
Sep 18 13:36:42 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_notify_0 (46) confirmed on node1 (rc=0)
Sep 18 13:36:42 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_notify_0 (call=167, rc=0, cib-update=0, confirmed=true) ok
Sep 18 13:36:42 [51047] node1    pengine:     info: clone_print:         Master/Slave Set: pgsql_master_slave [pgsql]
Sep 18 13:36:42 [51047] node1    pengine:     info: native_print:            pgsql_vip_rep      (ocf::heartbeat:IPaddr2):       Stopped
Sep 18 13:36:42 [51047] node1    pengine:     info: native_print:            pgsql_forward_listen_port  (ocf::heartbeat:portforward):   Stopped
Sep 18 13:36:42 [51047] node1    pengine:     info: get_failcount_full:         pgsql_master_slave has failed INFINITY times on node1
Sep 18 13:36:42 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:42 [51047] node1    pengine:     info: get_failcount_full:         pgsql_master_slave has failed INFINITY times on node1
Sep 18 13:36:42 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:42 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:42 [51047] node1    pengine:     info: native_color:       Resource pgsql:0 cannot run anywhere
Sep 18 13:36:42 [51047] node1    pengine:     info: native_color:       Resource pgsql:1 cannot run anywhere
Sep 18 13:36:42 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:42 [51047] node1    pengine:     info: master_color:       pgsql_master_slave: Promoted 0 instances of a possible 1 to master
Sep 18 13:36:42 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_vip_rep: Rolling back scores from pgsql_forward_listen_port
Sep 18 13:36:42 [51047] node1    pengine:     info: native_color:       Resource pgsql_vip_rep cannot run anywhere
Sep 18 13:36:42 [51047] node1    pengine:     info: native_color:       Resource pgsql_forward_listen_port cannot run anywhere
Sep 18 13:36:42 [51047] node1    pengine:     info: LogActions:         Leave   pgsql:0 (Stopped)
Sep 18 13:36:42 [51047] node1    pengine:     info: LogActions:         Leave   pgsql:1 (Stopped)
Sep 18 13:36:42 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_vip_rep   (Stopped)
Sep 18 13:36:42 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_forward_listen_port       (Stopped)
Sep 18 13:36:42 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 4: monitor pgsql:0_monitor_0 on node1 (local)
Sep 18 13:36:42 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=4:41:7:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_monitor_0
Sep 18 13:36:42 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 5: monitor pgsql_vip_rep_monitor_0 on node1 (local)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql_vip_rep' not found (1 active resources)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_rsc_register:  Added 'pgsql_vip_rep' to the rsc list (2 active resources)
Sep 18 13:36:42 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=5:41:7:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_vip_rep_monitor_0
Sep 18 13:36:42 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 6: monitor pgsql_forward_listen_port_monitor_0 on node1 (local)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_get_rsc_info:  Resource 'pgsql_forward_listen_port' not found (2 active resources)
Sep 18 13:36:42 [51045] node1       lrmd:     info: process_lrmd_rsc_register:  Added 'pgsql_forward_listen_port' to the rsc list (3 active resources)
Sep 18 13:36:42 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=6:41:7:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_forward_listen_port_monitor_0
Sep 18 13:36:42 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_forward_listen_port_monitor_0 (call=180, rc=7, cib-update=227, confirmed=true) not running
Sep 18 13:36:42 [51048] node1       crmd:   notice: process_lrm_event:  node1-pgsql_forward_listen_port_monitor_0:180 [ portforward REDIRECT rule for OUTPUT chain [tcp 5433 5432] is inactive\n ]
Sep 18 13:36:42 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_forward_listen_port_monitor_0 (6) confirmed on node1 (rc=0)
pgsql(pgsql)[62677]:    2014/09/18_13:36:42 INFO: Don't check /var/lib/pgsql/9.3/data/ during probe
Sep 18 13:36:42 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_vip_rep_monitor_0 (call=176, rc=7, cib-update=228, confirmed=true) not running
Sep 18 13:36:42 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_vip_rep_monitor_0 (5) confirmed on node1 (rc=0)
pgsql(pgsql)[62677]:    2014/09/18_13:36:43 ERROR: PostgreSQL template1 isn't running
pgsql(pgsql)[62677]:    2014/09/18_13:36:43 ERROR: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
Sep 18 13:36:43 [51045] node1       lrmd:   notice: operation_finished:         pgsql_monitor_0:62677:stderr [ psql: FATAL:  the database system is starting up ]
Sep 18 13:36:43 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_monitor_0 (call=172, rc=1, cib-update=229, confirmed=true) unknown error
Sep 18 13:36:43 [51048] node1       crmd:   notice: process_lrm_event:  node1-pgsql_monitor_0:172 [ psql: FATAL:  the database system is starting up\n ]
Sep 18 13:36:43 [51048] node1       crmd:  warning: status_from_rc:     Action 4 (pgsql:0_monitor_0) on node1 failed (target: 7 vs. rc: 1): Error
Sep 18 13:36:43 [51048] node1       crmd:     info: abort_transition_graph:     match_graph_event:313 - Triggered transition abort (complete=0, node=node1, tag=lrm_rsc_op, id=pgsql_last_failure_0, magic=0:1;4:41:7:2b32876f-fe95-470b-b770-9c34a79944e9, cib=50.83.6) : Event failed
Sep 18 13:36:43 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_monitor_0 (4) confirmed on node1 (rc=4)
Sep 18 13:36:43 [51048] node1       crmd:     info: process_graph_event:        Detected action (41.4) pgsql_monitor_0.172=unknown error: failed
Sep 18 13:36:43 [51047] node1    pengine:  warning: unpack_rsc_op:      Processing failed op monitor for pgsql:0 on node1: unknown error (1)
Sep 18 13:36:43 [51047] node1    pengine:     info: clone_print:         Master/Slave Set: pgsql_master_slave [pgsql]
Sep 18 13:36:43 [51047] node1    pengine:     info: native_print:            pgsql      (ocf::heartbeat:pgsql): FAILED node1
Sep 18 13:36:43 [51047] node1    pengine:     info: native_print:            pgsql_vip_rep      (ocf::heartbeat:IPaddr2):       Stopped
Sep 18 13:36:43 [51047] node1    pengine:     info: native_print:            pgsql_forward_listen_port  (ocf::heartbeat:portforward):   Stopped
Sep 18 13:36:43 [51047] node1    pengine:     info: get_failcount_full:         pgsql:0 has failed INFINITY times on node1
Sep 18 13:36:43 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:43 [51047] node1    pengine:     info: get_failcount_full:         pgsql_master_slave has failed INFINITY times on node1
Sep 18 13:36:43 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:43 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:43 [51047] node1    pengine:     info: native_color:       Resource pgsql:1 cannot run anywhere
Sep 18 13:36:43 [51047] node1    pengine:     info: native_color:       Resource pgsql:0 cannot run anywhere
Sep 18 13:36:43 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:43 [51047] node1    pengine:     info: master_color:       pgsql_master_slave: Promoted 0 instances of a possible 1 to master
Sep 18 13:36:43 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_vip_rep: Rolling back scores from pgsql_forward_listen_port
Sep 18 13:36:43 [51047] node1    pengine:     info: native_color:       Resource pgsql_vip_rep cannot run anywhere
Sep 18 13:36:43 [51047] node1    pengine:     info: native_color:       Resource pgsql_forward_listen_port cannot run anywhere
Sep 18 13:36:43 [51047] node1    pengine:   notice: LogActions:         Stop    pgsql:0 (node1)
Sep 18 13:36:43 [51047] node1    pengine:     info: LogActions:         Leave   pgsql:1 (Stopped)
Sep 18 13:36:43 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_vip_rep   (Stopped)
Sep 18 13:36:43 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_forward_listen_port       (Stopped)
Sep 18 13:36:43 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 42: notify pgsql_pre_notify_stop_0 on node1 (local)
Sep 18 13:36:43 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=42:42:0:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_notify_0
Sep 18 13:36:43 [51045] node1       lrmd:     info: log_execute:        executing - rsc:pgsql action:notify call_id:181
Sep 18 13:36:43 [51045] node1       lrmd:     info: log_finished:       finished - rsc:pgsql action:notify call_id:181 pid:62802 exit-code:0 exec-time:212ms queue-time:0ms
Sep 18 13:36:43 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_notify_0 (42) confirmed on node1 (rc=0)
Sep 18 13:36:43 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_notify_0 (call=181, rc=0, cib-update=0, confirmed=true) ok
Sep 18 13:36:43 [51048] node1       crmd:   notice: te_rsc_command:     Initiating action 1: stop pgsql_stop_0 on node1 (local)
Sep 18 13:36:43 [51048] node1       crmd:     info: do_lrm_rsc_op:      Performing key=1:42:0:2b32876f-fe95-470b-b770-9c34a79944e9 op=pgsql_stop_0
Sep 18 13:36:43 [51045] node1       lrmd:     info: log_execute:        executing - rsc:pgsql action:stop call_id:182
Sep 18 13:36:43 [51046] node1      attrd:   notice: attrd_trigger_update:       Sending flush op to all hosts for: master-pgsql (-INFINITY)
Sep 18 13:36:43 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-pgsql']: OK (rc=0, origin=local/attrd/114, version=50.83.6)
Sep 18 13:36:43 [51046] node1      attrd:   notice: attrd_perform_update:       Sent update 115: master-pgsql=-INFINITY
Sep 18 13:36:43 [51048] node1       crmd:     info: abort_transition_graph:     te_update_diff:172 - Triggered transition abort (complete=0, node=node1, tag=nvpair, id=status-1-master-pgsql, name=master-pgsql, value=-INFINITY, magic=NA, cib=50.83.7) : Transient attribute: update
pgsql(pgsql)[62853]:    2014/09/18_13:36:45 INFO: waiting for server to shut down..... done server stopped
pgsql(pgsql)[62853]:    2014/09/18_13:36:45 INFO: PostgreSQL is down
Sep 18 13:36:45 [51043] node1        cib:     info: cib_process_request:        Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='pgsql-status']: OK (rc=0, origin=local/crm_attribute/3, version=50.83.7)
Sep 18 13:36:45 [51045] node1       lrmd:     info: log_finished:       finished - rsc:pgsql action:stop call_id:182 pid:62853 exit-code:0 exec-time:2573ms queue-time:0ms
Sep 18 13:36:45 [51048] node1       crmd:   notice: process_lrm_event:  LRM operation pgsql_stop_0 (call=182, rc=0, cib-update=231, confirmed=true) ok
Sep 18 13:36:45 [51048] node1       crmd:     info: match_graph_event:  Action pgsql_stop_0 (1) confirmed on node1 (rc=0)
Sep 18 13:36:45 [51047] node1    pengine:  warning: unpack_rsc_op:      Processing failed op monitor for pgsql:0 on node1: unknown error (1)
Sep 18 13:36:45 [51047] node1    pengine:     info: clone_print:         Master/Slave Set: pgsql_master_slave [pgsql]
Sep 18 13:36:45 [51047] node1    pengine:     info: native_print:            pgsql_vip_rep      (ocf::heartbeat:IPaddr2):       Stopped
Sep 18 13:36:45 [51047] node1    pengine:     info: native_print:            pgsql_forward_listen_port  (ocf::heartbeat:portforward):   Stopped
Sep 18 13:36:45 [51047] node1    pengine:     info: get_failcount_full:         pgsql:0 has failed INFINITY times on node1
Sep 18 13:36:45 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:45 [51047] node1    pengine:     info: get_failcount_full:         pgsql_master_slave has failed INFINITY times on node1
Sep 18 13:36:45 [51047] node1    pengine:  warning: common_apply_stickiness:    Forcing pgsql_master_slave away from node1 after 1000000 failures (max=1)
Sep 18 13:36:45 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:45 [51047] node1    pengine:     info: native_color:       Resource pgsql:0 cannot run anywhere
Sep 18 13:36:45 [51047] node1    pengine:     info: native_color:       Resource pgsql:1 cannot run anywhere
Sep 18 13:36:45 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_master_slave: Rolling back scores from pgsql_vip_rep
Sep 18 13:36:45 [51047] node1    pengine:     info: master_color:       pgsql_master_slave: Promoted 0 instances of a possible 1 to master
Sep 18 13:36:45 [51047] node1    pengine:     info: rsc_merge_weights:  pgsql_vip_rep: Rolling back scores from pgsql_forward_listen_port
Sep 18 13:36:45 [51047] node1    pengine:     info: native_color:       Resource pgsql_vip_rep cannot run anywhere
Sep 18 13:36:45 [51047] node1    pengine:     info: native_color:       Resource pgsql_forward_listen_port cannot run anywhere
Sep 18 13:36:45 [51047] node1    pengine:     info: LogActions:         Leave   pgsql:0 (Stopped)
Sep 18 13:36:45 [51047] node1    pengine:     info: LogActions:         Leave   pgsql:1 (Stopped)
Sep 18 13:36:45 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_vip_rep   (Stopped)
Sep 18 13:36:45 [51047] node1    pengine:     info: LogActions:         Leave   pgsql_forward_listen_port       (Stopped)
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Takehiro Matsushima
Hi,

I've faced similer problem, with PostgreSQL 9.3.3.
(I know it fixed on PostgreSQL 9.3.4.)


You can avoid if this problem is same as mine. Try following

1. Change "restart_on_promote" value to "false".
2. Stop pacemaker.
3. Remove once "recovery.conf".
4. Start PostgreSQL manually(It runs as master, confirm you can
connect to it using psql)
5. Stop PostgreSQL
6. Start pacemaker.


Regards,
Takehiro Matsushima
_______________________________________________
Linux-HA mailing list
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Oliver Weichhold
> Change "restart_on_promote" value to "false".

How? :)
Reply | Threaded
Open this post in threaded view
|

Unable to start any node of pgsql Master/Slave Cluster

Takehiro Matsushima
I'm sorry.
restart_on_promote is a parameter of pgsql RA.
 # pcs resource update pgsql restart_on_promote=false


--
Regards,
Takehiro Matsushima
_______________________________________________
Linux-HA mailing list
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Oliver Weichhold
Okay, I've tried your suggestions but unfortunately it didn't make any difference. Please also note that I'm running PostgreSQL 9.3.5.
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Takatoshi MATSUO
Hi Oliver

> pgsql(pgsql)[61018]:    2014/09/18_13:36:41 WARNING: PostgreSQL template1 isn't running
> pgsql(pgsql)[61018]:    2014/09/18_13:36:41 WARNING: Connection error (connection to the server went bad and the
> session was not interactive) occurred while executing the psql command.

I have no knowledge of PG9.3.
But these logs may be output by SQL "select now()" on template1 db,
so you can check PostgreSQL status manulally during starting-up too.

Thanks,
Takatoshi MATSUO

2014-09-19 19:57 GMT+09:00 Oliver Weichhold <[hidden email]>:

> Okay, I've tried your suggestions but unfortunately it didn't make any
> difference. Please also note that I'm running PostgreSQL 9.3.5.
>
>
>
> --
> View this message in context: http://linux-ha.996297.n3.nabble.com/Unable-to-start-any-node-of-pgsql-Master-Slave-Cluster-tp15816p15822.html
> Sent from the Linux-HA mailing list archive at Nabble.com.
> _______________________________________________
> Linux-HA mailing list
> [hidden email]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
[hidden email]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

Amul
Hi,

I am also in same trouble, I need a little help.
 
I have  set-up two node(node1 & node2) initial postgres Master at node1 & standby on other node. Installed required cluster tool & configuration on both node.
But pgsql resource agent stating both  postgres cluster in recovery mode. I have confirmed my replication set-up working as expected(script attached with this post).


Any help is appreciated.
Thank you.

=============================
Environment :
=============================
OS : Cent-OS x86_64
RMP installed:
cman-3.0.12.1-59.el6.x86_64
pacemaker-cluster-libs-1.1.10-14.el6.x86_64
pacemaker-1.1.10-14.el6.x86_64
pacemaker-libs-1.1.10-14.el6.x86_64
pacemaker-cli-1.1.10-14.el6.x86_64
corosynclib-1.4.1-17.el6.x86_64
corosync-1.4.1-17.el6.x86_64

STATUS & CONFIGURATION :
=============================
1. crm_mon -Afr -1 output:
=============================
-------------START:: crm_mon -Afr -1 output-------------------------------------
Last updated: Wed Oct  8 16:34:02 2014
Last change: Wed Oct  8 16:22:56 2014 via crmd on node1
Stack: cman
Current DC: node1 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured
4 Resources configured


Online: [ node1 node2 ]

Full list of resources:

 Resource Group: master-group
     vip-master (ocf::heartbeat:IPaddr2): Stopped
     vip-rep (ocf::heartbeat:IPaddr2): Stopped
 Master/Slave Set: msPostgresql [pgsql]
     Slaves: [ node1 node2 ]

Node Attributes:
* Node node1:
    + master-pgsql                     : -INFINITY
    + pgsql-data-status               : DISCONNECT
* Node node2:
    + master-pgsql                     : -INFINITY
    + pgsql-data-status               : DISCONNECT
    + pgsql-status                     : HS:alone  

Migration summary:
* Node node1:
* Node node2:

Failed actions:
    pgsql_monitor_0 on node1 'unknown error' (1): call=72, status=Timed Out,
        last-rc-change='Wed Oct  8 16:22:56 2014', queued=60002ms, exec=0ms
-------------END:: crm_mon -Afr -1 output------------------------------------------

=============================
2. crm configure show output:
=============================
-------------START:: crm configure show output------------------------------------------

node node1 \
        attributes pgsql-data-status=DISCONNECT
node node2 \
        attributes pgsql-data-status=DISCONNECT
primitive pgsql pgsql \
        params pgctl="/usr/pgsql-9.3/bin/pg_ctl" psql="/usr/pgsql-9.3/bin/psql" config="/var/lib/pgsql/data/postgresql.conf" pgdata="/var/lib/pgsql/data" start_opt="-p 5432" tmpdir="/var/lib/pgsql/tmpdir" rep_mode=sync node_list="node1 node2" restore_command="scp node1:/var/lib/pgsql/arc/%f %p" primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" master_ip=192.168.2.3 stop_escalate=0 restart_on_promote=true xlog_check_count=3 crm_attr_timeout=5 \
        op start interval=0s on-fail=restart timeout=60s \
        op monitor interval=4s on-fail=restart timeout=60s \
        op monitor interval=3s on-fail=restart role=Master timeout=60s \
        op promote interval=0s on-fail=restart timeout=60s \
        op demote interval=0s on-fail=stop timeout=60s \
        op stop interval=0s on-fail=block timeout=60s \
        op notify interval=0s timeout=60s
primitive vip-master IPaddr2 \
        params ip=192.168.0.3 nic=eth0 cidr_netmask=24 \
        op start interval=0s on-fail=restart timeout=60s \
        op monitor interval=10s on-fail=restart timeout=60s \
        op stop interval=0s on-fail=block timeout=60s
primitive vip-rep IPaddr2 \
        params ip=192.168.2.3 nic=eth2 cidr_netmask=24 \
        op start interval=0s on-fail=stop timeout=60s \
        op monitor interval=10s on-fail=restart timeout=60s \
        op stop interval=0s on-fail=ignore timeout=60s \
        meta migration-threshold=0
group master-group vip-master vip-rep
ms msPostgresql pgsql \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
colocation colocation-master-group-msPostgresql-INFINITY inf: master-group:Started msPostgresql:Master
order order-msPostgresql-master-group-0 0: msPostgresql:demote master-group:stop symmetrical=false
order order-msPostgresql-master-group-INFINITY inf: msPostgresql:promote master-group:start symmetrical=false
property cib-bootstrap-options: \
        dc-version=1.1.10-14.el6-368c726 \
        cluster-infrastructure=cman \
        no-quorum-policy=ignore \
        stonith-enabled=false \
        last-lrm-refresh=1412765576
rsc_defaults rsc_defaults-options: \
        resource-stickiness=INFINITY \
        migration-threshold=1
-------------END:: crm configure show output------------------------------------------

====================================================
3. Script attached to create Postgres replication
===================================================
PFA.
replication_remote.sh
====================================================
4. corosync.conf & cluster.conf attached
===================================================
PFA

cluster.conf

corosync.conf


Regards,
Amul Sul
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

wikinger10
This post has NOT been accepted by the mailing list yet.
In reply to this post by Oliver Weichhold
Hi Oliver,

did you find any solution. I think I am struggeling on the same problem with PostgreSQL 9.3.5 on Ubuntu 14.04. With the slight difference that my Slave is comming up. But I never can start my master, except by hand.

Here an excerpt from my syslog:

Feb  1 16:03:46 vps127956 pgsql(PostgreSQL)[27361]: INFO: PostgreSQL is running as a primary.
Feb  1 16:03:46 vps127956 pgsql(PostgreSQL)[27361]: INFO: PostgreSQL is started, checked by pgsql_real_monitor with rc=8 (Success=0, Master=8)
Feb  1 16:03:56 vps127956 pgsql(PostgreSQL)[27361]: INFO: PostgreSQL is started.
Feb  1 16:03:56 vps127956 lrmd[23804]:   notice: operation_finished: PostgreSQL_start_0:27361:stderr [ psql: FATAL:  the database system is starting up ]
Feb  1 16:03:56 vps127956 crmd[23807]:   notice: process_lrm_event: LRM operation PostgreSQL_start_0 (call=939, rc=1, cib-update=691, confirmed=true) unknown error

I dont know what I can do to fix this. I now spent X hours and will bive up soon.

Best
Mario
Reply | Threaded
Open this post in threaded view
|

Re: Unable to start any node of pgsql Master/Slave Cluster

agustin
This post has NOT been accepted by the mailing list yet.
In reply to this post by Oliver Weichhold
I have the same problem, any idea?

but the node2 take the control and node1 is died.

pgsql(pydb-motor)[380]: 2017/08/11_16:34:21 WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.

the problem start when insert this command

pcs resource create pydb-motor pgsql \
   pgctl="/usr/pgsql-9.3/bin/pg_ctl" \
   psql="/usr/pgsql-9.3/bin/psql" \
   pgdata="/bases/py/data/" \
   start_opt="" \
   pgport="50016" \
   pgdba="postgres" \
   pgdb="postgres" \
   config="/bases/py/data/postgresql.conf" \
   logfile="/bases/py/log/py-cluster.log" \
   tmpdir="/bases/py/var" \
   socketdir="/bases/py/var" \
   rep_mode="async" \
   node_list="uy-fm-clpilodb03 uy-fm-clpilodb04" \
   restore_command="" \
   primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \
   master_ip="172.16.20.245" \
   restart_on_promote='true' \
   op start   timeout="60s" interval="0s"  on-fail="restart" \
   op monitor timeout="60s" interval="4s" on-fail="restart" \
   op monitor timeout="60s" interval="3s"  on-fail="restart" role="Master" \
   op promote timeout="60s" interval="0s"  on-fail="restart" \
   op demote  timeout="60s" interval="0s"  on-fail="stop" \
   op stop    timeout="60s" interval="0s"  on-fail="block" \
   op notify  timeout="60s" interval="0s"

pcs resource master pydb-msPostgresql pydb-motor \
   master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

pcs resource group add pydb-group pydb-vip-master pydb-vip-rep

pcs constraint colocation add pydb-group with Master pydb-msPostgresql INFINITY
#pcs constraint order start pydb-group then start pydb-pgpool symmetrical=true score=INFINITY
pcs constraint order promote pydb-msPostgresql then start pydb-group symmetrical=false score=INFINITY
pcs constraint order demote  pydb-msPostgresql then stop pydb-group symmetrical=false score=0


[root@uy-fm-clpilodb03 aotaduy]# pcs status
Cluster name: hapostgres
Stack: corosync
Current DC: uy-fm-clpilodb04 (version 1.1.15-11.el7_3.5-e174ec8) - partition with quorum
Last updated: Fri Aug 11 16:43:31 2017 Last change: Fri Aug 11 16:34:09 2017 by hacluster via crmd on uy-fm-clpilodb04

2 nodes and 6 resources configured

Online: [ uy-fm-clpilodb03 uy-fm-clpilodb04 ]

Full list of resources:

 fence_vm_03 (stonith:fence_virsh): Started uy-fm-clpilodb04
 fence_vm_04 (stonith:fence_virsh): Started uy-fm-clpilodb03
 Master/Slave Set: pydb-msPostgresql [pydb-motor]
     Masters: [ uy-fm-clpilodb04 ]
     Stopped: [ uy-fm-clpilodb03 ]
 Resource Group: pydb-group
     pydb-vip-master (ocf::heartbeat:IPaddr2): Started uy-fm-clpilodb04
     pydb-vip-rep (ocf::heartbeat:IPaddr2): Started uy-fm-clpilodb04

Failed Actions:
* pydb-motor_start_0 on uy-fm-clpilodb03 'unknown error' (1): call=116, status=Timed Out, exitreason='none',
    last-rc-change='Fri Aug 11 16:34:11 2017', queued=0ms, exec=60016m

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/disabled

Any suggestions?