Manual Failover (Disabling ACM)

On the active node, disable ACM locally.

$ /opt/aspera/acm/bin/acmctl –d
ACM is disabled locally

Stop all services.

$ asctl all:stop
Orchestrator is being stopped forcefully: 
-> Killing Orchestrator Engine process with pid: 24542 
-> Killing Mongrel serving orchestrator on port 3000 with pid: 24638 
-> Killing Mongrel serving orchestrator on port 3001 with pid: 24641 
-> Killing Mongrel serving orchestrator on port 3002 with pid: 24644 
-> Orchestrator Monitor running with pid: 24663 
-> Killing Orchestrator Monitor with pid: 24663 
-> Killing Asynchronous Worker Process 0 with pid 24665 
-> Killing Asynchronous Worker Process 1 with pid 24668 
-> Killing Synchronous Worker Process 2 with pid 24670 
-> Killing Synchronous Worker Process 3 with pid 24672 
-> Killing Synchronous Worker Process 4 with pid 24675 
-> Killing Synchronous Worker Process 5 with pid 24677
Apache: Stop... done
MySQL: Stop... done

Run the acmctl -i command to verify that all Orchestrator services have been stopped.

The ACM software checks the /opt/aspera/acm/run/acm4orchestrator.status file to determine if the file is older than 2 minutes. If the file is older than 2 minutes, the acmctl -i command reports the Status [of] file as expired, which causes ACM to start the failover process to the other Orchestrator server.

Note: This command can be used to verify that all Orchestrator services have been stopped, even if the actual failover has not yet occurred.

$ /opt/aspera/acm/bin/acmctl –i
Checking current ACM status...
Aspera Cluster Manager for Orchestrator - status
------------------------------------------
Local hostname:              orchestrator1
Active node:                 orchestrator1 (me)
Status of this node:         active
Status file:                 current
Disabled globally:           no
Disabled on this node:       yes

Database configuration file
---------------------------
Database host:         localhost

Orchestrator active/active services status
------------------------------------
Apache:                stopped

Orchestrator Status:  
  -> Orchestrator Engine NOT running 
  -> Mongrel serving orchestrator on port 3000 is NOT running 
  -> Mongrel serving orchestrator on port 3001 is NOT running 
  -> Mongrel serving orchestrator on port 3002 is NOT running 
  -> Orchestrator Monitor NOT running 
  -> Asynchronous Worker Process 0 is NOT running 
  -> Asynchronous Worker Process 1 is NOT running 
  -> Synchronous Worker Process 2 is NOT running 
  -> Synchronous Worker Process 3 is NOT running 
  -> Synchronous Worker Process 4 is NOT running 
  -> Synchronous Worker Process 5 is NOT running 

Orchestrator active/passive services status
------------------------------------------
MySQL:                 stopped

After a minute or so, confirm that the formerly passive node has become active with one of the following methods:

Run the acmctl –i command, as in the following example:

$ /opt/aspera/acm/bin/acmctl -i
Checking current ACM status...
Aspera Cluster Manager for Orchestrator - status
------------------------------------------
Local hostname:         orchestrator2
Active node:            orchestrator2 (me)
Status of this node:    active
Status file:            current
Disabled globally:      no
Disabled on this node:  no

Database configuration file
---------------------------
Database host:         localhost

Orchestrator active/active services status
------------------------------------
Apache:                 running

Orchestrator Status: 
-> Orchestrator Engine running with pid: 31524 
-> Mongrel serving orchestrator on port 3000 is running with pid: 31636 
-> Mongrel serving orchestrator on port 3001 is running with pid: 31681 
-> Mongrel serving orchestrator on port 3002 is running with pid: 31685 
-> Orchestrator Monitor running with pid: 31713 
-> Asynchronous Worker Process 0 is running with pid: 31715 
-> Asynchronous Worker Process 1 is running with pid: 31717 
-> Synchronous Worker Process 2 is running with pid: 31719 
-> Synchronous Worker Process 3 is running with pid: 31721 
-> Synchronous Worker Process 4 is running with pid: 31723 
-> Synchronous Worker Process 5 is running with pid: 31725 

Orchestrator active/passive services status
------------------------------------------
MySQL:                 running

If the acm4orchestrator.log file has been enabled, review the file.

# tail –f /opt/aspera/acm/log/acm4orchestrator.log
2016-10-28 16:48:01 (-0800) acm4orchestrator haorchestrator1 (21470): ACM is now disabled globally: aborting
2016-10-28 16:48:01 (-0800) acm4orchestrator haorchestrator1 (14664): ACM is now disabled globally: aborting

On the newly passive node, re-enable ACM to automatically start the active services.

$ /opt/aspera/acm/bin/acmctl –e
ACM is enabled locally

This action restarts the active/active Orchestrator services in order to provide HA functionality.

After a few minutes, verify that the active/active services have started on the passive node with the following command:

$ /opt/aspera/acm/bin/acmctl -i
Checking current ACM status...

Aspera Cluster Manager for Orchestrator - status
------------------------------------------
Local hostname:            orchestrator1
Active node:               orchestrator2
Status of this node:       passive
Status file:               current
Disabled globally:         no
Disabled on this node:     no

Database configuration file
---------------------------
Database host:         localhost

Orchestrator active/active services status
------------------------------------
Apache:                running

Orchestrator Status: 
  -> Orchestrator Engine running with pid: 26932 
  -> Mongrel serving orchestrator on port 3000 is running with pid: 26967 
  -> Mongrel serving orchestrator on port 3001 is running with pid: 26970 
  -> Mongrel serving orchestrator on port 3002 is running with pid: 26973 
  -> Orchestrator Monitor running with pid: 26979 
  -> Asynchronous Worker Process 0 is running with pid: 26981 
  -> Asynchronous Worker Process 1 is running with pid: 26983 
  -> Synchronous Worker Process 2 is running with pid: 26985 
  -> Synchronous Worker Process 3 is running with pid: 26987 
  -> Synchronous Worker Process 4 is running with pid: 26989 
  -> Synchronous Worker Process 5 is running with pid: 26991

Orchestrator active/passive services status
------------------------------------------
MySQL:                 running