Status: DEGRADED 'Tableau Server Gateway 0' is in an error state com.tableausoftware.tabadmin.webapp.exceptions.ServiceFailedStateException

2019-05-28 16:59:56.368 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:56.468 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:56.899 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:57.478 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:57.920 -0700  Thread-1-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x1000000d3250004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:58.588 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.490 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.584 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:59.693 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)

------------------------------------------------------
2019-05-28 16:59:56.368 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:56.468 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:56.899 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:57.478 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:57.920 -0700  Thread-1-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x1000000d3250004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:58.588 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.490 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.584 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:59.693 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
------------------------------------------------------------------
2019-05-28 16:59:56.368 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:56.468 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:56.899 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:57.478 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:57.920 -0700  Thread-1-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x1000000d3250004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:58.588 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.490 -0700  Thread-1-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/112.000.00.33:8249. Will not attempt to authenticate using SASL (unknown error)
2019-05-28 16:59:59.584 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
2019-05-28 16:59:59.693 -0700  ZKWorker-ScheduledTask-0-SendThread(tab-ser1:8249) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server tab-ser1/0:0:0:0:0:0:0:1:8249. Will not attempt to authenticate using SASL (unknown error)

------------------------------------------------------------------------

2019-05-29 16:12:31.631 -0700  Curator-LeaderSelector-0 : ERROR com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService - Error running job 15 of type StartServerJob
java.util.concurrent.ExecutionException: com.tableausoftware.tabadmin.webapp.exceptions.ServiceFailedStateException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.runJob(AsyncJobService.java:210)
at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.leaderCallback(AsyncJobService.java:113)
at com.tableausoftware.tabadmin.agent.zookeeper.LeaderWorker$LeaderListener.takeLeadership(LeaderWorker.java:94)
at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:009)
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:421)
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:466)
at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:65)
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:246)
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:240)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.tableausoftware.tabadmin.webapp.exceptions.ServiceFailedStateException
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.abortIfAnyTerminallyFailed(ServiceStatusWatcher.java:523)
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.startingServicesSucceeded(ServiceStatusWatcher.java:468)
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.lambda$waitForServicesToStartInternal$9(ServiceStatusWatcher.java:446)
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.doInLoop(ServiceStatusWatcher.java:278)
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.waitForServicesToStartInternal(ServiceStatusWatcher.java:445)
at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.waitForServiceInstancesToStart(ServiceStatusWatcher.java:162)
at com.tableausoftware.tabadmin.webapp.asyncjobs.StartServerJob.lambda$run$2(StartServerJob.java:93)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.lambda$runStep$1(JobStepRunner.java:154)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:266)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:92)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:136)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:67)
at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:152)
at com.tableausoftware.tabadmin.webapp.asyncjobs.StartServerJob.run(StartServerJob.java:82)
at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.lambda$runJob$1(AsyncJobService.java:202)
... 4 more

Starting service...
Job id is '15', timeout is 30 minutes.
Unable to determine if all components of the service sta
node1: localhost
        Status: DEGRADED
        'Tableau Server Gateway 0' is in an error state.


Solution -
Confirm that the 'tabsvc-service-resurrection-data_tabsvc-clusterid.json' is empty or corrupt or missing. By default it can be found in at:
C:\ProgramData\Tableau\Tableau Server\data\tabsvc\tabsvc\0\tabsvc-service-resurrection-data_tabsvc-clusterid.json
If so, we can attempt to correct this using the following process:
1: Open the Command Prompt as Administrator.
2: Go to the scripts <version> folder, by default found:
\Tableau\Tableau Server\packages\scripts.<version>
3: Run the script to stop all administrative services: stop-administrative-services.cmd
4: Delete only the servicestate.json located by default in: 
\Tableau\Tableau Server\data\tabsvc\tabadminagent\0
5: Delete tabsvc-service-resurrection-data_tabsvc-clusterid.json if present at:
C:\ProgramData\Tableau\Tableau Server\data\tabsvc\tabsvc\0
6: Go to back to the scripts.<version> folder at:
\Tableau\Tableau Server\packages\scripts.<version>
7: Run the script to start all administrative services: start-administrative-services.cmd
8: Restart Tableau Server via either the TSM Manager or by running "tsm restart". 
9: If services do not automatically start, run tsm restart one or two additional times.