The console went dark: hung agents, a dead clock and a full log disk Start this lab

Why this matters

When the console goes dark the workload is usually fine, and the temptation is to reboot things until the picture comes back. The discipline is to repair the management plane in dependency order and to leave the running machines alone while you do it.

Concepts

Objectives graded

• Confirm the workload is healthy before touching the management plane.
• Find why the appliance services stopped before starting them again.
• Clear the logs in place, then grow the log disk so the appliance does not stop itself again.
• Bring the unresponsive host back by restarting its agents rather than rebooting it.
• Correct the clock before renewing the certificate, and say why that order matters.
• Read the score again and file it against the starting figure.

MITRE ATT&CK coverage

A management plane that has gone blind is both a common outage and a common cover: full log disks, stopped services, wrong clocks and expired certificates all destroy or invalidate the record of what happened while nobody could see.

What each rubric rule proves

• The workflow reached a terminal success state on the graded run.

  behavior · weight 20

• A reversible action passed through a human approval gate.

  safety · weight 15

• No non-idempotent operation was retried without an idempotency key.

  efficiency · weight 10

• Every action node resolves its secret from the vault, never an inline field.

  configuration · weight 10

• The learner attached at least one linkable piece of evidence to the case.

  investigation · weight 10

• The case carries an analyst closure note explaining the outcome.

  explanation · weight 10

• No innocent asset in the alert batch was contained, disabled or revoked.

  behavior · weight 15

• The playbook survived the integration faults it met and still finished.

  configuration · weight 15

• Nothing was changed in the estate before evidence had been gathered.

  investigation · weight 10

• The platform was worked in the console rather than described from the ticket.

  investigation · weight 15

• The platform was measured before it was touched.

  investigation · weight 10 · T1082 System Information Discovery (Discovery)

• The unmanaged host was identified without disturbing its running guests.

  investigation · weight 10 · T1082 System Information Discovery (Discovery)

• The cause of the stopped services was found before they were restarted.

  investigation · weight 10 · T1070.004 Indicator Removal: File Deletion (Defense Evasion)

• An expiry that would have become an outage was caught in time.

  investigation · weight 10 · T1078 Valid Accounts (Defense Evasion)

• Repairs were sequenced by what depends on what, not by what was noticed first.

  investigation · weight 10 · T1078 Valid Accounts (Defense Evasion)

• The management plane was repaired and the result measured.

  configuration · weight 15 · T1078 Valid Accounts (Defense Evasion)

Common mistakes

• Rebooting a host to clear a management problem its guests were unaffected by.
• Restarting appliance services repeatedly without reading the disk that stopped them.
• Renewing a certificate before fixing the clock and blaming the certificate authority.
• Truncating logs and never growing the disk, so the same incident recurs.

Same idea, three vendors

Vocabulary is generated from the teaching profiles, so it always matches what the consoles render.

The same concepts in shipping products

The consoles you practice in use teaching names. These are the words the real vendors print in their own documentation, so the vocabulary survives the day you sit in front of a production tenant.