CYBRARY The unit file is the malware’s configuration
ExecStart names the binary and where it reports. A unit you can read tells you the C2, the user it runs as, and whether it restarts — before you touch a single process.
H-303·Cross-platform·70 min
SELinux went quiet on the Northwind API server Start this lab
A RHEL 9 API server stopped enforcing SELinux at 23:10, and two minutes later a systemd service and timer began beaconing every ten minutes. Work the host with journalctl, systemctl and the SELinux tooling, contain it, and record why the denials that used to stop this stopped firing.
Mission brief · H-303
Tier 3 analyst on the incident response bench
Tier 3 — deep host and forensics
Business important
Track default
You are acting as
the break-glass IR account `svc-ir`, checked out of the vault for this incident — never the compromised identity
How you got here
The detection fired in the EDR console. You opened a live-response session from that detection, so your shell rides the EDR agent channel — containing the host will *not* cut you off, which is exactly why you contain from here rather than over SSH.
Your access
EDR console (detections + live response)
Endpoint detection & response
- Read the detection and process tree
- Look up host criticality, owner and VIP flags
- Open a live-response session on the host
- Contain the host over the agent channel
- Not yours: Disable user accounts (that's the identity team)
- Not yours: Delete evidence
Live-response shell on the affected host
Host shell session
- Read the filesystem, logs and running state
- Collect a triage package
- Not yours: Reboot or reimage without owner approval
What's at stake
The affected host
Owner: The system's business owner, named on the asset record
Containing a host you haven't identified can stop revenue. Check criticality and owner before you isolate anything.
Rules of engagement
You may act alone
- Investigate from the live-response session
- Collect evidence and triage packages
- Contain a standard-criticality host
Ask a human first
- Containing a mission-critical host — asset owner and on-call manager first
- Disabling a user identity — identity team, and HR/legal if it looks like an insider
Why this matters
On Linux, persistence and the control that should have stopped it live in the same place: systemd and SELinux. This case makes the analyst read both. The attacker did not defeat SELinux — they turned it off, and the audit log recorded the exact moment the denials stopped.
Concepts
journalctl and ausearch tell different halves of the story
The journal shows sudo, the unit starting and the timer re-arming. The audit log shows the AVC denials at permissive=0 and the MAC_STATUS record where enforcement flipped. Together they date the control failure.
Isolating an API server is a customer-facing outage
Containment is consequential and passes a gate. The build host answered a legitimate SSH session and is not a containment target.
A timer is the persistence, the service is only the symptom
Stopping nw-metrics.service buys ten minutes. Eradication means the timer, the unit, the binary and the SELinux mode all going back to where they belong.
Objectives graded
- Work a RHEL host with journalctl, systemctl, ausearch and the SELinux tooling
- Prove the control failure, the persistence unit, its timer and the live beacon
- Contain the host and the service identity under approval, then record root cause
Live walkthrough
The reference response, executed by the same engine that grades your attempt. Yours will differ — the rubric reads end state and evidence, not the shape of your workflow.
- 1.Approve consequential actionpromptapproval gateApprove consequential action
- 2.Write case field credentialRefactioncredentialRef → vault/soc-automation
- 3.Attach evidence: case/soar-h-303/evidenceactionThe learner attached at least one linkable piece of evidence to the case.
- 4.Write case field closureNoteactionclosureNote → written
- 5.Attach evidence: range/hst-nw-api05/sessionactionThe analyst worked the RHEL host directly instead of grading the alert text.
- 6.Write case field rogueUnitactionrogueUnit → /etc/systemd/system/nw-metrics.service
- 7.Write case field attackerIpactionattackerIp → 203.0.113.77
- 8.Write case field selinuxModeactionselinuxMode → Permissive
- 9.Write case field beaconPidactionbeaconPid → 9142
- 10.isolated on hst-nw-api05actionapproval gatehst-nw-api05.isolated → true
- 11.enabled on usr-svc-apiactionusr-svc-api.enabled → false
- 12.Write case field persistenceTimeractionpersistenceTimer → nw-metrics.timer
- 13.Write case field appVersionactionappVersion → 2.6.3
- 14.Attach evidence: case/soar-h-303/timelineactionA post-incident timeline was filed: setenforce, unit install, first beacon, timer re-arm, containment.
- 15.Write case field segmentGatewayactionsegmentGateway → 10.42.32.1
- 16.Write case field seatRamGbactionseatRamGb → 16
- 17.Write case field seatDiskGbactionseatDiskGb → 512
Gate approved — 100%
22 of 22 rubric rules hold.
Gate denied — 87%
Refusing the consequential action is a defensible choice with a measured cost; the difference is what the lab is teaching.
- pass behavior · The workflow reached a terminal success state on the graded run.
- pass safety · A consequential action passed through a human approval gate.
- pass efficiency · No non-idempotent operation was retried without an idempotency key.
- pass configuration · Every action node resolves its secret from the vault, never an inline field.
- pass investigation · The learner attached at least one linkable piece of evidence to the case.
- pass explanation · The case carries an analyst closure note explaining the outcome.
- pass behavior · No innocent asset in the alert batch was contained, disabled or revoked.
- pass configuration · The playbook survived the integration faults it met and still finished.
- pass investigation · Nothing was changed in the estate before evidence had been gathered.
- pass investigation · The analyst worked the RHEL host directly instead of grading the alert text.
- pass investigation · The persistence unit was located on disk, so eradication has a concrete target.
- pass investigation · The external peer was proven from the host's own socket table, not from the alert.
- pass investigation · The analyst checked SELinux state and found the control had been dropped to permissive.
- pass behavior · The live beacon was identified by PID before anything was contained.
- pass behavior · The compromised API server is isolated at end state.
- pass behavior · The service account that ran setenforce and installed the unit is disabled pending rotation.
- pass behavior · The timer that re-arms the implant was found, so eradication outlives a service stop.
- pass explanation · Root cause is recorded as the unpatched API version carrying the advisory.
- pass explanation · A post-incident timeline was filed: setenforce, unit install, first beacon, timer re-arm, containment.
- pass investigation · The analyst scoped the host's network from the wire, so containment decisions know which segment is in play.
- pass investigation · The analyst scoped the machine itself, so memory-hungry triage steps are planned against real capacity rather than a guess.
- pass investigation · Disk capacity was taken from the volume itself, which is what makes an image or memory-dump decision defensible.
MITRE ATT&CK coverage
SELinux is dropped to permissive on a RHEL API server, a systemd service and matching timer install an unsigned collector that beacons over HTTPS, and the audit log keeps the denials that used to block it.
| Technique | Name | Tactic |
|---|---|---|
| T1562.001 | Disable or Modify Tools | Defense Evasion |
| T1543.002 | Create or Modify System Process: Systemd Service | Persistence |
| T1053.006 | Scheduled Task/Job: Systemd Timers | Persistence |
| T1071.001 | Application Layer Protocol: Web Protocols | Command and Control |
| T1078 | Valid Accounts | Defense Evasion |
What each rubric rule proves
The workflow reached a terminal success state on the graded run.
behavior · weight 20 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
A consequential action passed through a human approval gate.
safety · weight 15 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
No non-idempotent operation was retried without an idempotency key.
efficiency · weight 10 · T1071.001 Application Layer Protocol: Web Protocols (Command and Control)
Every action node resolves its secret from the vault, never an inline field.
configuration · weight 10 · T1078 Valid Accounts (Defense Evasion)
The learner attached at least one linkable piece of evidence to the case.
investigation · weight 10 · T1562.001 Disable or Modify Tools (Defense Evasion)
The case carries an analyst closure note explaining the outcome.
explanation · weight 10 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
No innocent asset in the alert batch was contained, disabled or revoked.
behavior · weight 15 · T1078 Valid Accounts (Defense Evasion)
The playbook survived the integration faults it met and still finished.
configuration · weight 15 · T1071.001 Application Layer Protocol: Web Protocols (Command and Control)
Nothing was changed in the estate before evidence had been gathered.
investigation · weight 10 · T1562.001 Disable or Modify Tools (Defense Evasion)
The analyst worked the RHEL host directly instead of grading the alert text.
investigation · weight 10 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
The persistence unit was located on disk, so eradication has a concrete target.
investigation · weight 10 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
The external peer was proven from the host's own socket table, not from the alert.
investigation · weight 10 · T1071.001 Application Layer Protocol: Web Protocols (Command and Control)
The analyst checked SELinux state and found the control had been dropped to permissive.
investigation · weight 10 · T1562.001 Disable or Modify Tools (Defense Evasion)
The live beacon was identified by PID before anything was contained.
behavior · weight 10 · T1071.001 Application Layer Protocol: Web Protocols (Command and Control)
The compromised API server is isolated at end state.
behavior · weight 15 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
The service account that ran setenforce and installed the unit is disabled pending rotation.
behavior · weight 10 · T1078 Valid Accounts (Defense Evasion)
The timer that re-arms the implant was found, so eradication outlives a service stop.
behavior · weight 10 · T1053.006 Scheduled Task/Job: Systemd Timers (Persistence)
Root cause is recorded as the unpatched API version carrying the advisory.
explanation · weight 10 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
A post-incident timeline was filed: setenforce, unit install, first beacon, timer re-arm, containment.
explanation · weight 15 · T1543.002 Create or Modify System Process: Systemd Service (Persistence)
The analyst scoped the host's network from the wire, so containment decisions know which segment is in play.
investigation · weight 5
The analyst scoped the machine itself, so memory-hungry triage steps are planned against real capacity rather than a guess.
investigation · weight 3
Disk capacity was taken from the volume itself, which is what makes an image or memory-dump decision defensible.
investigation · weight 3
Common mistakes
- Assuming SELinux is enforcing because the build standard says so — ask the host.
- Filing the unit name instead of the full path under /etc/systemd/system.
- Stopping the service and closing the case while the timer is still armed.
- Isolating the build host, which held a legitimate SSH session and is not compromised.
Same idea, three vendors
Vocabulary is generated from the teaching profiles, so it always matches what the consoles render.
| Concept | Cortex Forge | Summit SOAR | Storyline |
|---|---|---|---|
| artifact | Indicator | Artifact | Record |
| case | Incident | Container | Case |
| approval | Manual task | Prompt | Page |
| note | War Room entry | Note | Case note |
Where does this value come from?
The same reference to an enrichment node's verdict, written three ways:
- Cortex Forge: ${enrich-sender.verdict}
- Summit SOAR: enrich-sender.verdict
- Storyline: <<enrich-sender.verdict>>
The same concepts in shipping products
The consoles you practice in use teaching names. These are the words the real vendors print in their own documentation, so the vocabulary survives the day you sit in front of a production tenant.
| Concept | Cortex XSOARXSOAR 8.x | Splunk SOARSplunk SOAR 6.x (formerly Phantom) | TinesTines (2026 cloud) | Microsoft Sentinel + Logic AppsSentinel (Defender portal, 2026) | Google SecOps SOARSecOps SOAR (formerly Siemplify) | Swimlane TurbineTurbine (2026) | Torq HyperautomationTorq (2026) | IBM QRadar SOARQRadar SOAR (Resilient) 51.x | ServiceNow Security OperationsSecOps (Flow Designer era) |
|---|---|---|---|---|---|---|---|---|---|
| artifact | Indicator | Artifact (CEF fields) | Event payload field | Entity | Entity | Field / correlation value | Event field | Artifact | Observable |
| case | Incident | Container | Case | Incident | Case | Record | Case | Incident | Security incident (SIR) |
| approval | Manual (data collection) task | Prompt block | Page (form) / Send to Chat | Approval action (Teams / email) | Manual action | Task / approval action | Approval step (Torq Chat / Slack) | Manual task / phase gate | Approval activity |
| note | War Room entry | Note | Case note | Comment | Case wall comment | Comment | Case comment | Note | Work note |
What transfers
Impaired defenses are a finding in their own right. Wherever you work, the first question after “how did it run?” is “what should have stopped it, and who turned that off?”