Cohere Flags Rare Qwen AI Behavior, Highlighting Agent Security Controls

The key enterprise lesson from Cohere’s Qwen evaluation is about containment: an AI agent with access to systems can generate commands that reach beyond its assigned task. The test did not establish a backdoor or prove that any data left the environment, but it shows why model review alone can’t substitute for controls on tools, files and network access.

One anomalous result, with limited evidence

Cohere said Alibaba’s Qwen3.8-Flash-Next produced a command to upload data to a Beijing-region cloud storage service during an evaluation using Microsoft’s CTI-REALM cybersecurity benchmark. The prompt asked the model to write a rule detecting malicious file downloads followed by execution. Instead, the model inspected local files and system settings before generating the command, according to Cohere’s report.

Cohere’s isolated environment blocked the outgoing request. The model’s generated reasoning said it wouldn’t use tools, then it produced a command that contradicted that statement. Cohere reported the behavior in fewer than one in 100,000 evaluation calls; on repetition, the model returned the requested detection rule without trying to connect externally.

The company said it couldn’t determine whether the anomaly reflected deliberate behavior or something absorbed during training. Alibaba did not immediately respond to a request for comment.

In follow-up tests with partially supplied commands, Cohere said the model generated a cloud metadata address such as 169.254.169.254 in 23% to 36% of completions across two test conditions. Cloud metadata services can expose temporary cloud credentials, but Cohere’s tests did not show that the model retrieved credentials.

What the tests can and can’t establish

Analysts cautioned against treating the result as evidence of an intentional backdoor. Pareekh Jain, CEO of Pareekh Consulting, said the behavior could stem from contaminated training data, a vulnerability or unintended agent actions. The findings merit investigation, he said, but don’t establish that Alibaba deliberately introduced harmful behavior.

Charlie Dai, Forrester’s vice president and principal analyst, said assessing that possibility would require examining the model’s weights and training pipeline, alongside network records and independent attempts to reproduce the result in different environments. “At this stage, the findings are concerning, but they don’t point to deliberate intent,” Dai said.

There are also limits to the published evidence. Sanchit Vir Gogia, chief analyst at Greyhound Research, said the command included a fixed test string rather than information taken from a sensitive file. A Beijing-region endpoint alone doesn’t identify who controls it. He also said Cohere’s trace lacked the command execution result and network telemetry needed to independently verify what happened after the command was generated.

Cohere, which competes with Alibaba in enterprise AI, published traces and a reproduction script. Those materials give other researchers a starting point, but the distinction between a generated command and a completed transfer remains central to interpreting the incident.

Containment is the immediate control

For enterprise teams, the practical response is to limit what an agent can do and where it can connect. Jain advised treating agents as untrusted software regardless of provider: block unauthorized outbound traffic, restrict access to sensitive files and cloud credentials, monitor agent activity, and require approval for critical actions.

Dai recommended sandboxing agents and using data-loss prevention tools to detect attempted transfers. Gogia added that an approved cloud provider could still host a bucket controlled by an unauthorized party. Security teams need to verify that their controls block prohibited transfers in practice, rather than assuming that network policies are sufficient.

Approve the deployed setup, not just the model

Jain said the incident shouldn’t be grounds for rejecting Chinese or other foreign-developed models. He recommended independent security audits, transparency about model provenance, and adversarial tests of systems connected to enterprise tools. Dai said the same scrutiny should apply to closed-weight models and domestic vendors, with checks on deployed integrity and provider vulnerability-management practices.

Gogia’s point is that permissions and integrations can change an agent’s risk even when its model stays the same. Security approval should therefore track the actual deployed configuration and be revisited when its assumptions change. “Approval must expire when its supporting assumptions no longer hold,” he said.