Over the past week there’s been heightened concern about how LLMs can be used to facilitate cyber operations. Much of that concern is tightly linked to recent reports from Anthropic, which are facing growing criticism from the security community.
Anthropic claimed that a threat actor launched an AI-assisted operation which was up to 90% autonomous. But the LLM largely relied on pre-existing open source tools that operators already chain together, and the success rates appear low. Moreover, hallucinations meant that adversaries were often told that the LLM had done something, or had access to credentials, when it did not.
We should anticipate that LLMs will enable some adversaries to chain together code that could exploit vulnerabilities. But vibe‑coding an exploit chain is not the same as building something that can reliably compromise real systems. To date, experiments with vibe‑coded malware and autonomous agents suggest that generated outputs typically require skilled operators to debug, adapt, and operationalise them. Even then, the outputs of LLM‑assisted malware often fail outright when confronted with real‑world constraints and defences.
That’s partly because exploit development is a different skill set and capability than building “functional‑enough” software. Vibe coding for productivity apps might tolerate flaky edge cases and messy internals. Exploit chains, by contrast, often fail to exploit vulnerabilities unless they are properly tailored to a given target.
An AI system that can assemble a roughly working application from a series of prompts does not automatically inherit the ability to produce highly reliable, end‑to‑end exploit chains. Some capability will transfer, but we should be wary of assuming a neat, 100% carry‑over from vibe‑coded software to effective vibe‑coded malware.
For the largest model tested (13 billion parameters trained on 260 billion tokens), just 250 malicious documents representing 0.00016 percent of total training data proved sufficient to install the backdoor. The same held true for smaller models, even though the proportion of corrupted data relative to clean data varied dramatically across model sizes.
The findings apply to straightforward attacks like generating gibberish or switching languages. Whether the same pattern holds for more complex malicious behaviors remains unclear. The researchers note that more sophisticated attacks, such as making models write vulnerable code or reveal sensitive information, might require different amounts of malicious data.
The same pattern appeared in smaller models as well:
Despite larger models processing over 20 times more total training data, all models learned the same backdoor behavior after encountering roughly the same small number of malicious examples.
The authors note important limitations: the tested models were all relatively small, the results depend on tainted data being present in the training set, and real-world mitigations like guardrails or corrective fine-tuning may blunt such effects.
Even so, the findings point to the ongoing immaturity of LLM cybersecurity practices and the difficulty of assuring trustworthiness in systems trained at scale. Safely deploying AI in high-risk contexts will require not just policy oversight, but rigorous testing, data provenance controls, and continuous monitoring of model behaviour.
Of note: the discussion that current larger AI models that are in-use today will really have noticeable effects / changes in user behaviour on edge or end point devices in a 2-3 years once semiconductors have more properly caught up.
Significantly, this may mean policy makers still have some time to establish appropriate regulatory frameworks and guardrails ahead of what maybe more substantive and pervasive changes to daily computing.
These activities are occurring despite OpenAI’s warnings that Whisper should not be used in high-risk domains.
The article reports that a “machine learning engineer said he initially discovered hallucinations in about half of the over 100 hours of Whisper transcriptions he analyzed. A third developer said he found hallucinations in nearly every one of the 26,000 transcripts he created with Whisper. The problems persist even in well-recorded, short audio samples. A recent study by computer scientists uncovered 187 hallucinations in more than 13,000 clear audio snippets they examined.”
Transcription errors can be very serious. Research by Prof. Koenecke and Prof. Sloane of the University of Virgina found:
… that nearly 40% of the hallucinations were harmful or concerning because the speaker could be misinterpreted or misrepresented.
In an example they uncovered, a speaker said, “He, the boy, was going to, I’m not sure exactly, take the umbrella.”
But the transcription software added: “He took a big piece of a cross, a teeny, small piece … I’m sure he didn’t have a terror knife so he killed a number of people.”
A speaker in another recording described “two other girls and one lady.” Whisper invented extra commentary on race, adding “two other girls and one lady, um, which were Black.”
In a third transcription, Whisper invented a non-existent medication called “hyperactivated antibiotics.”
While, in some cases, voice data is deleted for privacy reasons this can impede physicians (or other medical personnel) from double checking the accuracy of transcription. While some may be caught, easily and quickly, more subtle errors or mistakes may be less likely to be caught.
One area where work stills needs to be done is to assess the relative accuracy of the AI scribes versus that of physicians. While there may be errors introduced by automated transcription what is the error rate of physicians? Also, what is the difference in quality of care between one whom is self-transcribing during a meeting vs reviewing transcriptions after the interaction? These are central questions that should play a significant role in assessments of when and how these technologies are deployed.
Are we approaching “Google zero”, where Google searches will use generative AI systems to summarize responses to queries, thus ending the reason for people to visit website? And if that happens what is lost?
These are common questions that have been building month over month as more advanced foundational models are built, deployed, and iterated upon. But there has been relatively little assessment in public forums around the social dimensions of making a web search. Instead, the focus has tended to be on loss of traffic and subsequent economic effects of this transition.
A 2022 paper entitled “Situating Search” identifies what a search engine does, and what it is used for, in order for the authors to argue that search that only provides specific requested information (often inaccurately) fails to account for the broader range of things that people use search for.
Specifically, when people search they:
lookup
learn
investigate
When a ChatGPT or Gemini approach to search is applied, however, it limits the range of options before a user. Specifically, in binding search to conversational responses we may impair individuals from conducting search/learning in ways that expand domain knowledge or that rely on sensemaking of results to come to a given conclusion.
Page 227 of the paper has a helpful overview of the dimensions of Information Seeking Strategies (ISS), which explain the links between search and the kinds of activities in which individuals engage. Why, also, might chat-based (or other multimodal) search be a problem?
it can come across as too authoritative
by synthesizing data from multiple sources and masking the available range of sources, it cuts the individual’s ability to expose the broader knowledge space
LLMs, in synthesizing text, may provide results that are not true
All of the above issues are compounded in situations where individuals have low information literacy and, thus, are challenged in their ability to recognize deficient responses from an AI-based search system.
The authors ultimately conclude with the following:
…we should be looking to build tools that help users find and make sense of information rather than tools that purport to do it all for them. We should also acknowledge that the search systems are used and will continue to be used for tasks other than simply finding an answer to a question; that there is tremendous value in information seekers exploring, stumbling, and learning through the process of querying and discovery through these systems.
As we race to upend the systems we use, today, we should avoid moving quickly and breaking things and instead opt to enhance and improve our knowledge ecosystem. There is a place for these emerging technologies but rather than bolting them onto–and into–all of our information technologies we should determine when they are or are not fit for a given purpose.
A good article by The Markup assessed the accuracy of New York City’s municipal chatbot. The chatbot is intended to provide New Yorkers with information about starting or operating a business in the city. The journalists found the chatbot regularly provided false or incorrect information which could result in legal repercussions for businesses and significantly discriminate against city residents. Problematic outputs included incorrect housing-related information, whether businesses must accept cash for services rendered, whether employers can take cuts of employees’ tips, and more.
While New York does include a warning to those using the chatbot, it remains unclear (and perhaps doubtful) that residents who use it will know when to dispute outputs. Moreover, the statements of how the tool can be helpful and sources it is trained on may cause individuals to trust the chatbot.
In aggregate, this speaks to how important it is to effectively communicate with users, in excess of policies simply mandating some kind of disclosure of the risks associated with these tools, as well as demonstrates the importance of government institutions more carefully assessing (and appreciating) the risks of these systems prior to deploying them.
While some emerging generative technologies may positively affect various domains (e.g., certain aspects of drug discovery and biological research, efficient translation between certain languages, speeding up certain administrative tasks, etc) they are, also, enabling new forms of harmful activities. Case in point, some individuals and groups are using generative technologies to generate child sexual abuse or exploitation materials:
Sexton says criminals are using older versions of AI models and fine-tuning them to create illegal material of children. This involves feeding a model existing abuse images or photos of people’s faces, allowing the AI to create images of specific individuals. “We’re seeing fine-tuned models which create new imagery of existing victims,” Sexton says. Perpetrators are “exchanging hundreds of new images of existing victims” and making requests about individuals, he says. Some threads on dark web forums share sets of faces of victims, the research says, and one thread was called: “Photo Resources for AI and Deepfaking Specific Girls.”
…
… realism also presents potential problems for investigators who spend hours trawling through abuse images to classify them and help identify victims. Analysts at the IWF, according to the organization’s new report, say the quality has improved quickly—although there are still some simple signs that images may not be real, such as extra fingers or incorrect lighting. “I am also concerned that future images may be of such good quality that we won’t even notice,” says one unnamed analyst quoted in the report.
The ability to produce generative child abuse content is becoming a wicked problem with few (if any) “good” solutions. It will be imperative for policy professionals to learn from past situations where technologies were found to sometimes facilitate child abuse related harms. In doing so, these professionals will need to draw lessons concerning what kinds of responses demonstrate necessity and proportionality with respect to the emergent harms of the day.
As just one example, we will have to carefully consider how generative AI-created child sexual abuse content is similar to, and distinctive from, past policy debates on the policing of online child sexual abuse content. Such care in developing policy responses will be needed to address these harms and to avoid undertaking performative actions that do little to address the underlying issues that drive this kind of behaviour.
Relatedly, we must also beware the promise that past (ineffective) solutions will somehow address the newest wicked problem. Novel solutions that are custom built to generative systems may be needed, and these solutions must simultaneously protect our privacy, Charter, and human rights while mitigating harms. Doing anything less will, at best, “merely” exchange one class of emergent harms for others.
A whole computing infrastructure based on tracking metadata reliably and then presenting it to users in ways they understand and care about, and which is adopted by the masses.
That generative outputs will need to remain the exception as opposed to the norm: when generative image manipulation (not full image creation) is normal then how much will this glyph help to notify people of ‘fake’ imagery or other content?
That there are sufficiently low benefits to offering metadata-stripping or content-modification or content-creation systems that there are no widespread or easy-to-adopt ways of removing the identifying metadata from generative content.
Finally, where the intent behind fraudulent media is to intimidate, embarrass, or harass (e.g., non-consensual deepfake pornographic content, violence content), then what will the glyph in question do to allay these harms? I suspect very little unless it is, also, used to identify individuals who create content for the purposes of addressing criminal or civil offences. And, if that’s the case, then the outputs would constitute a form of data that are designed to deliberately enable state intervention in private life, which could raise a series of separate, unique, and difficult to address problems.
For the past several months Neale James has talked about how new laws which prevent taking pictures of people on the street will inhibit the documenting of history in certain jurisdictions. I’ve been mulling this over while trying to determine what I really think about this line of assessment and photographic concern. As a street photographer it seems like an issue where I’ve got some skin in the game!
In short, while I’m sympathetic with this line of argumentation I’m not certain that I agree. So I wrote a longish email to Neale—which was included in this week’s Photowalk podcast—and I’ve largely reproduced the email below as a blog post.
I should probably start by stating my priors:
As a street photographer I pretty well always try to include people in my images, and typically aim to get at least some nose and chin. No shade to people who take images of peoples’ backs (and I selectively do this too) but I think that capturing some of the face’s profile can really bring many street photos to life.1
I, also, am usually pretty obvious when I’m taking photos. I find a scene and often will ‘set up’ and wait for folks to move through it. And when people tell me they aren’t pleased or want a photo deleted (not common but it happens sometimes) I’m usually happy to do so. I shoot between 28-50mm (equiv.) focal lengths and so it’s always pretty obvious when I’m taking photos, which isn’t the case with some street photographers who are shooting at 100mm . To each their own but I think if I’m taking a photo the subjects should be able to identify that’s happening and take issue with it, directly, if they so choose to.
Anyhow, with that out of the way:
If you think of street photography in the broader history of photography, it started with a lot of images with hazy or ghostly individuals (e.g. ‘Panorama of Saint Lucia, Naples’ by Jones or ’Physic Street, Canton’ by Thomson or ‘Rue de Hautefeuille’ by Marville). Even some of the great work—such as by Cartier-Bresson, Levitt, Bucquet, van Schaick, Atget, Friedlander, Robert French, etc—include photographs where the subjects are not clearly identified. Now, of course, some of their photographs include obvious subjects, but I think that it’s worth recognizing that many of the historical ‘greats’ include images where you can’t really identify the subject. And… that was just fine. Then, it was mostly a limitation of the kit whereas now, in some places, we’re dealing with the limitations of the law.
Indeed, I wonder if we can’t consider the legal requirement that individuals’ identifiable images not be captured as potentially a real forcing point for creativity that might inspire additional geographically distinctive street photography traditions: think about whether, in some jurisdictions, instead of aperture priority being a preferred setting, that shutter priority is a default, with speeds of 5-15 second shutters to get ghostly images.2
Now, if such a geographical tradition arises, will that mean we get all the details of the clothing and such that people are wearing, today? Well…no. Unless, of course, street photographers embrace creativity and develop photo essays that incorporate this in interesting or novel ways. But street photography can include a lot more than just the people, and the history of street photography and the photos we often praise as masterpieces showcase that blurred subjects can generate interesting and exciting and historically-significant images.
One thing that might be worth thinking about is what this will mean for how geographical spaces are created by generative AI in the future. Specifically:
These AI systems will often default to norms based on the weighting of what has been collected in training data. Will they ‘learn’ that some parts of the world are more or less devoid of people based on street photos and so, when generating images of certain jurisdictions, create imagery that is similarly devoid of people? Or, instead, will we see generative imagery that includes people whereas real photos will have to blur or obfuscate them?
Will we see some photographers, at least, take up a blending of the real and the generative, where they capture streets but then use programs to add people into those streetscapes based on other information they collect (e.g., local fashions etc)? Basically, will we see some street photographers adopt a hybrid real/generative image-making process in an effort to comply with law while still adhering to some of the Western norms around street photography?
As a final point, while I identify as a street photographer and avoid taking images of people in distress, the nature of AI regulation and law means that there are indeed some good reasons for people to be concerned about the taking of street photos. The laws frustrating some street photographers are born from arguably real concerns or issues.
For example, companies such as Cleaview AI (in Canada) engaged in the collection of images and, subsequently, generated biometric profiles of people based on scraping publicly available images.
Most people don’t really know how to prevent such companies from being developed or selling their products but do know that if they stop the creation of training data—photographs—then they’re at least less likely to be captured in a compromising or unfortunate situation.
It’s not the photographers, then, that are necessarily ‘bad’ but the companies who illegally exploit our work to our detriment, as well as to the detriment of the public writ large.
All to say: as street photographers, and photographers more generally, we should think broader than our own interests to appreciate why individuals may not want their images taken in light of technical developments that are all around us. And importantly, the difference is that as photographers we do often share our work whereas CCTV cameras and such do not, with the effect that the images we take can end up in generative AI, and non-generative AI training data systems, whereas the cameras that are monitoring all of us always are (currently…) less likely to be feeding the biometric surveillance training data beast.
While, at the same time, recognizing that sometimes a photo is preferred because people are walking away from the camera/towards something else in the scene. ↩︎
Excited Pixels 