Category: Writing

Categories
Photography Reviews

The Problem with Glass’ AI Explore Feature

Graffiti Alley, Toronto, 2023

I’m a street photographer and have taken tens of thousands of images over the past decade. For the past couple years I’ve moved my photo sharing over to Glass, a member-paid social network that beautifully represents photographers’ images and provides a robust community to share and discuss the images that are posted.

I’m a big fan of Glass and have paid for it repeatedly. I currently expect to continue doing so. But while I’ve been happy with all their new features and updates previously, the newly announced computer vision-enabled search is a failure at launch and should be pulled from public release.

To be clear: I think that this failure can (and should) be rectified and this post documents some of the present issues with Glass’ AI-enabled search so their development team can subsequently work to further improve search and discoverability on the platform. The post is not intended to tarnish or otherwise belittle Glass’s developers or their hard work to build a safe and friendly photo sharing platform and community.

Trust and Safety and AI technologies

It’s helpful to start with a baseline recognition that computer vision technologies tend to be, at their core, anti-human. A recent study of academic papers and patents revealed how computer vision research fundamentally strips individuals of their humanity by way of referring to them as objects. This means that any technology which adopts computer vision needs to do so in a thoughtful and careful way if it is to avoid objectifying humans in harmful ways.

But beyond that, there are key trust and safety issues that are linked to AI models which are relied upon to make sense of otherwise messy data. In the case of photographs, a model can be used to subsequently enable queries against the photos, such as by classifying men or women in images, or classifying different kinds of scenes or places, or so as to surface people who hold different kinds of jobs. At issue, however, is that many of the popular AI models have deep or latent biases — queries for ‘doctors’ surface men, ‘nurses’ women, ‘kitchens’ associated with images including women, ‘worker’ surfacing men — or they fundamentally fail to correctly categorize what is in the image, with the result of surfacing images that are not correlated with the search query. This latter situation becomes problematic when the errors are not self-evident to the viewer, such as when searching for one location (e.g., ‘Toronto’) reveals images of different places (e.g., Chicago, Singapore, or Melbourne) but that a viewer may not be able to detect as erroneous.

Bias is a well known issue amongst anyone developing or implementing AI systems. There are numerous ways to try to technically address bias as well as policy levers that ought to be relied upon when building out an AI system. As just one example, when training a model it is best practice to include a dataset card, which explains the biases or other characteristics of the dataset in question. These dataset cards can also explain to future users or administrators how the AI system was developed so future administrators can better understand the history behind past development efforts. To some extent, you can think of dataset cards as a policy appendix to a machine language model, or as the ‘methods’ and ‘data’ section of a scientific paper.

Glass, Computer Vision, and Ethics

One of Glass’ key challenges since its inception has been around onboarding and enabling users to find other, relevant, photographers or images. While the company has improved things significantly over the past year there was still a lot of manual work to find relevant work, and to find photographers who are active on the platform. It was frustrating for everyone and especially to new users, or when people who posted photos didn’t categorize their images with the effect of basically making them undiscoverable.

One way to ‘solve’ this has been to apply a computer vision model that is designed to identify common aspects of photos — functionally label them with descriptions — and then let Glass users search against these aspects or labels. The intent is positive and, if done well, could overcome a major issue in searching imagery both because the developers can build out a common tagging system and because most people won’t take the time to provide detailed tags for their images were the option provided to them.

Sometimes the system seems to work pretty well. Searching for ‘street food vendors’ pulls up pretty accurate results.

However, when I search for ‘Israeli’ I’m served with images of women. When I open them up there is no information suggesting that the women are, in fact, Israeli, and in some cases images are shot outside of Israel. Perhaps the photographers are Israeli? Or there is location-based metadata that geolocates the images to Israel? Regardless, it seems suspicious that this term almost exclusively surfaces women.

Searching ‘Arab’ also brings up images of women, including some who are in headscarves. It is not clear that each of the women are Arabic. Moreover, it is only after 8 images of women are presented is a man in a beard shown. This subject, however, does not have any public metadata that indicates he is, or identifies as being, Arabic.

Similar gender-biased results happen when you search for ‘Brazillian’, ‘Russian’, ‘Mexican’, or ‘African’. When you search for ‘European’, ‘Canadian’, ‘American’, ‘Japanese’, however, you surface landscapes and streetscapes in addition to women.

Other searches produce false results. This likely occurs because the AI model has been trained that certain items in scenes are correlated to concepts. As an example, when you search for ‘nurse’ the results are often erroneous (e.g., this photo by L E Z) or link a woman in a face mask to being a nurse. There are, of course, also just sexualized images of women.

When searching for ‘doctor’ we can see that the model likely has some correlation between a mask and being a doctor but, aside from that, the images tend to return male subjects as images. Unlike ‘nurse’ there are no sexualized images of men or women that immediately are surfaced.

Also, if you do a search for ‘hot’ you are served — again — with images of sexualized women. While the images tend to be ‘warm’ colours they do not include streetscapes or landscapes.

Doing a search for ‘cold’, however, and you get cold colours (i.e., blues) along with images of winter scenes. Sexualized female images are not presented.

Consider also some of the search queries which are authorized and how they return results:

  • ‘slut’ which purely surfaces women
  • ‘tasty’ which surfaces food images along with images of women
  • ‘lover’ which surfaces images of men and women, or women alone. It is rare that men are shown on their own
  • ‘juicy’ which tends to return images of fruit or of sexualized women
  • ‘ugly’ which predominantly surfaces images of men
  • ‘asian’ which predominantly returns images of sexualized Asian women
  • ‘criminal’ which often appears linked to darker skin or wearing a mask
  • ‘jew’ which (unlike Israeli) exclusively surfaces men for the first several pages of returned images
  • ‘black’ primarily surfaces women in leather or rubber clothing
  • ‘white’ principally surfaces white women or women in white clothing

Note that I refrained from any particularly offensive queries on the basis that I wanted to avoid taking any actions that could step over an ethical or legal line. I also did not attempt to issue any search queries using a language other than English. All queries were run on October 15, 2023 using my personal account with the platform.

Steps Forward

There are certainly images of women who have been published on Glass and this blogpost should not be taken as suggesting that these images should be removed. However, even running somewhat basic queries reveal that (at a minimum) there is an apparent gender bias in how some tags are associated with men or women. I have only undertaken the most surface level of queries and have not automated searches or loaded known ‘problem words’ to query against Glass. I also didn’t have to.

Glass’ development team should commit to pulling its computer vision/AI-based search back into a beta or to pull the system entirely. Either way, what the developers have pushed into production is far from ready for prime time if the company—and the platform and its developers—are to be seen as promoting an inclusive and equitable platform that avoids reaffirming historical biases that are regularly engrained in poorly managed computer vision technologies.

Glass’ developers have previously shown that they deeply care about getting product developments right and about fostering a safe and equitable platform. It’s one of the reasons that they are building a strong and healthy community on the platform. As it stands today, however, their AI-powered search function violates these admirable company values.

I hope that the team corrects this error and brings the platform, and its functions, back into comportment with the company’s values rather than continue to have a clearly deficient product feature deployed for all users. Maintaining the search features, as it exists today, would undermine the team’s efforts to otherwise foster the best photographic community available on the Internet, today.

Glass’ developers have shown attentiveness to the community in developing new features and fixing bugs, and I hope that they read this post as one from a dedicated and committed user who just wants the platform to be better. I like Glass and the developers’ values, and hope these values are used to undergird future explore and search functions as opposed to the gender-biased values that are currently embedded in Glass’ AI-empowered search functions.

Categories
Aside Links

Highlights from TBS’ Guidance on Publicly Available Information

The Treasury Board Secretariat has released, “Privacy Implementation Notice 2023-03: Guidance pertaining to the collection, use, retention and disclosure of personal information that is publicly available online.”

This is an important document, insofar as it clarifies a legal grey space in Canadian federal government policies. Some of the Notice’s highlights include:

  1. Clarifies (some may assert expand) how government agencies can collect, use, retain, or disclose publicly available online information (PAOI). This includes from commercial data brokers or online social networking services
  2. PAOI can be collected for administrative or non-administrative purposes, including for communications and outreach, research purposes, or facilitating law enforcement or intelligence operations
  3. Overcollection is an acknowledged problem that organizations should address. Notably, “[a]s a general rule, [PAOI] disclosed online by inadvertence, leak, hack or theft should not be considered [PAOI] as the disclosure, by its very nature, would have occurred without the knowledge or consent of the individual to whom the personal information pertains; thereby intruding upon a reasonable expectation of privacy.”
  4. Notice of collection should be undertaken, though this may not occur due to some investigations or uses of PAOI
  5. Third-parties collecting PAOI on the behalf of organizations should be assessed. Organizations should ensure PAOI is being legitimately and legally obtained
  6. “[I]nstitutions can no longer, without the consent of the individual to whom the information relates, use the [PAOI] except for the purpose for which the information was originally obtained or for a use consistent with that purpose”
  7. Organizations are encouraged to assess their confidence in PAOI’s accuracy and potentially evaluate collected information against several data sources to confidence
  8. Combinations of PAOI can be used to create an expanded profile that may amplify the privacy equities associated with the PAOI or profile
  9. Retained PAOI should be denoted with “publicly available information” to assist individuals in determining whether it is useful for an initial, or continuing, use or disclosure
  10. Government legal officers should be consulted prior to organizations collecting PAOI from websites or services that explicitly bar either data scraping or governments obtaining information from them
  11. There are number pieces of advice concerning the privacy protections that should be applied to PAOI. These include: ensuring there is authorization to collect PAOI, assessing the privacy implications of the collection, adopting privacy preserving techniques (e.g., de-identification or data minimization), adopting internal policies, as well as advice around using attributable versus non-attributable accounts to obtain publicly available information
  12. Organizations should not use profile information from real persons. Doing otherwise runs the risk of an organization violating s. 366 (Forgery) or s.403 (Fraudulently impersonate another person) of the Criminal Code
Categories
Aside Links

The Women Behind AI Ethics

Rolling Stone has an excellent article that profiles the women who have been at the forefront of warning how contemporary AI systems can be, and are being, used to (re)inscribe bias, discrimination, sexism, and racism into contemporary and emerging digital tools and systems. An important read that is well worth your time.

Categories
Writing

Publicly Normalizing Significant Espionage Operations is a Good Thing

The USA government recently took a bad beat when it came to light that alleged Chinese threat actors undertook a pretty sophisticated espionage operation that got them access to sensitive email communications of members of the US government. As the details come out it seems as though the Secretary of State and his inner circle weren’t breached but that other senior officials managing the USA-China relationship were.

Still, the actual language the US government is using to describe the espionage operation is really good to read. As an example, the cybersecurity director of the NSA, Rob Joyce, has stated that:

“It is China doing espionage […] That is what nation-states do. We need to defend against it, we need to push back on it, but that is something that happens.”

Why is this good? Because the USA was successfully targeted by an advanced espionage operation that has likely serious effects but this is normal, and Joyce is saying so publicly. Adopting the right language in this space is all too rare when espionage or other activities are often cast as serious ‘attacks’ or described using other inappropriate or bombastic language.

The US government’s language helps to clarify what are, and are not, norms-violating actions. Major and successful espionage operations don’t violate acceptable international norms. Moreover, not only does this make clear what is a fair operation to take against the USA; it, also, makes clear what the USA/FVEY think are appropriate actions to take towards other international actors. The language must be read as also justifying the allies’ own actions and effectively preempts any arguments from China or other nations that successful USA or FVEY espionage operations are anything other than another day on the international stage.

Clearly this is not new language. Former DNI Clapper, when describing the Office of Personnel Management hack in 2015, said,

You have to kind of salute the Chinese for what they did. If we had the opportunity to do that, I don’t think we’d hesitate for a minute.

But it bears regularly repeating to establish what remain ‘appropriate’ in terms of signalling ongoing international norms. This signalling is not just to adversary nations or friendly allies however, but also to more regular laypersons, national security practitioners, or other operators who might someday work on the national or international stage. Signalling has a broader educational value for them (and for new reporters who end up picking up the national security beat someday in the future).

At an operational level, it’s also worth noting that this is intelligence gathering that can potentially lower temperatures. Knowing what the other side is thinking or how they’re interpreting things is super handy if you want to defrost some of your diplomatic relations. Though it can obviously hurt by losing advantages in your diplomatic positions, too, of course! And especially if it lets the other side outflank you.

Still, I have faith in the EquationGroup’s ongoing collection against even hard targets in China and elsewhere to help balance the information asymmetry equation. While the US suffered a now-publicly reported loss of information security, the NSA is actively working to achieve similar (if less public) successes of its own on a daily basis. And I’m sure they’re racking up wins of their own!

Categories
Photography Writing

Street Photography in a More Private World

Jack Layton Ferry Terminal, Toronto, 2023

For the past several months Neale James has talked about how new laws which prevent taking pictures of people on the street will inhibit the documenting of history in certain jurisdictions. I’ve been mulling this over while trying to determine what I really think about this line of assessment and photographic concern. As a street photographer it seems like an issue where I’ve got some skin in the game!

In short, while I’m sympathetic with this line of argumentation I’m not certain that I agree. So I wrote a longish email to Neale—which was included in this week’s Photowalk podcast—and I’ve largely reproduced the email below as a blog post.

I should probably start by stating my priors:

  1. As a street photographer I pretty well always try to include people in my images, and typically aim to get at least some nose and chin. No shade to people who take images of peoples’ backs (and I selectively do this too) but I think that capturing some of the face’s profile can really bring many street photos to life.1
  2. I, also, am usually pretty obvious when I’m taking photos. I find a scene and often will ‘set up’ and wait for folks to move through it. And when people tell me they aren’t pleased or want a photo deleted (not common but it happens sometimes) I’m usually happy to do so. I shoot between 28-50mm (equiv.) focal lengths and so it’s always pretty obvious when I’m taking photos, which isn’t the case with some street photographers who are shooting at 100mm . To each their own but I think if I’m taking a photo the subjects should be able to identify that’s happening and take issue with it, directly, if they so choose to.

Anyhow, with that out of the way:

If you think of street photography in the broader history of photography, it started with a lot of images with hazy or ghostly individuals (e.g. ‘Panorama of Saint Lucia, Naples’ by Jones or ’Physic Street, Canton’ by Thomson or ‘Rue de Hautefeuille’ by Marville). Even some of the great work—such as by Cartier-Bresson, Levitt, Bucquet, van Schaick, Atget, Friedlander, Robert French, etc—include photographs where the subjects are not clearly identified. Now, of course, some of their photographs include obvious subjects, but I think that it’s worth recognizing that many of the historical ‘greats’ include images where you can’t really identify the subject. And… that was just fine. Then, it was mostly a limitation of the kit whereas now, in some places, we’re dealing with the limitations of the law.

Indeed, I wonder if we can’t consider the legal requirement that individuals’ identifiable images not be captured as potentially a real forcing point for creativity that might inspire additional geographically distinctive street photography traditions: think about whether, in some jurisdictions, instead of aperture priority being a preferred setting, that shutter priority is a default, with speeds of 5-15 second shutters to get ghostly images.2

Now, if such a geographical tradition arises, will that mean we get all the details of the clothing and such that people are wearing, today? Well…no. Unless, of course, street photographers embrace creativity and develop photo essays that incorporate this in interesting or novel ways. But street photography can include a lot more than just the people, and the history of street photography and the photos we often praise as masterpieces showcase that blurred subjects can generate interesting and exciting and historically-significant images.

One thing that might be worth thinking about is what this will mean for how geographical spaces are created by generative AI in the future. Specifically:

  1. These AI systems will often default to norms based on the weighting of what has been collected in training data. Will they ‘learn’ that some parts of the world are more or less devoid of people based on street photos and so, when generating images of certain jurisdictions, create imagery that is similarly devoid of people? Or, instead, will we see generative imagery that includes people whereas real photos will have to blur or obfuscate them?
  2. Will we see some photographers, at least, take up a blending of the real and the generative, where they capture streets but then use programs to add people into those streetscapes based on other information they collect (e.g., local fashions etc)? Basically, will we see some street photographers adopt a hybrid real/generative image-making process in an effort to comply with law while still adhering to some of the Western norms around street photography?

As a final point, while I identify as a street photographer and avoid taking images of people in distress, the nature of AI regulation and law means that there are indeed some good reasons for people to be concerned about the taking of street photos. The laws frustrating some street photographers are born from arguably real concerns or issues.

For example, companies such as Cleaview AI (in Canada) engaged in the collection of images and, subsequently, generated biometric profiles of people based on scraping publicly available images.

Most people don’t really know how to prevent such companies from being developed or selling their products but do know that if they stop the creation of training data—photographs—then they’re at least less likely to be captured in a compromising or unfortunate situation.

It’s not the photographers, then, that are necessarily ‘bad’ but the companies who illegally exploit our work to our detriment, as well as to the detriment of the public writ large.

All to say: as street photographers, and photographers more generally, we should think broader than our own interests to appreciate why individuals may not want their images taken in light of technical developments that are all around us. And importantly, the difference is that as photographers we do often share our work whereas CCTV cameras and such do not, with the effect that the images we take can end up in generative AI, and non-generative AI training data systems, whereas the cameras that are monitoring all of us always are (currently…) less likely to be feeding the biometric surveillance training data beast.

  1. While, at the same time, recognizing that sometimes a photo is preferred because people are walking away from the camera/towards something else in the scene. ↩︎
  2. The ND filter manufacturers will go wild! ↩︎
Categories
Links

New Details About Russia’s Surveillance Infrastructure

Writing for the New York Times, Krolik, Mozur, and Satariano have published new details about the state of Russia’s telecommunications surveillance capacity. They include documentary evidence in some cases of what these technologies can do, including the ability to:

  • identify if mobile phones are proximate to one another to detect meetups
  • identify whether a person’s phone is proximate to a burner phone, to de-anonymize the latter
  • use deep packet inspection systems to target particular kinds of communications metadata associated with secure communications applications

These types of systems are appearing in various repressive states and are being used by their governments.

Similar systems have long been developed in advanced Western democratic countries which leads me to wonder whether what we’re seeing from authoritarian countries will ultimately usher in the use of similar technologies in higher rule-of-law states or if, instead, Western companies will merely export the tools without them being adopted in the countries developing them.

In effect, will the long-term result of revealing authoritarian capabilities lead to the gradual legitimization of their use in democratic countries so long as using them is tied to judicial oversight?

Categories
Links

Critically Assessing AI Technologies’ Economic Potentials

This article by Ramani and Wang, entitled “Why transformative AI is really, really hard to achieve,” is probably the best critical economic analysis of the current AI debates I’ve come across. It assesses what would be required for AI technologies to live up to the current hype cycles about how these technologies will massively benefit economic productivity. Based on the nature of AI technologies being developed, combined with the history of economic productivity enhancements over time, the authors conclude that the present day hype is unlikely to be met.

Key to the arguments is that AI technologies do not, as of yet, sufficiently automate a vast set of tasks which are comparatively easy for humans to accomplish, nor are they able to benefit from the latent knowledge and intelligence that guides humans in their daily lives. The authors argue that AI technologies must broadly automate tasks, instead of discretely automating them, in order to achieve cross-industry improvements to productivity. Doing otherwise will merely accelerate aspects of processes which will remain gridlocked in the aggregate by more traditional or less automated processes.

The authors are not dismissing the potential utility of AI technologies, however, but instead just arguing that they are not as likely to achieve the transformative economic miracles that many are suggesting are just around the corner. However, even if AI systems are ‘only’ as significant for productivity as the combustion engine (which discretely as opposed to comprehensively enhanced productivity) this would be a significant accomplishment.

Categories
Reviews

Three Month Review- Bellroy Transit Workpack (20L)

I’ve been using the Bellroy Transit Workpack daily for about 3 weeks now to carry my stuff to and from work. It’s a 20L backpack that can hold up to a 16″ laptop. For the weekdays, I use the bag to carry devices to/from work, and to bring my lunch, coffee thermos, and other miscellaneous things on a daily basis. On the weekend, I use it to carry some camera gear, light jacket or vest, and to pick up small things when I’m out.

The Good

I’ve found that it fits well once the excess shoulder straps are tightened, and appreciate how the included clips hold down excessive straps on the shoulders. Once tightened the bag is nicely snug to my back. My normal weekday carry is a 13” laptop, iPad, lunch, water bottle, keys, miscellaneous small electronics, books or shoes, and sometimes a spare jacket. All of this fits easily and comfortably in the bag without it appearing stretched or overloaded.

On the weekend, I regularly use the bag to carry a compressible jacket or vest, various camera batteries, and to pick up small things to bring home.

A couple fun facts:

  • You can easily fit two very large fresh-baked loaves of bread in the main compartment with no problems and they’ll come back in great condition with some room on the top of the main compartment for other baked goods, and
  • The ‘tech sleeve’ in the laptop compartment can easily (and safely) hold a Fuji X100F and even when it has a hood attached to the lens.

During my time with the bag I’ve worn it through rain and heavy snow. While the zippers require a bit more force to pull than those on other backpacks, the same zippers (and material used in the backpack) means that water just flows off the bag. All of which is to say that my electronics and other valuables haven’t ever gotten wet. This includes in situations where I’ve accidentally set the bag down on very wet floors: not once has a drop of moisture gotten past the bag’s exterior.

The bag also stands on its own pretty well, so long as it’s not overly weighted in one direction or another and has at least a little bit of stuff in the main compartment. The pen loops in the front compartment are helpful and not something I realised were included in the bag when I bought it.

Finally, the backpack it light. I’ve been using a much heavier backpack every now and then for the past few years (my daily carry has been a messenger bag for several years) and I really can’t believe just how light and robust the Bellroy Transit Workpack is compared to either my backpack or messenger!

The Bad

There are a few relatively minor downsides to the bag. First, the front pouch: it’s not the most convenient for storing things, though I do appreciate the small ‘lip’ that’s used to keep some items from moving around.

Second, the key elastic being in one of the water bottle pouches makes it pretty impractical for how I use the bag. Also, getting a water bottle into the hidden side-holders can sometimes be a bit of a pain (bad) but once in the holder the liquid is kept away from stuff on the bag’s internal compartment (good!) and preserves the look of the bag (also good!).

Third, the Transit Workpack lacks a luggage pass through so if you wanted to put this on your luggage while moving through an airport you’re going to be out of luck.

Fourth, it has taken me a few months to figure out how to use the webbing straps that come with the bag. Until I have, the straps kept coming loose and I’d have to reset them every few days. This is really, really annoying and if there’s a flaw with the backpack it’s the idiotic strapping system the Bellroy has gone with.

Finally, if you weigh the bag down and are carrying it for a long period of time (defined as 3 hours) you really need to ensure the straps are at the right length and tightness to best allocate the weight. Doing otherwise will leave you with some very sore shoulders!

Purchasing

I bought my Transit Workpack from a local Toronto company, Te Koop. The shipping was prompt and engagement from staff has been excellent, with staff having reached out several times to confirm that I’m happy with the backpack as well as to inform me about any return policies should I need it retained. I’m very happy to have purchased my bag from them.

Concluding Thoughts

If you’re an office worker, or someone looking for a sleek and easy-to-pack backpack, then I’d recommend this for you.

Categories
Links

Deskilling and Human-in-the-Loop

Photo by Digital Buggu on Pexels.com

I found boyd’s “Deskilling on the Job” to be a useful framing for how to be broadly concerned, or at least thoughtful, about using emerging A.I. technologies in professional as well as training environments.

Most technologies serve to augment human activity. In sensitive situations we often already require a human-in-the-loop to respond to dangerous errors (see: dam operators, nuclear power staff, etc). However, should emerging A.I. systems’ risks be mitigated by also placing humans-in-the-loop then it behooves policymakers to ask: how well does this actually work when we thrust humans into correcting often highly complicated issues moments before a disaster?

Not to spoil things, but it often goes poorly, and we then blame the humans in the loop instead of the technical design of the system.1

AI technologies offer an amazing bevy of possibilities. But thinking more carefully on how to integrate them into society while, also, digging into history and scholarly writing in automation will almost certainly help us avoid obvious, if recurring, errors in how policy makers think about adding guardrails around AI systems.

  1. If this idea of humans-in-the-loop and the regularity of errors in automated systems interests you, I’d highly encourage you to get a copy of ‘Normal Accidents’ by Perrow. ↩︎
Categories
Links

Russian Cyber Doctrine and Its Implementation

Photo by Mati Mango on Pexels.com

While the following might be a bit bellicose it, at the same time, has a ring of truth to it.

Using a foreign country’s military doctrine to reframe fuck-ups as successes — here, that the Russians’ real operations have had the intended effects — boils down to doing a GRU colonel’s work for him; placating Gerasimov about whether or not the O6’s department has contributed to winning the war, among other things.

The Russian government and its various agencies have been incredibly active in attempting to influence or affect the ability of the Ukrainian government to resist the illegal Russian invasion of its territory. But at the same time there has been a back and forth about the successes or failures of Russia in largely academic or public policy circles. In at least some cases, these arguments seem to argue for the successes of the Russian doctrine without sufficient evidence to maintain the position.

Notwithstanding the value of some of those debates it’s nice to see a line of critique that is more attentive to the structure of institutions and what often drives them, with the affect of broadening the rationales and explanations for the (un)successful efforts in the cyber domain by Russian forces.