A new study from the London School of Economics highlights how AI systems can reinforce existing inequalities when used for high risk activities like social care.
Writing in The Guardian, Jessica Murray describes how Google’s Gemma model summarized identical case notes differently depending on gender.
An 84-year-old man, “Mr Smith,” was described as having a “complex medical history, no care package and poor mobility,” while “Mrs Smith” was portrayed as “[d]espite her limitations, she is independent and able to maintain her personal care.” In another example, Mr Smith was noted as “unable to access the community,” but Mrs Smith as “able to manage her daily activities.”
These subtle but significant differences risk making women’s needs appear less urgent, and could influence the care and resources provided. By contrast, Meta’s Llama 3 did not use different language based on gender, underscoring that bias can vary across models and the need to measure bias in LLMs adopted for public service delivery
These findings reinforce why AI systems must be valid and reliable, safe, transparent, accountable, privacy-protective, and human-rights affirming. This is especially the case in high risk settings where AI systems affect decisions linked with accessing essential public services.
I’m a street photographer and have taken tens of thousands of images over the past decade. For the past couple years I’ve moved my photo sharing over to Glass, a member-paid social network that beautifully represents photographers’ images and provides a robust community to share and discuss the images that are posted.
I’m a big fan of Glass and have paid for it repeatedly. I currently expect to continue doing so. But while I’ve been happy with all their new features and updates previously, the newly announced computer vision-enabled search is a failure at launch and should be pulled from public release.
To be clear: I think that this failure can (and should) be rectified and this post documents some of the present issues with Glass’ AI-enabled search so their development team can subsequently work to further improve search and discoverability on the platform. The post is not intended to tarnish or otherwise belittle Glass’s developers or their hard work to build a safe and friendly photo sharing platform and community.
Trust and Safety and AI technologies
It’s helpful to start with a baseline recognition that computer vision technologies tend to be, at their core, anti-human. A recent study of academic papers and patents revealed how computer vision research fundamentally strips individuals of their humanity by way of referring to them as objects. This means that any technology which adopts computer vision needs to do so in a thoughtful and careful way if it is to avoid objectifying humans in harmful ways.
But beyond that, there are key trust and safety issues that are linked to AI models which are relied upon to make sense of otherwise messy data. In the case of photographs, a model can be used to subsequently enable queries against the photos, such as by classifying men or women in images, or classifying different kinds of scenes or places, or so as to surface people who hold different kinds of jobs. At issue, however, is thatmanyofthe popular AI models have deep or latent biases — queries for ‘doctors’ surface men, ‘nurses’ women, ‘kitchens’ associated with images including women, ‘worker’ surfacing men — or they fundamentally fail to correctly categorize what is in the image, with the result of surfacing images that are not correlated with the search query. This latter situation becomes problematic when the errors are not self-evident to the viewer, such as when searching for one location (e.g., ‘Toronto’) reveals images of different places (e.g., Chicago, Singapore, or Melbourne) but that a viewer may not be able to detect as erroneous.
Bias is a well known issue amongst anyone developing or implementing AI systems. There are numerous ways to try to technically address bias as well as policy levers that ought to be relied upon when building out an AI system. As just one example, when training a model it is best practice to include a dataset card, which explains the biases or other characteristics of the dataset in question. These dataset cards can also explain to future users or administrators how the AI system was developed so future administrators can better understand the history behind past development efforts. To some extent, you can think of dataset cards as a policy appendix to a machine language model, or as the ‘methods’ and ‘data’ section of a scientific paper.
Glass, Computer Vision, and Ethics
One of Glass’ key challenges since its inception has been around onboarding and enabling users to find other, relevant, photographers or images. While the company has improved things significantly over the past year there was still a lot of manual work to find relevant work, and to find photographers who are active on the platform. It was frustrating for everyone and especially to new users, or when people who posted photos didn’t categorize their images with the effect of basically making them undiscoverable.
One way to ‘solve’ this has been to apply a computer vision model that is designed to identify common aspects of photos — functionally label them with descriptions — and then let Glass users search against these aspects or labels. The intent is positive and, if done well, could overcome a major issue in searching imagery both because the developers can build out a common tagging system and because most people won’t take the time to provide detailed tags for their images were the option provided to them.
Sometimes the system seems to work pretty well. Searching for ‘street food vendors’ pulls up pretty accurate results.
However, when I search for ‘Israeli’ I’m served with images of women. When I open them up there is no information suggesting that the women are, in fact, Israeli, and in some cases images are shot outside of Israel. Perhaps the photographers are Israeli? Or there is location-based metadata that geolocates the images to Israel? Regardless, it seems suspicious that this term almost exclusively surfaces women.
Searching ‘Arab’ also brings up images of women, including some who are in headscarves. It is not clear that each of the women are Arabic. Moreover, it is only after 8 images of women are presented is a man in a beard shown. This subject, however, does not have any public metadata that indicates he is, or identifies as being, Arabic.
Similar gender-biased results happen when you search for ‘Brazillian’, ‘Russian’, ‘Mexican’, or ‘African’. When you search for ‘European’, ‘Canadian’, ‘American’, ‘Japanese’, however, you surface landscapes and streetscapes in addition to women.
Other searches produce false results. This likely occurs because the AI model has been trained that certain items in scenes are correlated to concepts. As an example, when you search for ‘nurse’ the results are often erroneous (e.g., this photo by L E Z) or link a woman in a face mask to being a nurse. There are, of course, also just sexualized images of women.
When searching for ‘doctor’ we can see that the model likely has some correlation between a mask and being a doctor but, aside from that, the images tend to return male subjects as images. Unlike ‘nurse’ there are no sexualized images of men or women that immediately are surfaced.
Also, if you do a search for ‘hot’ you are served — again — with images of sexualized women. While the images tend to be ‘warm’ colours they do not include streetscapes or landscapes.
Doing a search for ‘cold’, however, and you get cold colours (i.e., blues) along with images of winter scenes. Sexualized female images are not presented.
Consider also some of the search queries which are authorized and how they return results:
‘slut’ which purely surfaces women
‘tasty’ which surfaces food images along with images of women
‘lover’ which surfaces images of men and women, or women alone. It is rare that men are shown on their own
‘juicy’ which tends to return images of fruit or of sexualized women
‘ugly’ which predominantly surfaces images of men
‘asian’ which predominantly returns images of sexualized Asian women
‘criminal’ which often appears linked to darker skin or wearing a mask
‘jew’ which (unlike Israeli) exclusively surfaces men for the first several pages of returned images
‘black’ primarily surfaces women in leather or rubber clothing
‘white’ principally surfaces white women or women in white clothing
Note that I refrained from any particularly offensive queries on the basis that I wanted to avoid taking any actions that could step over an ethical or legal line. I also did not attempt to issue any search queries using a language other than English. All queries were run on October 15, 2023 using my personal account with the platform.
Steps Forward
There are certainly images of women who have been published on Glass and this blogpost should not be taken as suggesting that these images should be removed. However, even running somewhat basic queries reveal that (at a minimum) there is an apparent gender bias in how some tags are associated with men or women. I have only undertaken the most surface level of queries and have not automated searches or loaded known ‘problem words’ to query against Glass. I also didn’t have to.
Glass’ development team should commit to pulling its computer vision/AI-based search back into a beta or to pull the system entirely. Either way, what the developers have pushed into production is far from ready for prime time if the company—and the platform and its developers—are to be seen as promoting an inclusive and equitable platform that avoids reaffirming historical biases that are regularly engrained in poorly managed computer vision technologies.
Glass’ developers have previously shown that they deeply care about getting product developments right and about fostering a safe and equitable platform. It’s one of the reasons that they are building a strong and healthy community on the platform. As it stands today, however, their AI-powered search function violates these admirable company values.
I hope that the team corrects this error and brings the platform, and its functions, back into comportment with the company’s values rather than continue to have a clearly deficient product feature deployed for all users. Maintaining the search features, as it exists today, would undermine the team’s efforts to otherwise foster the best photographic community available on the Internet, today.
Glass’ developers have shown attentiveness to the community in developing new features and fixing bugs, and I hope that they read this post as one from a dedicated and committed user who just wants the platform to be better. I like Glass and the developers’ values, and hope these values are used to undergird future explore and search functions as opposed to the gender-biased values that are currently embedded in Glass’ AI-empowered search functions.
Rolling Stone has an excellent article that profiles the women who have been at the forefront of warning how contemporary AI systems can be, and are being, used to (re)inscribe bias, discrimination, sexism, and racism into contemporary and emerging digital tools and systems. An important read that is well worth your time.
Ikea Canada notified approximately 95,000 Canadian customers in recent weeks about a data breach the company has suffered. An Ikea employee conducted a series of searches between March 1 to March 3 which surfaced the account records of the aforementioned customers.1
While Ikea promised that financial information–credit card and banking information–hadn’t been revealed a raft of other personal information had been. That information included:
full first and last name;
postal code or home address;
phone number and other contact information;
IKEA loyalty number.
Ikea did not disclose who specifically accessed the information nor their motivations for doing so.
The notice provided by Ikea was better than most data breach alerts insofar as it informed customers what exactly had been accessed. For some individuals, however, this information is highly revelatory and could cause significant concern.
For example, imagine a case where someone has previously been the victim of either physical or digital stalking. Should their former stalker be an Ikea employee the data breach victim may ask whether their stalker now has confidential information that can be used to renew, or further amplify, harmful activities. With the customer information in hand, as an example, it would be relatively easy for a stalker to obtain more information such as where precisely someone lived. If they are aggrieved then they could also use the information to engage in digital harassment or threatening behaviour.
Without more information about the motivations behind why the Ikea employee searched the database those who have been stalked or had abusive relations with an Ikea employee might be driven to think about changing how they live their lives. They might feel the need to change their safety habits, get new phone numbers, or cycle to a new email. In a worst case scenario they might contemplate vacating their residence for a time. Even if they do not take any of these actions they might experience a heightened sense of unease or anxiety.
Of course, Ikea is far from alone in suffering these kinds of breaches. They happen on an almost daily basis for most of us, whether we’re alerted of the breach or not. Many news reports about such breaches focus on whether there is an existent or impending financial harm and stop the story there. The result is that journalist reporting can conceal some of the broader harms linked with data breaches.
Imagine a world where our personal information–how you can call us or find our homes–was protected equivalent to how our credit card numbers are current protected. In such a world stalkers and other abusive actors might be less able to exploit stolen or inappropriately accessed information. Yes, there will always be ways by which bad actors can operate badly, but it would be possible to mitigate some of the ways this badness can take place.
Companies could still create meaningful consent frameworks whereby some (perhaps most!) individuals could agree to have their information stored by the company. But, for those who have a different risk threshold they could make a meaningful choice so they could still make purchases and receive deliveries without, at the same time, permanently increasing the risks that their information might fall into the wrong hand. However, getting to this point requires expanded threat modelling: we can’t just worry about a bad credit card purchase but, instead, would need to take seriously the gendered and intersectional nature of violence and its intersection with cybersecurity practices.
In the interests of disclosure, I was contacted as an affected party by Ikea Canada. ↩︎
Content moderation is something that is fraught with challenges; is too much speech being blocked as a result? Too little bad speech that harms others being permitted? Does moderation enable political actors to distort the public sphere? Enable corporations to advance their interests at the expense of competitors and innovators?
These are all important elements of the ‘content moderation’ debate. But it’s not what has me thinking about moderation at the moment. Instead, it was a more localized environment — a conference setting — that left me with a bad taste in my mouth because of how things were not moderated. Specifically, in a situation where there were only men on a panel addressing threats to electoral processes, another older white man asked how society should deal with cases where women accuse politicians of sexual impropriety, abuse, or other misdeeds: how do we deal with such threats to the political process that run the risk of undermining white men’s abilities to run for office?
The panelists muddled through the question/statement and noted how these disruptions could be challenging for electoral processes. None asserted, as panelists, that women do not tend to allege such activities unless they genuinely happened; women know the costs of making even the most absolutely best-founded allegation, insofar as society will demonize the accuser and tend to shield or defend whomever is accused. Moreover, while an accuser may suffer for the rest of their life for raising the allegations — they may be less likely to be employed, as an example — the accused tends to be fine: they can re-enter society after a minimal ‘cooling off’ period and shrug off the allegation or accusation.
So I was annoyed by the panelists and their decision(s) to not engage with the question head on. But I was most upset that the moderator for the panel didn’t just slap down the ‘question’ and move on: the very fact that the question was upheld as legitimate by the moderator showcased the structural problem that continues to face women who merely want to declare that given persons are, or have been, dangerous. Moderation at the global scale carries with it a unique set of challenges — noted in the opening paragraph — whereas in more localized settings those challenges are remarkably less problematic. It was deeply disappointing that in such a localized setting male white privilege was permitted to reign supreme, with no moderation, though it did affirm to me — and made much clearer — that panel moderators and panelists themselves need to be more affirmative in not accepting the premise of the question in the first place.
And, failing a willingness to stand up and push back against questions that raise doubt about women’s experiences, how people react to such questions at least indicate which men are not committed to equity in a meaningful sense and, as such, are not persons who strike me as suitable to collaborate with on current projects. I just don’t think that I could, or would want to, work with someone who carries a latent suspicion of women either consciously or unconsciously.1 That’s a value-set that I cannot appreciate or understand, and think is fundamentally the latent set of values that had led to the passive approval of individuals and political parties which are substantially committed to the supremacy of (white) men over all other persons in the political commons. And that’s a kind of value-set that needs to be stamped out and have a stake driven through its ideological heart.
Inspiring Quotation of the Week
“Power concedes nothing without a demand. It never did and it never will.”
– Fredrick Douglass
Piece of Poetry
Love after love
The time will come
when, with elation,
you will greet yourself arriving
at your own door, in your own mirror,
and each will smile at the other’s welcome,
and say, sit here. Eat.
You will love again the stranger who was your self.
Give wine. Give bread. Give back your heart
to itself, to the stranger who has loved you
all your life, whom you ignored
for another, who knows you by heart.
Take down the love letters from the bookshelf,
the photographs, the desperate notes,
peel your own image from the mirror.
Sit. Feast on your life.
– Derek Walcott
Great Photography Shots
Grey Chow’s astrophotgraphy is absolutely stunning; looking at it, it makes clear that the universe is so much larger than we imagine and surrounds us, though often in ways in which our sense of time prevents us from immediately perceiving.
Music I’m Digging
Tom Misch – Reverie (EP) // Misch has a kind of jazzy album which I’m enjoying listening to when I’m cooking or reading, or just generally want to generate a downtempo mood in my home. It’s not the most magical of sounds but it is pleasant to have playing in my backgrounds.
Logic’s Bobby Tarantino, Bobby Tarantino II, and YSIV // Logic is a rapper who came from Maryland and, for his first few years, thrived principally on mixtapes. The character/play of the Bobby Tarantino series showcases both a kind of nihilism in the lyrics as well as solid rhythms and poetic inflections, and strong homages to the classic eras of west coast wrap. YSIV has a series of tracks that I’m absolutely captured by: Wu Tang Forever is one of the best Wu Tang songs from the past decade or so, 100 Miles and Running speaks to the challenges and triumphs that come with success, and the final track on the album — Last Call — is a really beautiful story of his life and what he went through to become where he is now. I’ve been listening to logic on near-constant replay for a week and I’m still just picking out more depth and appreciation for the work he’s doing.
Abir – Mint (EP) // I’ve been listening to Abir’s 2017 album over a series of playlists for over a year, but it just never struck me that it was part of the same album. That’s not because it lacks cohesion — it does! — and more that I just hadn’t paid sufficient attention to link everything together. The album significantly speaks to being alone, or single, and surviving in the world. Survival, I think, is probably the right word: Tango, Young & Rude, and Finest Hour all speak to the challenges that can arise especially following challenging relationships, or even preceding new ones. The album, on the whole, feels cohesive and as though it could also be merged with a larger series of works to create a narrative arc of relationships through a full album.
Neat Podcast Episodes
Modern Love – I Was Hardly the Perfect Fit // This podcast, about a distant father trying to connect with his son who lived with his ex-wife, resonated with me; though the relationship that I had with my own father was notably different, elements of the story sounded similar to the relationship that I had with my own dad. The ending — where the strength of their present relationship was revealed — was painful: it was exactly the kind of relationship I’d have dreamed to one day built with my own father.
Lawfare – Jim Baker on AI and Counterintelligence // Jim has a good, broad, assessment of the counterintelligence challenges associated with AI technologies. He isn’t a technologist so the assessment of AI is pretty high level/superficial at the technical level, but the analysis of ways that foreign state actors might interfere with or compromise the development of domestic (USA in this case) AI systems, algorithms, and technologies is relatively comprehensive. It’s a useful listen if you want a good and fast intro to some of the challenges in this space.
Good Reads for the Week
Four Hundred And Eighty Two (On Vulnerability // I found this transcription of David Whyte to be beautiful and powerful; the thrust is to unpack what is vulnerability and why it’s not something to run from but to embrace. Fundamentally our relationships, at their core, are best when they involve committing to vulnerability to one another. The pursuit of vulnerable relationships is the pursuit of relationships that matter the most, and resonate the most, throughout the course of our lives.
How to get that great “hoppy” beer taste without the exploding bottles // Jennifer Ouellette has a cool story of how Brewer-scientists figured out how dry hopping beer leads to refermentation and, by extension, increases to pressure in cans and bottles. Specifically, brewers can add hops after the heated fermentation process to impart flavours but without significantly contributing to the bitterness that is often associated with hops when they are heated. The culprit to the refermentation was found and that may mean there are fewer exploding dry hopped beers on shelves and homes as brewers take the results to heart.
Why Public Transportation Works Better Outside the U.S. // While I may disagree with some of the cheery assessment of Toronto’s transit infrastructure, English’s article nicely summarized the core differences between transit systems in the United States and other jurisdictions around the world. Key is that investment never has stopped in other jurisdictions and urban planners have built transit with the idea of people and businesses then coming to encircle the transit hubs, as opposed to trying to build hubs into existing urban infrastructure.
Senate Truce Collapses as G.O.P. Rush to Confirm More Judges Begins Anew // The norms of governance continue to be challenged by the GOP while they seek to transform the quality and types of justice that will likely be meted out in the coming decades. The systematic stacking of the judiciary with Republicans will mean that even should Democrats manage to disrupt and undue the GOP’s gerrymandering and voter suppression efforts, that any legislation they pass will likely undergo undue scrutiny and hostility from an increasingly politicized judicial branch of government.
Eight Stories of Men’s Regret // This set of eight stories demonstrate different levels of sexual aggressiveness or assault or inappropriate behaviour. They’re not the sole worst kinds of stories that exist but, in a way, that’s what makes them most significant: they are public revelations of the misdeeds of men that reflect their failures and, in some cases, the social pressures that led to their misdeeds. Those pressures do not excuse the behaviours, nor do they justify them. They do, however, provide a mirror upon which men can see themselves, through these other men, in questioning their own pasts and considering how to engage with other persons in the future.
Collapse of ancient city’s water system may have led to its demise // The failure of Angkor’s irrigation and water delivery system is a warning that societies are typically ill-suited to deal with massive changes in weather, let alone climate. It can and should be read as a herald of what may come six centuries later as our politicians and publics steadfastly fail to address the real, serious, and imminent threats posed by climate change.
The Goal in Love // I like this essay because it asserts we should be seeking ourselves, first, in our relationships as opposed to trying to find ourselves in the persons we enter into relationships with. Indeed, if I can think of single major lesson I’ve learned in the past few years it is the importance of accepting yourself and not depending on others to enable such acceptance; it’s by being comfortable with ourselves as whole persons that we are able to engage in wholesome relationships with one another.
When to Open a Bottle: Aging Wine Without the Anxiety // While I move too often to even contemplate what it would be like to cellar a wine for a decade or more — let along have the space to do so! — this article from the New York Times is helpful in guiding a novice through the process of properly investing, aging, and testing wines that have been cellared.
The Ultimate What To Bring Guide // I understand the rationales provided for making sure that you always have all the camera lenses you might need when on vacation, with a focus on covering off a fast prime, as well as having a short-, and long-range zoom. But I actually think that most travel is better done with a pair of lenses, maximum. My preference is a 35mm or 50mm equivalent, and a long-range (e.g. 80mm-300mm) zoom if I’m going to be travelling into the wilderness. I personally find that by having a fixed focal distance I’m inspired to be more creative and mindful of what I’m shooting, and spend more time just shooting as compared to thinking about what lens I need and when I need it.
Yes, people can awaken and change. And so in the future it’s always possible that people holding these values might turn into someone I’d feel comfortable working with. But in the here and now I don’t think it would be appropriate to work with, or support, persons who hold (or at least don’t oppose) such views. ↩
Theoretically, more women on ballots means more women in office, as research shows voters have no bias against them as candidates. But Thomas’s own research shows that parties are not setting up female candidates for a fair fight. In 2011, she found that all three national parties (the Green party did not run a full slate) placed more women than men in “unwinnable” ridings. Women were more likely to run in another party’s stronghold (59 per cent vs. 47 per cent for men), less likely to run in competitive battlegrounds (24 per cent vs. 28 per cent) and less likely to run in their own party’s strongholds (17 per cent vs. 25 per cent) than men.
A good long-form piece on gender equality in politics. It digs a lot deeper than most of the often-feelgood pieces that are written about the federal government’s gender policies, providing both historical information on gender equality, analysis of contemporary practices, and how other jurisdictions work far more diligently to foster equality in public offices.
Casey Johnston, over at Ars Technica, has a two-pager complaining about how tech companies design and market so-called “Ladyphones.” It’s a quick read that picks up on earlier critiques about how certain colours, and reduced technical capabilities, are associated with derogatory gender perceptions.
That said, there are at least two elements of her piece that fall short to my mind: her analysis of the BlackBerry Pearl and of the LG Windows Phone.
Johnston argues that the BlackBerry Pearl was a device marketed for women, and emphasizes the device’s high costs and pink colouration in the UK as an example of trying to extract more money from a female demographic than would be extracted from a male demographic. She also cites the Pearl’s bizarre keyboard format and limited technical specifications to further reinforce her thesis that manufactures sell second-rate products to the female market.
As someone who owned an original Pearl 8100 I don’t know how fair her critique of RIM’s product is. Pearls were RIM’s attempt to get into the consumer market generally, with the position that a full-sized keyboard was intimidating and offsetting to male and female consumers alike. Moreover, the sizes of RIM’s other smartphones at the time – designed pre-iPhone, let’s not forget! – were offsetting to most regular, non-business, consumers.
The Pearl tried to find a balance between size, consumer market expectations, and traditional BlackBerry functionality. It was also comparatively cheaper than most other smartphones at the time (and, I would note, cheaper than the popular Motorola RAZER phones), though RIM and its carrier partners haven’t necessarily reduced the costs of the phone appropriately in all regional markets. Original colours lacked pink entirely: you could buy them in black or red. New colouring – and targeting – towards particular market segments is arguably more the result of an expanded smartphone market than anything else.
I would note than Johnston is far more generous towards RIM’s marketing and branding departments than, well, any other journalist that I’ve previously read. Her assumption that RIM was so forward thinking as to brand a consumer device ‘Pearl’ to target women is massively overestimating RIM’s (traditionally very, very, very, very poor) marketing and branding departments. Finally, the technical specs of RIM’s devices are criticized from all corners, regardless of the colour or class of device (i.e. Pearl, Curve, Torch, Bold, etc), and regardless of whether the device is targeted at professional, prosumer, or consumer markets.
The other issue with the article is her analysis of the LG Windows Phone. What she’s dead right on: LG ‘partnered’ with Jill Sander to inflate the device’s cost and try to make it appeal to a certain market segment. Yep, that’s attempting to sell a device to consumers interested in or intrigued by Sander’s line of products. Where Johnston is wrong, however, is in her effort to equate low-speced Windows Phones with high cost phones.
Unlike Android and iPhone, Microsoft’s mobile phones almost universally have poor technical specifications compared to the competition. That said, Microsoft has tweaked their devices such that the specifications really don’t matter: you get excellent performance in spite of the device using older tech. As such, I don’t really think that the technical critique rings terribly true – women aren’t expected to purchase crappy Windows phones any differently then men are – though I certainly agree around the ‘branding’ of the LG device to unnecessarily inflate costs and attract a dominantly female market.
Anyways: go read the piece and develop your own opinion. Despite my two bones to pick with her evidence I think that the thesis holds and is well supported. She’s created a piece that’s short and critical, if not as deep or as powerful a critique as I’d have liked. Hopefully we see more tech sites – and mainstream news sources! – similarly take companies to task for their attempts to sell second-rate, unnecessarily gendered, products to women.
Excited Pixels 