Britain in the 50s was a country stratified by class and region. But in most of its cities, suburbs, towns and villages there was a good chance of predicting the attitudes, even the behaviour, of the people living in your immediate neighbourhood.
In many parts of Britain today that is no longer true. The country has long since ceased to be Orwell's "family" (albeit with the wrong members in charge).
To some people this is a cause of regret and disorientation - a change that they associate with the growing incivility of modern urban life. To others it is a sign of the inevitable, and welcome, march of modernity. After three centuries of homogenisation through industrialisation, urbanisation, nation-building and war, the British have become freer and more varied. Fifty years of peace, wealth and mobility have allowed a greater diversity in lifestyles and values. To this "value diversity" has been added ethnic diversity through two big waves of immigration: the mainly Commonwealth immigration from the West Indies and Asia in the 1950s and 1960s, followed by asylum-driven migrants from Europe, Africa and the greater Middle East in the late 1990s.
The diversity, individualism and mobility that characterise developed economies - especially in the era of globalisation - mean that more of our lives is spent among strangers. Ever since the invention of agriculture 10,000 years ago, humans have been used to dealing with people from beyond their own extended kin groups. The difference now in a developed country such as Britain is that we not only live among stranger citizens but we must share with them. We share public services and parts of our income in the welfare state, we share public spaces in towns and cities where we are squashed together on buses, trains and tubes, and we share in a democratic conversation - filtered by the media - about the collective choices we wish to make. All such acts of sharing are more smoothly and generously negotiated if we can take for granted a limited set of common values and assumptions. But as Britain becomes more diverse that common culture is being eroded.
And therein lies one of the central dilemmas of political life in developed societies: sharing and solidarity can conflict with diversity. This is an especially acute dilemma for progressives who want plenty of both solidarity (high social cohesion and generous welfare paid out of a progressive tax system) and diversity (equal respect for a wide range of peoples, values and ways of life). The tension between the two values is a reminder that serious politics is about trade-offs. It also suggests that the left's recent love affair with diversity may come at the expense of the values and even the people that it once championed.
It was the Conservative politician David Willetts who drew my attention to the "progressive dilemma". Speaking at a roundtable on welfare reform, he said: "The basis on which you can extract large sums of money in tax and pay it out in benefits is that most people think the recipients are people like themselves, facing difficulties that they themselves could face. If values become more diverse, if lifestyles become more differentiated, then it becomes more difficult to sustain the legitimacy of a universal risk-pooling welfare state. People ask: 'Why should I pay for them when they are doing things that I wouldn't do?' This is America versus Sweden. You can have a Swedish welfare state provided that you are a homogeneous society with intensely shared values. In the United States you have a very diverse, individualistic society where people feel fewer obligations to fellow citizens. Progressives want diversity, but they thereby undermine part of the moral consensus on which a large welfare state rests."
These words alerted me to how the progressive dilemma lurks beneath many aspects of current politics: national tax and redistribution policies; the asylum and immigration debate; development aid budgets; European Union integration and spending on the poorer southern and east European states; and even the tensions between America (built on political ideals and mass immigration) and Europe (based on nation states with core ethnic-linguistic solidarities).
Thinking about the conflict between solidarity and diversity is another way of asking a question as old as human society itself: who is my brother, with whom do I share mutual obligations? The traditional conservative, Burkean view is that our affinities ripple out from our families and localities to the nation, and not very far beyond. That view is pitted against a liberal universalist one that sees us in some sense equally obligated to all human beings, from Bolton to Burundi - an idea that is associated with the universalist aspects of Christianity and Islam, with Kantian universalism and with left-wing internationalism. Science is neutral in this dispute, or rather it stands on both sides of the argument. Evolutionary psychology stresses both the universality of most human traits and - through the notion of kin selection and reciprocal altruism - the instinct to favour our own. Social psychologists also argue that the tendency to perceive in-groups and out-groups, however ephemeral, is innate. In any case, Burkeans claim to have common sense on their side. They argue that we feel more comfortable with, and are readier to share with and sacrifice for, those with whom we have shared histories and similar values. To put it bluntly - most of us prefer our own kind.
The category "own kind", or in-group, will set alarm bells ringing in the minds of many readers. So it is worth stressing what preferring our own kind does not mean, even for a Burkean. It does not mean that we are necessarily hostile to other kinds or that we cannot empathise with outsiders. (There are those who do dislike other kinds, but in Britain they seem to be quite a small minority.) In complex societies, most of us belong simultaneously to many in-groups - family, profession, class, hobby, locality, nation - and an ability to move with ease between groups is a sign of maturity. An in-group is not, except in the case of families, a natural or biological category and the people who are deemed to belong to it can change quickly, as we saw so disastrously in Bosnia. Certainly, those we include in our in-group could be a pretty diverse crowd, especially in a city such as London.
Moreover, modern liberal societies cannot be based on a simple assertion of group identity - the very idea of the rule of law, of equal legal treatment for everyone regardless of religion, wealth, gender or ethnicity, conflicts with it. On the other hand, if you deny the assumption that humans are social, group-based primates with constraints, however imprecise, on their willingness to share, you find yourself having to defend some implausible positions: for example, that we should spend as much on development aid as on the NHS, or that Britain should have no immigration controls at all. The implicit "calculus of affinity" in media reporting of disasters is easily mocked - two dead Britons will get the same space as 200 Spaniards or 2,000 Somalis. Yet every day we make similar calculations in the distribution of our own resources. Even a well-off, liberal-minded Briton who already donates to charities will spend, say, £200 on a child's birthday party, knowing that such money could, in the right hands, save the life of a child in the third world. The extent of our obligation to those to whom we are not connected through either kinship or citizenship is in part a purely private, charitable decision. But it also has policy implications, and not just in the field of development aid. For example, significant NHS resources are spent each year on foreign visitors, especially in London. Many of us might agree in theory that the needs of desperate outsiders are often greater than our own. But we would object if our own parent or child received inferior treatment because of resources consumed by non-citizens.
Is it possible to reconcile these observations about human preferences with our increasingly open, fluid and value-diverse societies? At one level, yes. Our liberal democracies still work fairly well; indeed it is one of the achievements of modernity that people have learned to tolerate and share with people very unlike themselves. (Until the 20th century, today's welfare state would have been considered contrary to human nature.) On the other hand, the logic of solidarity, with its tendency to draw boundaries, and the logic of diversity, with its tendency to cross them, do at times pull apart. Thanks to the erosion of collective norms and identities, in particular of class and nation, and the recent surge of immigration into Europe, this may be such a time.
The modern idea of citizenship goes some way to accommodating the tension between solidarity and diversity. Citizenship is not an ethnic, blood-and-soil concept, but a more abstract political idea - implying equal legal, political and social rights (and duties) for people inhabiting a given national space. But citizenship is not just an abstract idea about rights and duties; for most of us it is something we do not choose but are born into - it arises out of a shared history, shared experiences and, often, shared suffering; as the American writer Alan Wolfe puts it: "Behind every citizen lies a graveyard."
Both aspects of citizenship imply a notion of mutual obligation. Critics have argued that this idea of national community is anachronistic - swept away by globalisation, individualism and migration - but it still has political resonance. When politicians talk about the "British people" they refer not just to a set of individuals with specific rights and duties, but to a group of people with a special commitment to one another. Membership of such a community implies acceptance of moral rules, however fuzzy, that underpin the laws and welfare systems of the state.
In the rhetoric of the modern liberal state, the glue of ethnicity ("people who look and talk like us") has been replaced with the glue of values ("people who think and behave like us"). But British values grow, in part, out of a specific history and even geography. Too rapid a change in the make-up of a community not only changes the present, it also, potentially, changes our link with the past. As Bob Rowthorn wrotein Prospect in February 2003, we may lose a sense of responsibility for our own history - the good things as well as the shameful things in it - if too many citizens no longer identify with it.
Is this a problem? Surely Britain in 2004 has become too diverse and complex to give expression to a common culture in the present, let alone the past. Diversity in this context is usually code for ethnic difference. But that is only one part of the diversity story, albeit the easiest to quantify and most emotionally charged. The progressive dilemma is also revealed in the value and generational rifts that emerged with such force in the 1960s. At the Prospect roundtable mentioned above, Patricia Hewitt, now secretary of state for trade and industry, recalled an example of generational conflict from her Leicester constituency. She was canvassing on a council estate when an elderly white couple saw her Labour rosette and one of them said: "We're not voting Labour - you hand taxpayers' money to our daughter." She apparently lived on a nearby estate, with three children all by different fathers, and her parents had cut her off (evidence that even close genetic ties do not always produce solidarity).
Greater diversity can produce real conflicts of values and interests, but it also generates unjustified fears. Exposure to a wider spread of lifestyles, plus more mobility and better education, has helped to combat some of those fears - a trend reinforced by popular culture and the expansion of higher education (graduates are notably more tolerant than non-graduates). There is less overt homophobia, sexism or racism (and much more racial intermarriage) in Britain than 30 years ago and racial discrimination is the most politically sensitive form of unfairness. But 31% of people still admit to being racially prejudiced. Researchers such as Isaac Marks at London's Institute of Psychiatry warn that it is not possible to neatly divide the population between a small group of xenophobes and the rest. Feelings of suspicion and hostility towards outsiders are latent in most of us.
The visibility of ethnic difference means that it often overshadows other forms of diversity. Changes in the ethnic composition of a city or neighbourhood can come to stand for the wider changes of modern life. Some expressions of racism, especially by old people, can be read as declarations of dismay at the passing of old ways of life (though this makes it no less unpleasant to be on the receiving end). The different appearance of many immigrants is an outward reminder that they are, at least initially, strangers. If welfare states demand that we pay into a common fund on which we can all draw at times of need, it is important that we feel that most people have made the same effort to be self-supporting and will not take advantage. We need to be reassured that strangers, especially those from other countries, have the same idea of reciprocity as we do. Absorbing outsiders into a community that is worthy of the name takes time.
Negotiating the tension between solidarity and diversity is at the heart of politics. But both left and right have, for different reasons, downplayed the issue. The left is reluctant to acknowledge a conflict between values it cherishes; it is ready to stress the erosion of community from "bad" forms of diversity, such as market individualism, but not from "good" forms of diversity, such as sexual freedom and immigration. And the right, in Britain at least, has sidestepped the conflict, partly because it is less interested in solidarity than the left, but also because it is still trying to prove that it is comfortable with diversity.
But is there any hard evidence that the progressive dilemma actually exists in the real world of political and social choices? In most EU states the percentage of GDP taken in tax is still at historically high levels, despite the increase in diversity of all kinds. Yet it is also true that Scandinavian countries with the biggest welfare states have been the most socially and ethnically homogeneous states in the west. By the same token, the welfare state has always been weaker in the individualistic, ethnically divided US compared with more homogeneous Europe. And the three bursts of welfarist legislation that the US did see - Franklin Roosevelt's New Deal, Harry Truman's Fair Deal and Lyndon Johnson's Great Society - came during the long pause in mass immigration between the first world war and 1968. (They were also, clearly, a response to the depression and to two world wars.)
In their 2001 Harvard Institute of Economic Research paper "Why Doesn't the US Have a European-style Welfare State?", Alberto Alesina, Edward Glaeser and Bruce Sacerdote argue that the answer is that too many people at the bottom of the pile in the US are black or Hispanic. Across the US as a whole, 70% of the population are non-Hispanic whites - but of those in poverty only 46% are non-Hispanic whites. So a disproportionate amount of tax income spent on welfare is going to minorities. The paper also finds that US states that are more ethnically fragmented than average spend less on social services. The authors conclude that Americans think of the poor as members of a different group, whereas Europeans still think of the poor as members of the same group. Robert Putnam, the analyst of social capital, has also found a link between high ethnic mix and low trust in the US. There is some British evidence supporting this link, too. Researchers at Mori found that the average level of satisfaction with local authorities declines steeply as the extent of ethnic fragmentation increases. Even allowing for the fact that areas of high ethnic mix tend to be poorer, Mori found that ethnic fractionalisation still had a substantial negative impact on attitudes to local government.
Finally, Sweden and Denmark may provide a social laboratory for the solidarity/diversity trade-off in the coming years. Starting from similar positions as homogeneous countries with high levels of redistribution, they have taken rather different approaches to immigration over the past few years. Although both countries place great stress on integrating outsiders, Sweden has adopted a moderately multicultural outlook. It has also adapted its economy somewhat, reducing job protection for older native males in order to create more low-wage jobs for immigrants in the public sector. About 12% of Swedes are now foreign-born, and it is expected that by 2015 about 25% of under-18s will be either foreign-born or the children of the foreign-born. This is a radical change and Sweden is adapting to it rather well. (The first clips of mourning Swedes after the murder of the foreign minister Anna Lindh were of crying immigrants expressing their sorrow in perfect Swedish.) But not all Swedes are happy about it.
Denmark has a more restrictive and "nativist" approach to immigration. Only 6% of the population is foreign-born, and native Danes enjoy superior welfare benefits to incomers. If the solidarity/diversity trade-off is a real one and current trends continue, then one would expect in, say, 20 years that Sweden will have a less redistributive welfare state than Denmark; or rather that Denmark will have a more developed two-tier welfare state with higher benefits for insiders, while Sweden will have a universal but less generous system.
What are the main objections, at least from the left, to this argument about solidarity and diversity? Multiculturalists stress Britain's multiple diversities, of class and region, that preceded recent waves of immigration. They also argue that all humans share similar needs and a common interest in ensuring that they are met with minimum conflict; this, they say, can now be done through human rights laws. And hostility to diversity, they conclude, is usually a form of "false consciousness".
Critics of the dilemma also say, rightly, that the moral norms underpinning a community need not be hard for outsiders to comply with: broad common standards of right and wrong, some agreement on the nature of marriage and the family, respect for law and some consensus about the role of religion in public life. Moreover, they add, there are places such as Canada (even Australia) that are happily combining European-style welfare with an officially multicultural politics. London, too, has American levels of ethnic diversity but is the most leftwing part of Britain.
In the autumn 2003 issue of the US magazine Dissent, two academics, Keith Banting and Will Kymlicka, show that there is no link between the adoption of multiculturalist policies in countries such as Canada, Sweden and Britain and the erosion of the welfare state.
But many of the policies they describe are either too technical (allowing dual citizenship) or too anodyne (the existence of a government body to consult minorities) to stimulate serious tax resistance. They also assume too swift a reaction to growing diversity - these are forces that take effect over decades, if not over generations.
Similarly, two British academics, Bhikhu Parekh and Ali Rattansi, have offered a critique of the solidarity v diversity thesis (partly in response to Prospect articles) that also assumes an implausibly rapid connection between social cause and effect. They argue that because the expansion of Britain's welfare state in the late 40s coincided with the first big wave of non-white immigration into Britain, ethnic diversity cannot be a drag on social solidarity. But the post-1945 welfare state was the result of at least 100 years of experience and agitation. The arrival of a small number of immigrants in the 40s and 50s was unlikely to have much bearing on that history. Parekh, Kymlicka and others also argue that labour movement strength, not ethnic homogeneity, is the best indicator of the size of a welfare state. But labour movements themselves are stronger where there are no significant religious or ethnic divisions. In any case, we are not concerned here with the formation of welfare states so much as with their continued flourishing today.
A further point made by the multiculturalists is more telling. They argue that a single national story is not a sound base for a common culture because it has always been contested by class, region and religion. In Britain, the left traces democracy back to the peasants' revolt, the right back to Magna Carta, and so on. But while that is true, it is also the case that these different stories refer to a shared history. This does not imply a single narrative or national identity any more than a husband and wife will describe their married life together in the same way. Nor does it mean that the stress on the binding force of a shared history (or historical institutions such as parliament) condemns immigrants to a second-class citizenship. Newcomers can and should adopt the history of their new country as well as, over time, contributing to it - moving from immigrant "them" to citizen "us". Helpfully, Britain's story includes, through empire, the story of many of our immigrant groups - empire soldiers, for example, fought in many of the wars that created modern Britain.
I would add a further qualification to the progressive dilemma. Attitudes to welfare have, for many people, become more instrumental: I pay so much in, the state gives me this in return. As we grow richer, the ties that used to bind workers together in a risk-pooling welfare state (first locally, later nationally) have loosened - "generosity" is more abstract and compulsory, a matter of enlightened self-interest rather than mutual obligation. Moreover, welfare is less redistributive than most people imagine - most of the tax paid out by citizens comes back to them in one form or another so the amount of the average person's income going to someone they might consider undeserving is small. This, however, does little to allay anxieties based on perceptions rather than fiscal truths. And poor whites, who have relatively little, are more likely to resent even small transfers compared with those on higher incomes.
Despite these qualifications, it still seems to me that those who value solidarity should take care that it is not eroded by a refusal to acknowledge the constraints upon it. The politician who has recently laid most stress on those constraints, especially in relation to immigration, is the home secretary, David Blunkett. He has spoken about the need for more integration of some immigrant communities - especially Muslim ones - while continuing to welcome high levels of net immigration into Britain of over 150,000 a year.
· Read the second part of David Goodhart's essay here.
· This article appeared in the February issue of Prospect
· Join the debate by emailing unlimited@guardianunlimited.co.uk