The Null Device

Data wonks at the social music-streaming site last.fm have been taking advantage of their vast repository of recorded music to correlate analyses of the music (made using cold, hard signal-processing algorithms, not anything more subjective or fuzzy) with data from sales charts, determining how the characteristics of popular music have changed in response to cultural trends. The results make for fascinating reading.

Among findings: by looking at how percussive tracks in the charts were (i.e., how strong and regular a rhythm they had, according to spectral analyses) they pretty much pinpoint the rise of disco in the mid-1970s, a change towards more strongly rhythmic tracks which has never been reversed:

Our first thought was that songwriters in the 80s must have turned on their drum machines, loved what they heard and wrote a song to that beat - without changing the default tempo setting of 120 bpm. I would love this to be correct, but I have a hunch that it's not, especially after having found this highly interesting manual for writing a hit single written by The KLF in 1988. They say that "the different styles in modern club records are usually clustered around certain BPM’s: 120 is the classic BPM for House music and its various variants, although it is beginning to creep up", and also, "no song with a BPM over 135 will ever have a chance of getting to Number One" because "the vast majority of regular club goers will not be able to dance to it and still look cool".

last.fm's DSP algorithms also pick out the rise of punk, with its simplistic rock'n'roll arrangements and emphasis on DIY enthusiasm over polished virtuosity, and the vanquishment of prog rock, glam and other more experimental genres; this manifested itself in a steep rise in the proportion of the charts occupied by records of low harmonic and timbre complexity (i.e., both simple melodic/chord structures and unostentatious selections of instruments) between 1976 and 1979, and map the Loudness Wars of the past few decades, as the rise of the CD and a competition for sounding louder and more kick-ass than all the music that came before conspired to annihilate dynamic range:

The percentage of loud tracks has increased from 10% in 1964 (by definition) to over 40% in recent years. So music has got louder. Well, isn't that in the spirit of Rock'n'Roll? Sadly, it isn't, because the increase in loudness has led to worse sound quality. Granted, it's louder, but boy is it flat!

¶ culture dance music data mining last.fm music punk rock'n'roll 0 Share

Following in the footsteps of OKCupid's data-mining blog, some people at Facebook have recently analysed a sample of status updates by word category, extracting correlations between word categories (as well as overall subject matter and positivity/negativity), time of day and probability of updates being liked/commented on. The analysis has shown, among other things:

• there are correlations between word categories and age; older people use more first-person plurals, positive emotions and references to religion and family, while young people tend to talk in the singular first-person (presumably adolescent alienation?), mention sadness and death, swear a lot and talk about sex, music and TV.
• People with more friends talk more about social processes and other people, and have higher total word counts; whereas, while talking about home, family and emotions are correlated with having fewer friends, the most strongly correlated categories are time and the past.
• Positive emotions are one of the most likely categories to be liked, but least likely to attract comments. Negative emotions, however, attract a lot of comments (presumably from the people posting empathetic "Don't Like" messages).
• The one thing less likeable than negative emotions is talk about sleeping.
• People who talk about metaphysical or religious subjects are most likely to be friends. And people who use prepositions a lot tend not to be friends with people who swear a lot or exhibit anger or negative emotions.

¶ data mining facebook language psychology 3 Share

The latest instalment of OKCupid's data-mining blog looks at the thorny question of race again; this time, analysing the text of users' profiles, correlated by self-identified racial group.

One part of the article mines keywords unique to racial groups from profiles and presents them as tag clouds, resulting in unsubtle stereotypes. It appears that white people here are not White People; white males are straight-up bros/bogans, into Tom Clancy, sweaty guitar rock, and petrol consumption as recreation, and the females are into spectator sports and a mess of wild-nature clichés, such as thunderstorms, horses and bonfires. (An Irish-American cast looms over both genders, with "Ireland" and plastic-Paddy brocore band Dropkick Murphys rating a mention.) Meanwhile, black people are religiously demonstrative (they're more than twice as likely to mention religion as white or Asian profiles), and Asian and Indian users mention interests in hard-headed professions such as mathematics, engineering and computers, and literature such as Freakonomics, Malcolm Gladwell and Calvin & Hobbes. That and the usual stereotypes.

Among the take-aways from this post: if you want to know if white dudes will like something, put "fucking" in the middle and see if it sounds badass. Hence "Van fucking Halen" and "The Big fucking Lebowski", but not "Alicia fucking Keys". (Of course, it breaks down if irony comes into it; if you're dealing not with bros but with hipsters mining the battlefront of the pop-cultural goldmine, they can get away with a lot of stuff. Take, for example, Fleetwood fucking Mac, or Hall and fucking Oates. This does has its limits, though; chances are, there isn't a hipster with big enough post-ironic cojones to make "Celine fucking Dion" sound right.)

Further down, OKCupid also ran a reading-level analysis algorithm over users' profiles, and correlated it with race and religion. The results were fairly close, though self-identified Indians and Asians had the best-written profiles, with "Latino", "black" and "white" profiles being in the bottom half. More interestingly, the analysis by religion shows a distinct inverse correlation between religiosity and writing level.

Note that for each of the faith-based belief systems I've listed, the people who are the least serious about them write at the highest level. On the other hand, the people who are most serious about not having faith (i.e. the "very serious" agnostics and atheists) score higher than any religious groups.

¶ culture data mining okcupid race religion stereotypes 0 Share

The latest instalment of the OKCupid team's data-mining project looks at the correlations between the attractiveness of profile pictures attractiveness and the EXIF metadata contained in them. Among other things, it has found that:

• While a better camera may not make you a better photographer, it will make you look sexier (assuming you're being photographed with it, that is). Photos taken with Micro Four-Thirds cameras looked the best, followed by those taken with DSLRs, then compacts and finally camera phones, with Windows and Motorola phones taking the most minging profile pictures.
• Apple products do get you laid. or at least iPhone users have richer love lives than users of other smartphones.
• If you wish to take a flattering photo, open the aperture, getting a nice short depth of field (having a good camera, or at least not a cameraphone, helps here) and, for God's sake, don't use flash. (Unless perhaps you're going for that abject hipster porn aesthetic you see in American Apparel ads and Terry Richardson features in Vice.)
• The golden hour is not a myth; attractiveness of photos does spike immediately after sunrise and before sunset.

¶ aesthetics beauty data mining dating okcupid photography sex 0 Share

Some good news from London: Transport For London, who run the city's public transport networks, have announced that they will be opening access to all their data by the end of June. The data will include station locations, bus routes and timetable information, and will be free from restrictions for commercial or noncommercial use.

The data will be hosted at the London DataStore, a site set up to give the public access to data from public-sector organisations serving London. A few sets are already up, as well as a beta API which returns the locations of Tube trains heading for a specific station. Which could probably be worked into a mobile app to tell you when to start walking to the station. If they had something like this giving the positions and estimated arrival times of buses (whose travel times are considerably more chaotic than those of trains, and which often run less frequently, especially at night), that would be even more useful. (Some approximation of this facility exists in the LED displays, which are installed at some bus stops and sometimes are operational; a XML feed and a mobile web app would probably be a more cost-effective way of getting this information into the hands of commuters.)

Another thing that would be useful would be an API for the Transport for London Journey Planner; being able to ping a URL, passing an some postcodes or station names, a departure/arrival time and some other constraints, and get back, at your option, a maximum journey time or a list of suitable journeys, in XML or JSON format, would be useful in a lot of applications, from device- or application-specific front ends (i.e., a "take me home from here" mobile app) to ways of calculating the "inconvenience distance" between two points by counting travel time and changes (i.e.,in terms of travel convenience, Stratford is closer to Notting Hill than Stoke Newington, despite being further in geographical terms, as it's a straight trip on the Central Line).

¶ apis data data mining geodata london public transport tech uk web web services 0 Share

The social network site Facebook is supported by advertising. Being a social network site, it has the advantage of being able to serve (anonymously) targeted ads to its users, who volunteer demographic information about themselves in using the site; advertisers can target ads to users whose profiles or recent activities match certain criteria. Unfortunately, when handled clumsily, the effect can be disconcerting or creepy:

One campaign that flooded the site in recent weeks, before Facebook cracked down on it, tries to take advantage of consumer interest in Apple’s iPad. “Are you a fan of Eddie Izzard? We need 100 music and movie lovers to test and KEEP the new Apple iPad,” one version of the ad says. Louis Allred Jr., 29, a Facebook user in Los Angeles who saw the ad, said he figured it was shown to him because he or a friend had expressed enthusiasm for Mr. Izzard, a British comedian, on their profiles.

Women who change their status to “engaged” on Facebook to share the news with their friends, for example, report seeing a flood of advertisements for services and products like wedding photographers, skin treatments and weight-loss regimens.

Jess Walker, 22, from central Florida, was recently presented an ad for Plan B, the morning-after pill. “What do I have on my Facebook page that would lead them to believe I would need that?” she asked, adding that she did not want her sexual behavior called into question.

¶ advertising data mining facebook fail mori's uncanny valley privacy scams 2 Share

Pete Warden, a programmer and amateur researcher, has analysed the data from public Facebook profiles, including the relative locations of pairs of friends and people's names and fan pages, and used this to divide the US into seven relatively self-contained clusters, which he terms "Stayathomia" (i.e., the northeast to midwest), "Dixie" (the old South), "Greater Texas" (encompassing Oklahoma and Arkansas), "Mormonia" (no prizes for guessing where that is), the "Nomadic West" (places like Idaho, Oregon and Arizona, where people's connections span wide areas), Socalistan (i.e., most of California and parts of Nevada) and Pacifica (essentially Seattle). The clusters were derived from performing cluster analysis on the social graph, and not imposed on the data a priori.

Warden posts various findings he gained from crunching the data on these clusters:

Probably the least surprising of the groupings, the Old South is known for its strong and shared culture, and the pattern of ties I see backs that up. Like Stayathomia, Dixie towns tend to have links mostly to other nearby cities rather than spanning the country. Atlanta is definitely the hub of the network, showing up in the top 5 list of almost every town in the region. Southern Florida is an exception to the cluster, with a lot of connections to the East Coast, presumably sun-seeking refugees. God is almost always in the top spot on the fan pages, and for some reason Ashley shows up as a popular name here, but almost nowhere else in the country.

(Greater Texas:)God shows up, but always comes in below the Dallas Cowboys for Texas proper, and other local sports teams outside the state. I've noticed a few interesting name hotspots, like Alexandria, LA boasting Ahmed and Mohamed as #2 and #3 on their top 10 names, and Laredo with Juan, Jose, Carlos and Luis as its four most popular.

(Mormonia:) It won't be any surprise to see that LDS-related pages like Thomas S. Monson, Gordon B. Hinckley and The Book of Mormon are at the top of the charts. I didn't expect to see Twilight showing up quite so much though, I have no idea what to make of that!

(Socalistan:) Keeping up with the stereotypes, God hardly makes an appearance on the fan pages, but sports aren't that popular either. Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot.

(via MeFi) ¶ culture data mining facebook psychogeography social networks society usa 1 Share

A chap named Virgil Griffith has correlated the most popular books at every college in the US (as fetched from Facebook) with that school's average test score to find a correlation between intelligence and favourite books. According to Griffith's study, the book most correlated with high scores is Vladimir Nabokov's Lolita, and that with low scores is the Holy Bible (not to be confused with the Bible, which is around the middle, just below Harry Potter); other books correlated with high scores are Ayn Rand's Atlas Shrugged, Kurt Vonnegut's Cat's Cradle, and Freakonomics, and books dumber than "I Dont Read" include various erotica and hip-hop/ghetto fiction, The Purpose Driven Life and Fahrenheit 451. Slightly smarter than not reading are the likes of Fight Club, Dan Brown and John Grisham, along with Shakespeare's Hamlet; sci-fi, fantasy and geek/fan-interest books like The Lord of the Rings, Dune and, umm, Eragon rate more highly. (Mind you, this is correlated to test scores, not cultural well-roundedness.)

It would be interesting to see one of these correlating a measure of intelligence (such as test scores) with other factors, such as favourite music (I imagine things that a lot of geeks listen to, like metal, industrial and prog rock would come out on top, and rap-metal/nu-metal and R&B would come out fairly low), movies, or even which Facebook groups/applications one has installed.

(via Boing Boing) ¶ culture data mining facebook intelligence society stupidity 3 Share

UnSuggester is a book recommendation engine in reverse; enter a book you liked, and it'll give you a list of books you probably won't like. Apparently, fans of William Gibson's Neuromancer and Michael Moore's Stupid White Men would least want to read books on theology, the opposites of Marx & Engels' Communist Manifesto look like erotica novels, Bulgakov's The Master and Margarita is the least like an array of romance novels, Star Wars novelisations and theological texts, and the opposites of Design Patterns are mostly chick-lit, whereas The Little Prince finds itself to be the antithesis of thrillers and scifi novels. Meanwhile, people who read Illuminatus! are unlikely to read Freakonomics, and the opposite of The Da Vinci Code, with its simplistic structure and grand revelations, appears to be, naturally enough, French postmodernist philosophy.

(via /.) ¶ ai books culture data mining tech web 2 Share

A researcher at the veritable MIT Media Lab is mining volunteers' mobile phone location and call data, and using it to determine all sorts of things, from simple things such as how long people work and how much they procrastinate to which people are friends and which ones are merely coworkers. Not only that, but the data can predict people's behaviour:

Given enough data, Eagle's algorithms were able to predict what people -- especially professors and Media Lab employees -- would do next and be right up to 85 percent of the time.

Eagle used Bluetooth-enabled Nokia 6600 smartphones running custom programs that logged cell-tower information to record the phones' locations. Every five minutes, the phones also scanned the immediate vicinity for other participating phones. Using data gleaned from cell-phone towers and calling information, the system is able to predict, for example, whether someone will go out for the evening based on the volume of calls they made to friends.

Eagle was also able to see that the Red Sox's improbable breaking of the World Series curse shook even the world of MIT engineers. "I actually saw deviation patterns when the Red Sox won," Eagle said. "Everyone went deviant."

(via schneier) ¶ bluetooth data mining mit mit media lab mobile phones surveillance 0 Share

A data-mining technology developed for hunting down criminals, and used to identify backpacker killer Ivan Milat, is now being adapted to identifying consumer preferences by analysing their purchases and media choices:

"We know the people who drink a certain type of coffee will also eat specific types of chocolate bars and eat at particular food chains," said Torque's managing partner, Oliver Rees. "It's not only interesting for marketing those products to specific people but also for how store layouts are designed and how brand alliances should or could develop."

¶ data mining marketing serial killers surveillance tech 4 Share