The Vons grocery store two miles from my home in Los Angeles, California sells 12 cans of Coca-Cola for $6.59 — 54 cents each. The tool chain that created this simple product is incomprehensibly complex.

Each can originated in a small town of 4,000 people on the Murray River in Western Australia called Pinjarra. Pinjarra is the site of the world’s largest bauxite mine. Bauxite is surface mined — basically scraped and dug from the top of the ground. The bauxite is crushed and washed with hot sodium hydroxide, which separates it into aluminum hydroxide and waste material called red mud. The aluminum hydroxide is cooled, then heated to over a thousand degrees celsius in a kiln, where it becomes aluminum oxide, or alumina. The alumina is dissolved in a molten substance called cryolite, a rare mineral first discovered in Greenland, and turned into pure aluminum using electricity in a process called electrolysis. The pure aluminum sinks to the bottom of the molten cryolite, is drained off and placed in a mold. It cools into the shape of a long cylindrical bar. The bar is transported west again, to the Port of Bunbury, and loaded onto a container ship bound for — in the case of Coke for sale in Los Angeles — Long Beach.

The bar is transported to Downey, California, where it is rolled flat in a rolling mill, and turned into aluminum sheets. The sheets are punched into circles and shaped into a cup by a mechanical process called drawing and ironing — this not only makes the can but also thins the aluminum. The transition from flat circle to something that resembles a can takes about a fifth of a second. The outside of the can is decorated using a base layer of urethane acrylate, then up to seven layers of colored acrylic paint and varnish that is cured using ultra violet light, and the inside of the can is painted too — with a complex chemical called a comestible polymeric coating that prevents any of the aluminum getting into the soda. So far, this vast tool chain has only produced an empty, open can with no lid. The next step is to fill it.

Coca-Cola is made from a syrup produced by the Coca-Cola Company of Atlanta. The main ingredient in the formula used in the United States is a sweetener called high-fructose corn syrup 55, so named because it is 55 per cent fructose or “fruit sugar” and 42 per cent glucose or “simple sugar” — the same ratio of fructose to glucose as natural honey. HFCS is made by grinding wet corn until it becomes cornstarch. The cornstarch is mixed with an enzyme secreted by a bacillus, a rod-shaped bacterium, and another enzyme secreted by a mold called Aspergillus. This process creates the glucose. A third enzyme, also derived from bacteria, is then used to turn some of the glucose into fructose.

The second ingredient, caramel coloring, gives the drink its distinctive dark brown color. There are four types of caramel coloring — Coca Cola uses type E150d, which is made by heating sugars with sulfite and ammonia to create bitter brown liquid. The syrup’s other principal ingredient is phosphoric acid, which adds acidity and is made by diluting burnt phosphorus (created by heating phosphate rock in an arc-furnace) and processing it to remove arsenic.

A much smaller proportion of the syrup is flavors. These include vanilla, which is the fruit of a Mexican orchid that has been dried and cured for around three months; cinnamon, the inner bark of a Sri Lankan tree; coca-leaf which comes from South America and is processed in a unique US government authorized factory in New Jersey to remove its addictive stimulant cocaine; and kola nut, a red nut found on a tree which grows in the African Rain Forest (this may be the origin of Coca-Cola’s distinctive red logo).

The final ingredient is caffeine, a stimulating alkaloid that can be derived from the kola nut, coffee beans and other sources.

All these ingredients are combined and boiled down to a concentrate, then transported from the Coca-Cola Company factory in Atlanta to Downey where the concentrate is diluted with water infused with carbon dioxide. Some of the carbon dioxide turns to gas in the water, and these gas bubbles give it effervescence, also known as “fizz,” after its sound. 12 ounces of this mixture is poured into the can.

The top of the can is then added. This is carefully engineered: it is made from aluminum, but it has to be thicker and stronger to withstand the pressure of the carbon dioxide gas, and so it uses an alloy with more magnesium than the rest of the can. The lid is punched and scored so that a tab opening, also made of aluminum, can be installed. The finished lid is put on top of the filled can, and the edges of the can are folded over it and welded shut. 12 of these cans are then packaged into a painted paperboard box called a fridge pack, using a machine capable of producing 300 such packs a minute.

The finished product is transported by road to a distribution center and then to my local Vons. This tool chain, which spans bauxite bulldozers, refrigerators, urethane, bacteria and cocaine, produces 70 million cans of Coca-Cola each day, one of which can be purchased for about two quarters on most street corners, and each of which contains far more than something to drink. Like every other tool, a can of Coke is a product of our world entire and contains inventions that trace all the way back to the origins of our species.

The number of individuals who know how to make a can of Coke is zero. The number of individual nations that could produce a can of Coke is zero. This famously American product is not American at all. Invention and creation is something we are all in together. Modern tool chains are so long and complex that they bind us into one people and one planet. They are not only chains of tools, they are also chains of minds: local and foreign, ancient and modern, living and dead — the result of disparate invention and intelligence distributed over time and space. Coca-Cola did not teach the world to sing, no matter what its commercials suggest, yet every can of Coke contains humanity’s choir.


“What Coke Contains” is an extract from my book, “How to Fly a Horse — The Secret History of Creation, Invention, and Discovery,” available here. If you would like to get news of more posts like this one, like my Facebook page,here.

After “What Coke Contains” was published, many people mentioned similar — and better — work that I did not know about and is well worth reading:

• ”I, Pencil,” a classic essay by Leonard E. Read (thank you, Chuck Grimmett of the Foundation For Economic EducationBryce Miller and Michael Joseph.) Tom Snyder maintains a great web site about this essay.

• “When Ideas Have Sex,” a TED talk by Matt Ridley

• ”Life Of The Automobile,” by Ilya Ehrenburg (thank you, John Glover ofBloomberg News.)

• A 1967 Christmas Sermon by Martin Luther King, which says, “before you finish eating breakfast in the morning, you’ve depended on more than half of the world.” (Thank you, Ben Glickman of ESL101.)

• The excellent, “The Toaster Project,” by Thomas Thwaites, which I read long after I wrote this piece, and had not considered in this context until Nick Douglas, Editor of Slacktory, pointed out the similarity. Thank you, Nick.

And, yes, there really is a Murray River in Western Australia.

Three prominent moon-landing conspiracy theories are debunked using the latest in dynamic-lighting technology.



In 2002, Buzz Aldrin, the second man to walk on the moon, punchedBart Sibrel in the face. Why? According to a Gallup poll from 1999, some 6% of Americans still believed then that the government faked the Apollo 11 moon landing in 1969—and Sibrel is one of the more vocal among them. If you search YouTube for moon-landing conspiracy videos today, it’s apparent there are still people like him who believe Aldrin and Neil Armstrong never left planet Earth.

For all those Coast To Coast AM listeners who think the government staged the moon landing, graphics card maker NVIDIA wants to set the record straight. On Thursday, the Santa Clara, California, company launched two new graphics cards, the GeForce GTX 980 and GTX 970. To show off their ability to render real-time dynamic lighting (what it calls Voxel Global Illumination), NVIDIA used this technology to re-create a model of that historic landing and debunk three prominent conspiracy theories around it.

"We talked to a lot of experts in the field to re-create what happened on the moon that day," GeForce general manager Scott Herkelman told Fast Company. “We re-created perfectly what they made and how the reflection would look off the suits, duct tape, aluminum foil.”

In the above video from NASA, Aldrin descends from Apollo 11 around the 20-second mark. Conspiracy theorists often point out three problems with the footage:

  1. Given the position of the sun behind Apollo 11, why can we see details that otherwise would be obscured by shadow?
  2. Why can’t we see any stars in the background?
  3. What’s the strange bright light seen between the ladder and the vehicle?

Debunking Theory No. 1

Fine silt on the moon’s surface reflects light onto objects that would otherwise be obscured by shadow.Image: NVIDIA

The conspiracy theorists who question why the footage isn’t obscured by darkness don’t take into account how light from the sun interacts with the moon’s surface. The fine lunar dust covering the moon has mirror-like properties, reflecting the sun and illuminating objects on the surface. “The sun is hitting the dust, and it’s illuminating the backside of the Apollo module and astronaut,” says Herkelman.

Debunking Theory No. 2

We can see the Earth, but not the stars because of the camera’s aperture.Image: NVIDIA

According to experts, some 84,000 stars would’ve been visible from the moon that evening, so why can’t we see any when looking at historic videos and photos? “This one is easier than the other two,” says Herkelman. “When they went to take the picture, the camera they used had a closed aperture.” Doing so allowed viewers back home to see the astronauts and spacecraft clearly. In asimulation, Herkelman opened up the aperture, which gave a clear view of the stars but produced a blown-out image of the moon and the objects on its surface.

Debunking Theory No. 3

Is that brightness a studio light or Buzz Aldrin?Image: NVIDIA

Is that seemingly out-of-place light source a studio light that some amateur forgot to turn off? “We couldn’t quite figure out what’s going on, but then we remembered we needed to place Neil Armstrong. The second we put Neil Armstrong there, we figured out the light source,” says Herkelman. Furthermore, the astronauts wore ultra-reflective space suits to keep them cool inside. When Herkelman changed the perspective of the model, he is able to confirm Armstrong’s position.

The view from another angle.Image: NVIDIA

This new information is unlikely to change the minds of die-hard moon landing deniers, but you’ve got to hand it to NVIDIA for coming up with a pretty neat way to show off the power of its graphics cards.

[Rendering courtesy of NVIDIA]

Great interview with the CEO of UrbanSitter (think Uber for babysitters, so a problem they need to tackle head on)

Most new tech companies simply would not work without consumer trust. People wouldn’t get into an Uber, list their home on Airbnb, or even buy shoes on Zappos if they didn’t trust those companies to deliver a high quality, secure service. UrbanSitter sets the bar even higher: It connects families with babysitters on the Internet. There are few things that require more faith.

“In many ways, we’re tackling the service that requires the most trust in someone’s life,” says UrbanSitter CEO Lynn Perkins. “If companies can replicate what we’ve done in other sectors, they’ll knock it out of the park.”

So how did UrbanSitter pull this off? How did they build a product that convinces parents that strangers can safely watch their children? Perkins has become an expert in this area, pointing to a combination of product features, logistics, and customer service efforts that have allowed them to become a reliable solution for hundreds of thousands of households nationwide. In this exclusive interview, she shares how UrbanSitter has approached trust-building and how other companies can do the same to grow fast.


Identify What Your Users Need

People have short attention spans. Your website has limited real estate. You need to know what you have to instantaneously display to users so they feel like you understand their needs. How can you demonstrate to them that you’re an expert and worthy of them making a transaction?

“The best way to do this is to first identify your ideal user,” says Perkins. “Our ideal user is a parent who has a somewhat urgent need for a sitter. It can be that broad, but you need to know that profile first so you can determine what information to show them.”

She goes on to say, “People might have a variety of needs, but you should try to figure out what those top three scenarios are and answer those first, as fast as you can. That’s what will make people feel like you’re the most relevant product for their search.”

There are several reasons customers go to UrbanSitter’s site: They have to find a sitter right away and time is of the essence; they want a lower-cost childcare solution; they just moved to a new area and they don’t know anyone to watch their child. To answer all of these questions, UrbanSitter immediately shows you available sitters, their average response times, their price ranges and — usingFacebook Connect (crucially) — stack ranks the sitters based on your degree of connection, so your friends’ sitters are shown first.

“As you develop trust, you develop brand equity. That should be your goal.”Tweet

Borrow Trust Wherever You Can

Facebook Connect has made a huge impact on UrbanSitter’s success. And now with other networks like LinkedIn and Google rolling out similar services that port user information, there’s more ways for startups to leverage social graphs than ever before.

“The way we use people’s social graphs is one of our biggest differentiators,” Perkins says, adding that they plan to roll out the ability to show LinkedIn connections soon. “When it comes to trusting a babysitter, friend and family recommendations carry the most weight. If they have used a sitter and been happy with them, you’re very likely to trust that person. On that second tier might be co-workers, or parents who have children at the same school. Social proof at this scale is hugely influential.”

When a parent joins UrbanSitter, the site also asks a few questions, including which parenting groups they belong to, like a local mom’s group, a specific PTA, even nearby museums that might have significance to their family.

“When people see that other parents from a group they belong to or respect have used a sitter, that’s compelling information,” says Perkins. “And when we launch in a new city, these groups are really important because they’ve already established trust. We’re borrowing brand credibility from them so new users will trust us too.”

When a user searches for a sitter on the site, they are presented with Facebook connections that have also hired the sitter, in addition to the names of groups the sitter may be affiliated with, and even certifications they have. “We’ll add to the search listing that a sitter is CPR trained at a class that was held at DayOne, which is a parenting organization that people in the Bay Area would likely know. In Los Angeles, the Pump Station is similarly credible, and so on.”

“Trust travels in channels, and some channels are more meaningful than others.” Tweet

“This doesn’t mean we should clutter sitter profiles with a ton of information. Gaining trust is just as much about maintaining focus,” says Perkins. “Whatever business you’re in, start with the few data points that will signify expertise to the group you’re trying to reach. Then when people click on a profile, you can grow the amount of information you show them from there.”

In UrbanSitter’s case, they started with people’s Facebook connections, and grew to include group affiliations. Then they expanded their efforts to collect testimonials and reviews from influential parenting bloggers who are already trusted by loyal readerships. “When a blogger with a following writes about his or her experience with your product, you absorb the trust of those followers.”

Don’t Make Ratings Your Only Metric

While reviews and average ratings are important to the UrbanSitter product — allowing the company to elevate great sitters and flag poor-performing sitters — they aren’t the only data driving user decisions. Other metrics can be used to better tailor your service and make it clear that you’re anticipating user needs. For example, UrbanSitter also emphasizes sitter response times (something the platform records, requiring no extra work from users).

“If I’m looking for someone last minute, it’s really good for me to know upfront that someone with a great star rating might take longer than 24 hours to respond,” says Perkins. “On the flipside, if someone else also has a great rating and responds in under an hour, that’s perfect.”

The company strives to provide a mix of quantitative and subjective data to help users determine whether or not to trust a sitter with the job, but they know a lot of decisions are based on numbers. Two of the most important stats are how many ratings a sitter has — giving a better sense of how accurate that score is — and “Repeat Families,” which shows how many families that sitter has worked with more than once.

“If someone has even 5 or more repeat families, that indicates they are a pretty good sitter,” says Perkins. “People are very sensitive about the experiences their children have, so to invite someone back means everything went very smoothly. That way someone might only have a handful of reviews, but if you see that five of them came from repeat families, you’d be pretty confident in their abilities.”

This is also helpful on the sitter side of the platform. If someone has chosen to work for the same family multiple times, it indicates that they were likely paid the agreed upon amount, that the environment felt safe, and that the experience was good overall. Those are the key concerns sitters generally have before accepting a job, she says, and they are easily and quickly answered by providing that information.

Most marketplace companies today rely solely on star ratings, but adding in more data specific to your service builds expertise, Perkins says. In this same vein, Uber estimates how far away cabs are from your current location in addition to showing you the driver’s average star rating. Consignment furniture service Move Loot has a standard system to show quality of goods. This is a big part of understanding what your customers need to know.

Let People Be Human

UrbanSitter had been available for a while before the site introduced video profiles for sitters — and it was a game changer.

“All of a sudden, video provided a three-dimensional feeling to the site,” says Perkins. “Before video, profiles were flat. You could see the data and read the reviews, but you didn’t really feel like you knew the person.”

“Hearing someone talk for just two minutes about themselves is really powerful.” Tweet

She cites one sitter, a young woman named Peri, who used her video profile to sing a song she wrote about why she would make an excellent babysitter. “You understand right away that she’s this warm, creative person who is likely to be good with kids.”

Launching videos had two other positive byproducts. First, people felt even more comfortable booking a sitter last minute when they weren’t able to interview them in real time beforehand. Second, less experienced sitters saw a jump in business.

“Our less experienced and less reviewed sitters weren’t getting a lot of bookings because of the competition,” says Perkins. “But as soon as we added the video option to profiles they started getting far more jobs.” She points to one sitter in particular who saw a spike in bookings after adding a video talking about training to be a pediatrician. Even a short video gives you a good sense of how capable a sitter will be on the job.

In many cases, the company says its sitters speak for themselves. One reason UrbanSitter has gotten so much positive coverage from bloggers is because they hold events in new cities where parents, including mom bloggers, can meet sitters in-person. “For us, putting our best foot forward means showcasing the sitters,” says Perkins. As compelling and comprehensive as an online product can be, it can’t beat the person-to-person connection, so enhancing that wherever possible is always a good strategy.

“Don’t be afraid to try unscalable things that will enhance your brand and build trust, especially in a new market,” she says.

Lynn Perkins has more than 15 years of experience building and growing consumer internet services. Previously, she served as founder and CEO of Xuny, and VP of Business Development at Bridgepath.

Marry Quality and Influence

Demonstrating remarkable quality has been vital to UrbanSitter’s expansion plan from the beginning. Today, its protocol for launching in new markets is practically turnkey, but it always starts with curating an incredible cohort of sitters on the supply side.

“We knew if we were going to succeed, we needed people to have not just good but great experiences using the service,” says Perkins. To assemble its initial batch of caretakers, UrbanSitter conducted rigorous interviews and made sure only the best of the best were on the platform to start. It’s second step: Introducing these sitters to incredibly connected parents in the area.

“We wanted people with influence to know what it was like to use UrbanSitter,” she says. “What we found was that it actually took a fairly small group of sitters and a small group of highly-connected parents to get the ball rolling.”

Going hyper-local is the most effective way to start. For UrbanSitter, this meant launching first in Brooklyn over Manhattan, then letting word of mouth grow and naturally expand over the entire metro area.

“Wherever your users are concentrated, start with those that are highly-connected.” Tweet

Whenever you’ve just entered a market, your mission should be to dazzle people first and promote engagement second, Perkins says. “First you’ll get the ‘Wow’ factor, and if you can repeat that over and over again, you’ll create a brand that sticks with people.” If you focus on this type of curated, highly-monitored experience at the beginning, people will trust that you deliver quality, and that’s the word that will spread.

UrbanSitter uses this same concept of combining quality and influence in its online marketing. They built Facebook’s Like button into sitter profile pages so that customers could post about individual sitters and the great experiences they had using the platform. Prospective users are more likely to respond to individual sitter success stories than general ad copy for the service. And when any of this is connected to people users already know on Facebook, the influence is that much stronger.

“This is a great idea if your brand is less personal than the individuals who provide the service,” says Perkins. “People connect much more directly with other people. You see the sitter’s face and when you click, you’re actually taken directly to their profile. You see they are fantastic, and your assumption is that you’ll find other great sitters on the site too.”

Once you have users’ faith that they will be served well, it’s paramount that you preserve that. You want to equip your service providers with all the tools they need to do a good job. UrbanSitter goes so far as to text its sitters minutes before they’re scheduled to start a job with information like “the five things that will make you the best sitter they ever had” and more.

The company also started building its dedicated user operations team early. This gives them the chance to be proactive about how they intercept and address user issues. If, for example, a sitter declines a job on short notice, the company reaches out immediately with other possible options. They communicate with users over email and even the phone depending on the scale of the user’s need in order to make people feel cared for.

It’s also important to be proactive when it comes to addressing problems that have already popped up. When things don’t go well, you need to take action immediately. Perkins says she once booked a Zipcar, and a half an hour before she picked it up they called to tell her it had been in an accident. They asked her where she was and identified cars that were even closer to her. Those cars might have been more expensive, but they offered them to her for the same price. Having a good plan for damage control in place is a critical part of earning trust.

Give Users Control

“When you need to quickly gain consumer trust, I think the best way is to give them control over their experience with your service and on your site,” says Perkins. “If they know what it is they’re looking for, make it easy for them to find it.”

In UrbanSitter’s case, this means presenting users with detailed options, not just pointing them to one sitter and saying, “This one’s for you,” even if that seems like it would make things easier. Instead, they provide different ways for users to filter their search. These even get granular if users want, allowing parents to select a caregiver who speaks French or another language.

Consistent user behavior and data tracking can give you a good sense of where and how to empower your users, Perkins explains. “Being able to understand how people are using your product lets you hone in on what people really want.”

When UrbanSitter first launched, Perkins and her team assumed that if a regular sitter wasn’t available, parents would go back and look at other sitters their friends had hired. But, it turned out that parents were more interested in booking the friends of the sitter they liked. The company realized that users were organically searching for other sitters this way, so they modified the product to surface sitters’ friends who were also working on the site.

“It was a random sign of trust for us that people liked their sitters so much that they preferred to find someone similar to them, even more so than someone else their friends would recommend,” says Perkins. “I think that actually approximates the experience people have in real life when they find a sitter the old-fashioned way. They probably ask that person if they know anyone from school.” Given this, it stands to reason that building mechanisms into your site that approximate real life can help foster trust too.

Perkins points to Goodreads as another site that takes advantage of this online-offline phenomenon. The site uses your social graph to help you recommend books you’ve liked and discover books from friends, just like you would in real life in real time. It generates the same type of buzz and community, which has led an increasing number of people to trust Goodreads for surfacing great books.

Pinpoint Decision Drivers

“Once you’ve defined your ideal users and developed basic functionality, it’s time to get even more precise about how people decide to use your service,” Perkins says. “You want to list all of the concerns they could possibly have and then the data you’ll use to respond to all of those concerns. That’s how you engender trust on an even deeper level.”

For instance, people might believe your service is good, but still might be wary if they haven’t heard of anyone using it, or if they don’t have any friends who have used it before. In this case, you can use data to your advantage. “It builds a lot of credibility when we can say X number of parents from your child’s school or your mom’s group have used UrbanSitter.” They can start to visualize users just like themselves being satisfied with the service.

Figuring out what these data points are shouldn’t be a guessing game either. Take strategic opportunities to ask people what they care about and what changed their minds.

“Just by looking at data that tracks how people use our site, we know how many of our bookings are people who needed a sitter within 24 hours. We know how many people were just browsing and for how long. But we also ask when we can. Following a full transaction, we ask if they would recommend UrbanSitter to a friend. If they are willing to answer that, we ask them why or why not. If they leave the site without booking, we ask them first why they were there and what they were looking for.”

This type of surveying and monitoring helped Perkins and her team discover that more people were simply browsing than they initially thought. Now, the next version of the site will make browsing and booking even easier.

Make Life Easy, Not Just Your Service

A big part of trust is believing that a company has your best interest at heart even though you’re a paying customer. Taking this into account, UrbanSitter has made it a primary goal to make its service a smoother, easier, stress-free version of finding childcare, and to take as much doubt out of the process as possible.

As an example, when parents pay an UrbanSitter, their credit card is simply charged by the company and the appropriate funds are transferred to the sitter. No one has to pull out a wallet or determine how much to pay or to tip. This eliminates a traditionally awkward experience for both parties involved.

“We find that a lot of sitters are uncomfortable asking for a higher rate, and a lot of parents simply don’t know whether to tip — only 20 percent do on a regular basis,” Perkins says. “We’re able to tell them what average rates are and take all the uncertainty out of this part of the evening.”

Cutting down on friction is another way you can prove you’re an expert in your field. You’ve thought of everything even before your customer does, and you basically tell them they don’t have to worry about it because you’ve got their back.

Other successful companies have relied on the same idea. Zappos famously allows customers to order many pairs of shoes and return the ones they don’t want for free, making it simple and easy to buy a pair of shoes without ever having to go to the store and eliminating the hesitation around buying online.

“Any time a customer’s hesitation is relieved by something you do, you win.”Tweet

Child clothing and toy company Zulily is another champion in this category. “Let’s say that you bought a couple of items that are clearly for an infant girl — suddenly all of the recommendations you get from them focus on purchasing for the child you’re buying for. They seem to magically know. In this case, you feel closer to the company because they seem to know who you are and show you what you’re looking for.”

The more creative you can get in your friction reduction, the better. Suddenly you don’t just seem user-friendly, you seem thoughtful, Perkins says. This is one of the reasons UrbanSitter pursued co-promotions with companies likeOpentable. Knowing that their users include tech savvy parents looking to plan a night out, the company made it that much easier to have a great evening by offering deals on babysitting and a great meal.

Invest in Content Marketing

Content marketing can go a long way toward solidifying expertise in a particular field, whether you’re in the enterprise or consumer market. UrbanSitter publishes articles that tackle all aspects of the childcare experience. They even interview parents and sitters about what makes sitting better and easier.

“We might write something like the top 10 questions to ask a babysitter in an interview,” says Perkins. “We end up attracting parents to our website through these articles and simultaneously build our brand awareness, while converting them into customers.”

UrbanSitter doesn’t just tap into parenting bloggers, it also advertises and provides content through newsletters and partnerships. The company makes connections with schools and seeks to partner with organizations that already have a lot of credibility so that they can absorb some of that trust.

“This may sound completely unscalable, but it really helps build initial brand awareness,” says Perkins. “Once we hit word-of-mouth scale we can start in with paid search and Facebook ads, SEO and things like that.”

GoPro is a great example of content marketing done right. The company takes advantage of user-generated content to promote its brand of cameras on YouTube. In fact, a recent video of a user swimming through a school of jellyfish received over 2 million views. As a result, GoPro is now one of the top contributing brands on YouTube, and has gotten that many more impressions.

As important as content marketing is, it’s even more critical that you don’t go overboard. “You can very quickly damage the value of your service and your network when you become spammy,” Perkins says. “Because we leverage Facebook so much, and we have all this information about who you’re connected to, we’re very careful to not take advantage of that, and I’d advise other companies to be very careful too.”

“Privacy concerns are a dealbreaker. Balance your need for user information with your desire to use it to benefit your company.” Tweet

Turn Users Into Ambassadors

“The most important thing you can do is turn a user into a referrer, because that means they’ve had the ultimate great experience,” says Perkins. “A huge percentage of people become Uber users because their friends tried it first and they were passengers in the car with them. It’s massively valuable.”

A recent study by Nielsen shows that word of mouth recommendations from family and friends spur the highest levels of action. Capturing this effect online should be one of your highest goals.

To accomplish this, the best thing you can do is build social media capabilities into your product to make word of mouth sharing simple. “You want to make it as easy as possible for people to share their experiences with the brand,” says Perkins. “It makes your brand bigger, and the bigger it gets the easier it becomes to get organic referrals.”

To capture referrals, UrbanSitter instituted a “give-to-get” program. That way, when families recommend other families to use the service, they get free babysitting credits. It’s been a popular away for the company to acquire new customers.

“You want to make it seamless,” says Perkins. “Every time you can convince someone to share because they had such a great experience with your product, the lower the cost of acquiring a new customer becomes. That’s trust. They trust you so much that they’re willing to share their experience with others, and that’s ultimately what will grow your business.”

Sep 19th 2014, 5:54 by Bagehot | GLASGOW

THE Union flag will still fly. By a margin of 55% to 45%, and on a vast 85% turnout, Scots voted to stick with the United Kingdom on September 18th. Thereby they ensured the continuation of the nation state that shaped the modern world, one which still retains great capacity for good. They also preserved the British identity which over a third of Scots, English, Welsh and Northern Irish consider of primary importance. Had around 200,000 more Scots answered “Yes” to the question “Should Scotland be an independent country”, these precious attributes would have been damaged, or destroyed, and Britain with them.

A rush of support for the Yes Scotland campaign in the fortnight before the vote had made that outcome eminently possible. A poll for YouGov had put the separatists in front; and though the latest polls pointed to a win for the No side, they suggested it would be by little more than the margin of error. As it turned out, the final result, though it would have seemed amazingly close only a month ago, was never in much doubt.

Beginning with tiny Clackmannanshire, a deprived fief of the separatist Scottish National Party (SNP) in central Scotland, which declared for the union at 1.30am, the No vote held up surprisingly strongly in most of Scotland’s 32 councils. The Gaelic-speaking, SNP-voting Western Isles delivered another early snub to the separatists. Dundee—dubbed by the SNP’s leader, and Scotland’s first minister, Alex Salmond, as the “Yes City”—gave him a rare victory, but on a relatively low turnout, of 79%, and by a narrower-than-expected margin. In Angus and Mr Salmond’s own Aberdeenshire, the Yes campaign suffered defeats in the SNP’s heartland. When, at around 4.30am, mighty Glasgow delivered only a modest win for the Yeses, with 53% of the vote, the verdict was clear.

As news of the early results filtered through to the venerable George Square in Glasgow, a gathering-point for the Yes camp’s most vehement supporters (“Our Tahrir Square”, some call it), a crowd wearing kilts and draped with the sky-blue Saltire hushed their chanting, but only a bit. They exuded the loyal defiance of a football crowd in adversity; it helped that the pubs were open all night, an arrangement that had caused local trepidation. “We know what’s happening in Glasgow tonight,” muttered a woman working at a supermarket checkout on the edge of the square, “A wild party or a riot”. It was a reasonable fear: a minority of Yes campaigners have behaved thuggishly at times during a protracted, two-year referendum campaign. Yet there were few incidents of violence. Perhaps few Scots, whether leaning to Yes or No, had any energy left for a fight.

The campaign had been gruelling, especially on the Yes side. Though designed and steered by the SNP, the Yes Scotland banner was carried by many different groups—including Radical Independence, Women for Independence and the Scottish Greens—many of them locally based, and all hugely motivated. By any measure, they outgunned the cross-party Better Together campaign, knocking on more doors, delivering more leaflets, placing more advertisements in newspapers and on billboards. In Dundee, Glasgow and even genteel Edinburgh, blue “Yes” stickers are everywhere; stepping in off a Glasgow street, your correspondent discovered two stuck beneath his shoe.

By contrast, purple “No, thanks” badges, advertising Better Together’s prim slogan, are hard to find. Yet on the day of voting, thousands of unexpected unionist volunteers were reported to have turned out, across Scotland, to help get out their vote. 

This points to the likeliest of three possible explanations for the late hardening of the unionist vote: a determined rallying of unionists, startled by the previously unimagined possibility of a Yes triumph and costly bifurcation. They received additional encouragement from the second possible reason, a belated and tempestuous entry into the campaign by Gordon Brown, the former Labour prime minister.

Having previously played little role in Better Together, Mr Brown has emerged over the last fortnight as the charismatic, positive and forceful voice of unionism it had previously lacked. Whether lacerating the Yes side’s wishful, or mendacious, predictions for an independent Scotland’s economic prospects; or glorifying the benefits of scale and co-operation that lie in the current arrangement, often using Biblical rhetoric, Mr Brown gave a glimpse of a brilliance that was seldom evident during his time in 10 Downing Street. His final turn of the campaign, delivered to a packed-out Glaswegian audience, was the speech of his life.

The nationalists, he fulminated, without notes or teleprompter, were promising “an economic minefield where problems could implode at any time, an economic trapdoor down which we go, from which we might never escape.” Have none of it! he cried. Embrace, instead, “the Scotland of Adam Smith and John Smith, the Scotland of civility and compassion, the Scotland of comradeship and community.” 

In addition, Mr Brown relayed a panicked response to the late Yes surge from Westminster, a promise of further devolution to the Scottish Parliament, which was the third possible explanation for the strengthened No vote. This also led to his successor, David Cameron, the Conservative prime minister, and his rivals, Ed Miliband, leader of the Labour Party, and Nick Clegg of the Liberal Democrats, scurrying north, in an emergency mission to promise these powers and protest their love for the Scots. Opinion polls suggest this made little difference to the vote; indeed, Mr Cameron’s hasty dash may even have done damage. Outside a polling booth in Inveraray, an 18th-century new town in Argyll, and seat of the ultra-unionist clan Campbell, one voter suggested she had been dissuaded from voting No because of it. “It made me think there was something funny going on,” she said. “It made me think Cameron was after something.” So thinks the Scottish street of distant, high-handed Westminster, a disdain that Mr Salmond has richly capitalised on.

Mr Cameron has sworn to begin cross-party negotiations on the promised new powers on September 19th, even as hangovers throb through the Yes and No camps. Already, all three party leaders have pledged to increase Scotland’s powers to raise income and other taxes, and it is hard to see how they could renege on this. That would be the death of their parties in Scotland. It would also turn the current clamour for independence into a deafening roar. Yet the outcome of the cross-party talks are unlikely to be so swiftly or easily deliverable as they made out, in their pledge to Scottish voters—not least because of the demands for new English powers, in Westminster and the regions, that it has elicited back home.

So the negotiations will be fraught; and new constitutional arrangements may not emerge, as Mr Cameron and the rest have promised, ahead of the next general election, due in nine months. But emerge they must, because Britain depends on it. A million Scots have just voted to quit the union, even in the knowledge that this would probably make them poorer. Only a strong turnout by Scottish pensioners—the only age-group thought likely to have voted mainly for the union—foiled them. This, on a night of huge relief for most Britons, is truly shocking. It means the British nation state has survived; yet it remains on life support.

THE United Kingdom was formed in 1707, when the parliaments of England and Scotland agreed to establish a single country. Today, 307 years later, the union remained, after a Scottish referendum on September 18th to separate from Britain failed. The “yes” campaign for independence won 45% of the vote, compared with 55% of Scots who voted “no”. The 85% turnout is among the highest in Scottish history.

Our story on the referendum result is here. An analysis of what it means for British politics is here. Our profile of George Osborne, the chancellor of the exchequer who in effect helped preserve the union is here. And as financial markets yawned, our Buttonwood columnist mused

SupergirlCREDIT: DC Comics 

The information on Supergirl, DC Entertainment’s latest superhero offering being developed for TV, has changed at an incredible pace. The latest? Well, Greg Berlanti (ArrowThe Flash at The CW) is still developing the show, but now it’s for CBS. The deal, as Deadline reports, hit Thursday night, and is for a series commitment, similar to how Fox initially ordered Gotham. Ali Adler is also producing/developing the show.

Supergirl will feature Kara Zor-El, Superman’s cousin, and is planned to have the character in full super form. She’s had her powers for years and is now, in the series, at age 24, embracing them and her responsibility to be a hero. Here’s the official description:

Born on the planet Krypton, Kara Zor-el escaped amid its destruction years ago. Since arriving on Earth, she’s been hiding the powers she shares with her famous cousin. But now at age 24, she decides to embrace her superhuman abilities and be the hero she was always meant to be.

Supergirl
CREDIT: DC Comics

Berlanti and Adler are writing the pilot script, and Geoff Johns is “closely involved.” As CCO of DCE, he has been involved in all of their recent TV success, from Arrowforward

Man, I haven’t gotten to this advanced level of standing desk yet… (HT: Sophia)

This Hamster-Wheel Desk Is a Sad Statement on Modern OfficesEXPAND

Corporate culture today is often said to be about working slavishly, with an above-all focus on profit and productivity. Nothing captures that idea more strikingly than this (fully functional)hamster-wheel desk.

The project, designed and built in 24 hours by Pier 9’s artist-in-residence Robb Godshaw, is actually a pretty clever piece of design: the wheel rests on four skateboard wheels built into the base, which lets the wheel rotate as the user walks round and round, churning through those spreadsheets and getting a sweetleg workout at the same time.

As Godshaw says (hopefully sarcastically):

Rise up, sedentary sentients, and unleash that untapped potential within by marching endlessly towards a brilliant future of focused work. Step forward into a world of infinite potential, bounded only by the smooth arcs of a wheel.

If you feel like making your own, the plans are up on Instructables. [Instructables]

A DNA sequencer the size of a cell phone could change where, and how, gene research occurs.

The DNA sequencer built by Oxford Nanopore draws power from a computer’s USB port.

One day in 1989, biophysicist David Deamer pulled his car off California’s Interstate 5 to hurriedly scribble down an idea. In a mental flash, he had pictured a strand of DNA threading its way through a microscopic pore. Grabbing a pen and a yellow pad, he sketched out a radical new way to study the molecule of life.

Twenty-five years later, the idea is now being commercialized as a gene sequencing machine that’s no larger than a smartphone, and whose effects might eventually be similarly transformative.

Early versions of the instrument, called the MinION, have been reaching scientific labs over the past few months after long delays (see “10 Breakthrough Technologies 2012: Nanopore Sequencing”). It’s built by a U.K. company, Oxford Nanopore, that has raised $292 million and spent 10 years developing Deamer’s idea into a DNA sequencer unlike any other now available. It is four inches long and gets its power from a USB port on a computer. Unlike other commercial sequencing machines, which can be the size of a refrigerator and require jugs of pricey chemicals, this one measures DNA directly as the molecule is drawn through a tiny pore suspended in a membrane. Changes in electrical current are used to read off the chain of genetic letters, A, G, C, and T.

Scientists with early access to prototypes of the first commercial “nanopore” sequencer say it’s glitchy and error-prone but may still be the way scientists study DNA in the future. After testing it, Mick Watson, a bioinformatics researcher at the Roslin Institute, in Scotland, says nanopore sequencing is a “disruptive technology that could, potentially, dominate the sequencing market for years to come.”

Although researchers say the device is still desperately inaccurate, it can already carry out some unheard-of scientific feats. And then there’s its size. A sequencer this small might one day let police read off a genome from a spot of blood at a crime scene, or permit doctors to pinpoint viruses in the midst of an epidemic. One scientist this month tweeted a picture of the sequencer on his dining room table, decoding DNA.

The MinION is the result of some very high-stakes R&D by Oxford, a 200-person company that’s long has had its eye on the expanding market for high-speed DNA sequencers. Cracking that market won’t be easy. About 90 percent of DNA data is produced on sequencing machines from a single company, Illumina of San Diego (see “50 Smartest Companies: Illumina”). Its sequencers are so good that most of its competitors have ended up in Chapter 11 or retreated in ignominy.

David Deamer made this sketch in 1989 when the idea for nanopore sequencing came to him.

But now some big companies are betting that nanopores could be the technology to break Illumina’s lucrative monopoly. Roche, which made a failed attempt to acquire Illumina in 2012, this year spent $125 million to buy Genia Technologies, a small nanopore company based in California, and invested in another, Stratos Genomics. Hitachi is also working on nanopore technology, as are startups like Electronic Biosciences.

Deamer says the idea of nanopore sequencing occurred to him in 1989, just three years after the first automated DNA sequencers were introduced. He had been trying to build artificial cells, spherical blobs of fat that could pump molecules in and out through microscopic pores the way real cells do. His flash of insight was that a molecule passing through one of these pores—especially a long molecule like DNA—would continuously change the blob’s electrical properties. That would create a signal you could measure.

It took another 25 years before the MinION was developed. That’s because the technical problems were so daunting. Each DNA letter is only about half a nanometer from the next and some differ by just an atom or two, so they’re hard to tell apart. And how could you pull the string of DNA letters through the pore, suspended in a layer one-100,000th as thick as a hair—much less at a rate of 30 letters a second, as the MinION does?

“There were a lot of smart people saying this is physically not possible to do,” says Jeffery Schloss, head of the division of genome sciences at the National Human Genome Research Institute, in Bethesda, Maryland. “Well, it definitely is possible. The question now is if it’s good enough to be used in a practical way.”

Oxford has disappointed its fans before. When it first announced the MinION, in 2012, expectations soared off the charts (see “Why a Portable DNA Device Could Yield Better Data”). But when the promised machines failed to appear,some started to wonder if the sequencer was vaporware.

By this spring, Oxford had the bugs worked out—enough, at any rate, to start mailing out beta versions of the nanopore sequencer to 500 hand-picked labs it is collaborating with. Another early creator of the technology, Mark Akeson, who works alongside Deamer in the bioengineering department at the University of California, Santa Cruz, says since June he’s received two updated versions, a sign of how quickly Oxford is scrambling to improve the device.

To the technology’s original inventors, the arrival of any commercial nanopore sequencer is a milestone. “The idea they can FedEx a 100-gram machine that actually works is pretty amazing,” says Akeson. While both he and Deamer have extensive financial ties to Oxford, they say it’s obvious to everyone that the MinION is just the start. “It’s doing okay,” says Deamer. “The question is accuracy, but improvements are coming, believe me.”

Several scientists using the device say it correctly reads only 60 to 85 percent of DNA letters. That’s the bad news. (Illumina’s machines, some of which can cost $1 million, are more than 99.9 percent accurate.) Yet nanopore sequencing is so different that even a machine that’s error-prone might be a boon to science.

One reason is that today’s fastest sequencers decode DNA after it’s been shredded into tiny bits, reading off just 150 letters at a time. Those bits then have to be puzzled back together to create a genome. Even with a supercomputer, the puzzle often can’t be solved—there can be too many repeated sequences or parts that go missing.

Nanopore sequencing may help because it produces what scientists call “long reads.” For instance, Akeson says this summer his lab read across a continuous strand of human DNA that was 79,000 letters long. That’s probably a record. Like having the edges of a puzzle, long reads make it much easier to reassemble a genome, especially of a species never studied before.

The eventual commercial price of the sequencer isn’t known. Nick Loman, a researcher at the University of Birmingham, said his lab has paid about $1,000 each to get hold of the machines, though disposable cells containing the matrix of pores (about 2,000 of them) would cost extra. He said he hoped the price would stay “very cheap” in the future.

Last week, Oxford’s chief technology officer, Clive Brown, said further instruments would be announced soon. One, dubbed PromethION, will use as many as 100,000 pores in parallel and could compete with Illumina’s top-of-the line sequencing system, which was introduced early this year (see “Does Illumina Have the First $1,000 Genome?”) to labs interested in sequencing hundreds of thousands of human genomes for medical research. 

Drug is one of thousands that may be produced by the human microbiome.

11 September 2014

BSIP SA / Alamy

The antibiotic lactocillin was isolated from a Lactobacillus bacterium (shown here).

Bacteria living on human bodies contain genes that are likely to code for a vast number of drug-like molecules — including a new antibiotic made by bacteria that live in the vagina, researchers report in this week’s issue of Cell1.

The drug, lactocillin, hints at the untapped medical potential of this microbial landscape.

“They have shown that there is a huge diverse potential of the microbiome for producing antimicrobial molecules,” says Marc Ouellette, a microbiologist at the University of Laval’s Hospital Centre (CHUL) in Quebec, Canada, who was not involved in the research.

Studies have suggested that the composition of our microbiomes — the whole suite of bacteria living on our bodies — has huge impacts on our health, but it has been difficult to show exactly how this works.

Michael Fischbach, a microbiologist and chemist at the University of California, San Francisco, led a team that aimed to fill in those blanks. The researchers built a machine-learning algorithm, training a computer program to recognize genes that are already known to make small molecules that could act as drugs. Then they asked the program to hunt for similar genes in the human microbiome. The search yielded thousands of these drug-making genes within microbes living on and in the body. Some are similar to drugs being tested in clinical trials, such as a class of antibiotics called thiopeptides.

“We used to think that drugs were discovered by drug companies and prescribed by a physician and then they get to you,” Fischbach says. “What we’ve found here is that bacteria that live on and inside of humans are doing an end-run around that process; they make drugs right on your body.”

Fischbach’s team then purified one of these: a thiopeptide made by a bacterium that normally lives in the human vagina. The researchers found that the drug could kill the same types of bacteria as other thiopeptides — for instance, Staphylococcus aureus, which can cause skin infections. The scientists did not actually show that the human vaginal bacteria make the drug on the body, but they did show that when they grew the bacteria, it made the antibiotic.

Big data boost

Finding specific molecules like these and studying what they do will help researchers to understand how the microbiome interacts with our bodies, says microbial genomicist Derrick Fouts of the J. Craig Venter Institute in Rockville, Maryland.

“This is a great example of the power of bioinformatics to not merely identify genes of interest from ‘big data’ ‘omics, but to connect together cassettes of genes to increase our fundamental understanding of how commensal bacteria maintain a healthy human microbiome,” Fouts says.

Other researchers say that the paper also demonstrates how the microbiome might be mined for new drugs. Scientists have long argued that the suite of microbes living on human bodies could be a rich source of such drugs,and many drug companies are trying to capitalize on that idea; Fischbach advises two of them.

“To my knowledge, this is the first work that isolates new compounds with strong drug potential from the human microbiome,” says Rob Knight, a microbial ecologist at the University of Colorado, Boulder. “This work provides an exciting platform for mining our microbiomes for new compounds of medical interest.”

A similar drug is in development at Novartis, but Fischbach doesn’t plan to develop the antibiotic that he has discovered into a drug. Instead, he wants to find novel types of molecule that are made by the microbiome. Studying these molecules might help researchers to understand how the microbiome influences our susceptibility to disease, he says.

“People are eager to learn what exactly helpful bacteria are doing,” Fischbach says. “Nobody had anticipated that they have the capability to make so many different kinds of drugs. I don’t think this is the only thing they do, but it’s a big thing.”

Long but fascinating history of how the tech industry fought email spam, how they won, and how new cryptography technology and messaging apps might impact this in the future

Mike Hearn mike at plan99.net 
Fri Sep 5 08:07:30 PDT 2014


Hey,

Trevor asked me to write up some thoughts on how spam filtering and fully
end to end crypto would interact, so it's all available in one message
instead of scattered over other threads. Specifically he asked for brain
dumps on:

   - how does antispam currently work at large email providers
   - how would widespread E2E crypto affect this
   - what are the options for moving things to the client (and pros, cons)
   - is this feasible for email?
   - How do things change when moving from email to other sorts of
async messaging
   (e.g. text messaging) or new protocols - i.e. are there unique aspects
   of existing email protocols, or are these general problems?

Brief note about my background, to establish credentials:  I worked at
Google for about 7.5 years. For about 4.5 of those I worked on the Gmail
abuse team, which is very tightly linked with the spam team (they use the
same software, share the same on-call rotations etc).

Starting around mid-2010 we had put sufficient pressure on spammers that
they were unable to make money using their older techniques, and some of
them switched to performing industrial-scale hacking of accounts using
compromised passwords (and then sending spam to the account's contacts), so
I became tech lead of a new anti-hijacking team. We spent about 2.5 years
beating the hijackers. In early 2013 we declared victory
<http://googleblog.blogspot.ch/2013/02/an-update-on-our-war-against-account.html>
and
a few months later, Edward Snowden revealed that the NSA/GCHQ was tapping
the security system we had designed
<http://www.theguardian.com/technology/2013/nov/06/google-nsa-gchq-spying-judicial-process>
.

Since then things seem to be pretty quiet. It's not implausible to say that
from Gmail's perspective the spam war has been won .... for now, at least.

In case you prefer videos to reading a few years ago I gave a talk at the
RIPE64 conference in Ljubljana:

   https://ripe64.ripe.net/archives/video/25/

In January I left Google to focus on Bitcoin full time. My current project
is a p2p crowdfunding app I want to use as a way to fund development of
decentralised infrastructure.

OK, here we go.

*A brief history of the spam war*

In the beginning ... there was the regex. Gmail does support regex
filtering but only as a last resort. It's easy to make mistakes, like the
time we accidentally blackholed email for an unfortunate Italian woman
named "Oli*via Gra*dina". Plus this technique does not internationalise,
and randomising text to miss the blacklists is easy.

The email community began sharing abusive IPs. Spamhaus was born. This
approach worked better because it involved burning something that the
spammer had to pay money to obtain. But it caused huge fights because the
blacklist operators became judge, jury and executioner over people's mail
streams. What spam actually is turned out to be a contentious issue. Many
bulk mailers didn't think they were spamming, but in the absence of a clear
definition sometimes blacklisters disagreed.

Botnets appeared as a way to get around RBLs, and in response spam fighters
mapped out the internet to create a "policy block list" - ranges of IPs
that were assigned to residential connections and thus should not be
sending any email at all. Botnets generate enormous amounts of spam by
volume, but it's also the easiest spam to filter. Very little of my time on
the Gmail spam/abuse team was spent thinking about botnets.

Webmail services like Gmail came on the scene. The very first release of
Gmail simply used spamassassin on the backend, but this was quickly deemed
not good enough and a custom filter was built. The architect of the Gmail
filter wrote a paper in 2006 which you can find here:

   http://ceas.cc/2006/19.pdf

I'll summarise it. The primary technique the new filter used was attempting
to heuristically guess the sending domain for email (domains being harder
to obtain and more stable than IPs), and then calculating *reputations* over
them. A reputation is a score between 0-100 where 100 is perfectly good and
0 means always spam. For example if a sender had a reputation of 70 that
means about 30% of the time we think their mail is spam and the rest of the
time it's legit. Reputations are moving averages that are calculated based
on a careful blend of manual feedbacks from the Report Spam/Not Spam
buttons and "auto feedbacks" generated by the spam filter itself.
Obviously, manual feedbacks have a lot more weight in the system and that
allows the filter to self correct.

This approach has another advantage - it eliminates all the political
fighting. The new definition of spam is "whatever our users say spam is", a
definition that cannot be argued with and is simultaneously crisp enough to
implement, yet vague enough to adapt to whatever spammers come up with.

It's worth noting a few things here:

   - Reputation systems require the ability to read *all* email. It's not
   good enough to be able to see only spam, because otherwise the reputations
   have no way to self correct. The flow of "not spam" reports is just as
   important as the flow of spam reports. Most not spam reports are generated
   implicitly of course, by the act of not marking the message at all.

   - You need to calculate reputations *fast*. If you receive mail with
   unknown reputations, you have no choice but to let it pass as otherwise you
   can't figure out if it's spam or not. That in turn incentivises spammers to
   try and outrun the learning system. The first version of the reputation
   system used MapReduce and calculated reputations in batch, so convergence
   took hours. Eventually it had to be replaced with an online system that
   recalculates scores on the fly. This system is a tremendously impressive
   piece of engineering - it's basically a global, real time peer to peer
   learning system. There are no masters. The filter is distributed throughout
   the world and can tolerate the loss of multiple datacenters.

   I don't want to think about how you'd build one of these outside a
   highly controlled environment, it was enough of a headache even in the
   proprietary/centralised setting ....

   - Reputations propagate between each other. If we know a link is bad and
   it appears in mail from an IP with unknown reputation, then that IP gets a
   bad reputation too and vice versa. It turns out that this is important - as
   the number of things upon which reputations are calculated goes up, it
   becomes harder and harder for spammers to rotate all of them
   simultaneously. Especially this is true if using a botnet where precise
   control over the sending machines is hard. If a spammer fails to randomize
   even one tiny aspect of their mail at the same time as the others, all
   their links and IPs get automatically burned and they lose money.

   - Reputation contains an inherent problem. You need lots of users, which
   implies accounts must be free. If accounts are free then spammers can sign
   up for accounts and mark their own email as not spam, effectively doing a
   sybil attack on the system. This is not a theoretical problem.


The reputation system was generalised to calculate reputations over
*features* of messages beyond just sending domain. A message feature can
be, for example, a list of the domains found in clickable hyperlinks. Links
would turn out to be a critical battleground that would be extensively
fought over in the years ahead. The reason is obvious: spammers want to
sell something. Therefore they must get users to their shop. No matter how
they phrase their offer, the URL to the destination must work. The fight
went like this:

   1. They start with clear clickable links in HTML emails. Filters start
   blocking any email with those links.

   2. They start obfuscating the links, and requesting users put the link
   back together. But this works poorly because many users either can't or
   won't figure it out, so profits fall.

   3. They start buying and creating randomised domains in bulk. TLDs like
   .com are expensive but others are cheap or free and the reputations of the
   entire TLDs went into freefall (like .cc)

   4. Spammers run out of abusable TLDs as registrars begin to crack down.
   They begin performing *reputation hijacking*, e.g. by creating blogs on
   sites which allow you to register *.blogspot.com,  *.livejournal.com and
   so on. URL shorteners become a spammers best friend. Literally every URL
   shortener immediately becomes a war zone as the operators and spammers
   fight to defend and attack the URL domain reputations.

   5. Spammers also start hacking websites but this doesn't work that well,
   because many websites don't often appear in legitimate mail often so they
   don't have strong reputations. Great source of passwords though.

   6. Big content hosting sites like Google begin connecting their spam
   filters to their hosting engines so once the reputation of a user-generated
   URL falls it's automatically terminated. The first iterations of this are
   too slow. One of my projects at Google was to build a real-time system to
   do this automatic content takedown.


Obtaining fresh sending IP addresses was a problem for them too of course.
The best fix was to use webmail services as anonymizing proxies. Gmail was
hit especially hard by this because early on Paul Buchheit (the creator)
decided not to include the client IP address in email headers. This was
either a win for user privacy or a blatant violation of the RFCs, depending
on who you asked. It also turned Gmail into the worlds biggest anonymous
remailer - a real asset for spammers that let them sail right past most
filters which couldn't block messages from a sender as large as Google.

Between about 2006 (open signups) and 2010 a lot of the anti-spam work
involved building a spam filter for account signups. We did a pretty good
job, even though I say so myself. You can see the prices of different kinds
of "free" webmail accounts at http://buyaccs.com (a Russian account shop).
Note that hotmail/outlook.com accounts cost $10 per thousand and gmails
cost an order of magnitude more. When we started gmails were about $25 per
1000 so we were able to quadruple the price. Going higher than that is hard
because all big websites use phone verification to handle false positives
and at these price levels it becomes profitable to just buy lots of SIM
cards and burn phone numbers.

There's a significant amount of magic involved in preventing bulk signups.
As an example, I created a system that randomly generates encrypted
JavaScripts that are designed to resist reverse engineering attempts. These
programs know how to detect automated signup scripts and entirely wiped
them out
<http://webcache.googleusercontent.com/search?q=cache:v6Iza2JzJCwJ:www.hackforums.net/archive/index.php/thread-2198360.html+&cd=8&hl=en&ct=clnk&gl=ch>
.

*How would widespread E2E crypto affect all this*

You can see several themes in the above story:

   - Large volumes of data is really important, of both legit and spam
   messages.
   - Extremely high speed is important. A lot of spam fights boil down to a
   game of who is faster. If your reputations converge in 3 minutes then
   you're going to be outrun.
   - Being able to police your user base is important. You can't establish
   reputations if you can't trust your user reports and that means creating a
   theoretically impossible situation:   accounts that are free yet also cost
   money   (if you need lots of them)

The first problem we have in the E2E context is that reputation databases
require input from *all* mail. We can imagine an email client that knows
how to decrypt a message, performs feature extraction and then uploads a
"good mail" or "bad mail" report to some <handwave> central facility. But
then that central facility is going to learn not only who you are talking
with but also what links are in the mail. That's probably quite valuable
information to have. As you add features this problem gets worse.

The second problem we have is that if the central reputation aggregator
can't read your mails, it doesn't know if you did feature extraction
honestly. This is not a problem in the unencrypted context because the spam
filter extracts features itself. Whilst spammers can try to game the
system, they still have to actually send their spams to themselves for
real, and this imposes a cost. In a world where spam filters cannot read
the message, spammers can just submit entirely fictional "good mail"
reports. Worse, competitors could interfere with each others mail streams
by submitting false reports. We see this sort of thing with AdWords.

The third problem is that spam filters rely quite heavily on security
through obscurity, because it works well. Though some features are well
known (sending IP, links) there are many others, and those are secret. If
calculation was pushed to the client then spammers could see exactly what
they had to randomise and the cross-propagation of reputations wouldn't
work as well.

It might be possible to resolve the above two problems using trusted
computing. With TC you can run encrypted software on private data and the
hardware will "prove" what it ran to a remote server. But security through
obscurity and end to end crypto are hard to mix - if you run your email
content through a black box, that black box could potentially steal the
contents. You have to trust the entity calculating the secret sauce with
your message, and then you could just use Gmail in the regular way as today.

The fourth problem we have is that anonymous usage and spam filters don't
really mix. Ultimately there's no replacement for cutting spam off at the
source. Account termination is a fundamental spam fighting tool.  All major
webmail and social services force users to perform phone verification if
they trip an abuse filter. This sends a random code via SMS or voice call
to a phone number and verifies the user can receive it. It works because
phone numbers are a resource that have a cost associated with them, yet
~all users have one. But in many countries it's illegal to have anonymous
mobile numbers and operators are forced to do ID verification before
handing out a SIM card. The fact that you can be "name checked" at any
moment with plausible deniability means that whilst you don't have to
provide any personal data to get a webmail account, a government could
force you to reveal your location and/or identity at any time. They don't
even have to do anything special; if they can phish your password they can
forcibly trip the abuse filter, wait for the user to pass phone
verification, then get a warrant for the users account metadata knowing
that it now contains what they need   (I never saw any evidence of this,
but it's theoretically possible).

The final problem we have is that spam filtering is resource intensive CPU
and disk wise. Many, many users now access their email *exclusively* via a
smartphone. Smartphones do not have many resources and the more work you
do, the worse the battery life. Simply waking up the radio to download a
message uses battery. Attempting to do even obsolete 1990's style spam
filtering of all mail received with a phone would probably be a non starter
unless there's some fundamental breakthrough in battery technology.

In conclusion, I don't see a return to pure client side filtering being
feasible.

*How do things change when moving from email to other sorts of
async messaging ?*

Well. SMS spam is a thing. It doesn't happen much because phone companies
act as spam filters. Also, because governments tend to get involved with
the punishment of SMS spammers, in order to discourage copycat offenders
and send a message (pun totally intended). Email spam blew up way before
governments could react to it, so it's interesting to see the different
paths these systems have taken.

Systems like WhatsApp don't seem to suffer spam, but I presume that's just
an indication that their spam/abuse team is doing a good job. They are in
the easiest position. When you have central control everything becomes a
million times easier because you can change anything at any time. You can
terminate accounts and control signups. If you don't have central control,
you have to rely exclusively on inbound filtering and have to just suck it
up when spammers try to find ways around your defences. Plus you often lose
control over the clients.


*General thoughts and conclusions*

When you look at what it's taken to win the spam war with cleartext, it's
been a pretty incredible effort stretched over many years. "War" is a good
analogy: there were two opposing sides and many interesting battles,
skirmishes tactics and weapons. I could tell stories all day but this email
is already way too long.

Trying to refight that in the encrypted context would be like trying to
fight a regular war blindfolded and handcuffed. You'd be dead within
minutes.

So I think we need totally new approaches. The first idea people have is to
make sending email cost money, but that sucks for several reasons; most
obviously - free global communication is IMHO one of humanities greatest
achievements, right up there with putting a man on the moon. Someone from
rural China can send me a message within seconds, for free, and I can
reply, for free! Think about that for a second.

The other reason it sucks is that it confuses bulk mail with spam. This is
a very common confusion. Lots of companies send vast amounts of mail that
users want to receive. Think Facebook, for example. If every mail cost
money, some legit and useful businesses wouldn't work, let alone things
like mailing lists.

A possibly better approach is to use money to create deposits. There is a
protocol that allows bitcoins to be sacrificed to miners fees, letting you
prove that you threw money away by signing challenges with the keys that
did so. This would allow very precise establishment of an anonymous yet
costly credential that can then send as much mail as it wants, and have
reputations calculated over it. Spam/not spam reports that *only* contain
proof of sending could then be scatter/gathered and used to calculate a
reputation, or if there is none, then such mails could be throttled until a
few volunteers have peeked inside. Another approach would be to allow
cross-signing - an entity with good reputation can temporarily countersign
mail to give it a reputational boost and trigger cross-propagation of
reputations. That entity could employ whatever techniques they liked to
verify the senders legitimacy.

It's for these reasons that I'm interested in the overlap between Bitcoin
and E2E messaging. It seems to me they are fundamentally linked.

Final thought. I'm somewhat notorious in the Bitcoin community for making
radical suggestions, like maybe there exists a tradeoff between privacy and
abuse. Lots of people in the crypto community passionately hate this idea
and (unfortunately) anyone who makes it. I guess you can see based on the
above stories why I think this way though. It's not clear to me that
chasing perfect privacy whilst ignoring abuse is the right path for any
system that wishes to achieve mainstream success.

We’re stood at the foothills of a very large and formidable mountain that has the perfect smartwatch at its peak. It’s still very early, not everyone’s sure of their footing or the right course to take, but we are all instinctively drawn toward that pinnacle. The thing is, even with all the months and maybe years standing between humanity and its ideal wrist gadget, the winners of the race are already known. Google and Apple won.

Attempts at standalone smartwatches seem to resurface every few years. There was Microsoft’s SPOT, the LG GD910 Watch Phone, and the Meta Watch — to name just three in the past decade — and none of them ever amounted to anything more than an intriguing bulletpoint in the history of personal electronics. Maybe you might have passed by one on your way to buying the latest and greatest new smartphone. It’s actually the evolution of smartphones into the primary computer for many people that has now created the opportunity for smartwatches to flourish.

LG GD910

The challenges of miniaturizing displays, processors, and battery technology to the size of a watch are still present, but connecting that watch to a smartphone can offload some of its requirements and make the thing strapped to your wrist more bearable to wear and attractive to use. It also plugs the smartwatch into something critically important: an ecosystem of supportive hardware vendors and willing software developers. They are the ones who can drive a product category out of niche status and into the hands of mainstream consumers, but they demand some certainty that their investment of time and resources won’t go to waste. That assurance comes from knowing that the connected smartphone is running the right operating system.

Yesterday’s Apple Watch launch spoke glowingly of the “seamless” integration between the wearable device and the iPhone. Similarly, when it introduced its ZenWatch at IFA, Asus said that the watch works brilliantly with its Zen UI on Android smartphones. Both are just positive spins on the fact that an Apple Watch or an Android Wear device without a smartphone attached to it is not a terribly useful thing to have. Most of them don’t even display the time all the time, making them poor substitutes for watches.

SEAMLESS SYNERGY BETWEEN SMARTPHONE AND WRIST ISN’T GOOD NEWS FOR EVERYONE

The close working relationship between smartwatch and smartphone ties down user choice in another way. If you really fancy the look of the Moto 360, you’ll want to combine it with an Android handset to make the most of Google’s ecosystem. The same is true of the Apple Watch and iPhone: they’ll work great together and fall apart when they are, well, apart. Windows Phone and BlackBerry users aren’t being served by these devices at all.

Wearables are perceived as the next frontier in consumer electronics, both because of new technological advancements and because of the rich potential to combine them with the smartphones that hundreds of millions of people already own. And because the vast majority of those users are already on either Apple or Google’s mobile platforms, it only makes sense for development efforts to be focused on those two juggernauts. It’s not dissimilar to how things panned out in the tablet market. Though they took different paths to their predominance, both Apple and Google relied on the strength of their smartphone software and ecosystems to drive the development of their tablet offerings. Compare them to BlackBerry’s disastrous PlayBook tablet, which lacked such basic functionality as a native email client, or HP’s beautiful but inadequately supported WebOS operating system and TouchPad devices. Microsoft’s Windows is the only tablet alternative precisely because it’s able to lean on the influence of its own ecosystem.

The Apple Watch and Android Wear at least guarantee a wide audience for any new products, whether they be hardware of software. Sony recognized this and abandoned four generations of its own smartwatch OS to join the ranks of Android Wear purveyors — aligning itself in direct competition with LG, Samsung, Motorola, and Asus, plus probably HTC and everyone else currently making Android smartphones. If even Sony, with all its resources, cannot strike out on its own, what hope is there for a third alternative beyond Android Wear and Apple’s Watch?

NOTHING’S MORE ATTRACTIVE TO ECOSYSTEM BUILDERS THAN AN ALREADY THRIVING ECOSYSTEM

Facebook tried to take over an entire smartphone with the HTC First and failed dismally. Amazon is currently in the process of learning the same bitter lessons with its own Fire Phone, and BlackBerry is too busy issuing weird new phones to look to other device categories. Microsoft is apparently working on making Windows wearable, though its rumored device will be a fitness-focused band rather than an out-and-out smartwatch. Almost everyone is thus evading a direct confrontation with the duopoly of Apple and Google.

The only serious holdout may be Samsung, whose Tizen-powered smartwatches (like the newGear S) could lie within a one-company ecosystem by sheer force of corporate muscle alone. Incumbent wearable leaders like Pebble, Jawbone, and Fitbit are wisely looking to embrace and work alongside the incoming wave of smartwatches rather than fight them directly. With longer battery life, cheaper prices, and lighter form factors, they have easy ways to differentiate themselves and survive as complementary products. Apple doesn’t care to fight for sales of devices with a low profit margin, and the Android Wear watches now on sale are comparatively expensive even while their makers are doing little more than breaking even.

The Pebble will continue to be a reasonable option while the Wear and Watch app stores are still maturing, however eventually it too will be surpassed by the greater variety of things you can achieve with a color display and gigabytes of memory. The smartwatches we are seeing today are still very raw, but they have already shown enough flashes of potential to hint at their widespread appeal once the current shortcomings have been ironed out.

There are still many unknown and as yet undetermined aspects of how (and why) smartwatches will be used in the future. Two things appear inevitable, however. One is that we will climb this mountain, out of curiosity as much as anything else, and the other is that we’ll probably do it with smartwatches acting as companions to phones rather than as autonomous devices. As things stand today, Apple and Google look set to extend their dominance in smartphones to smartwatches without any serious contestation from the rest of the tech world.

Wow… (HT: Kevin)

Originally formed for self-protection, prison gangs have become the unlikely custodians of order behind bars—and of crime on the streets.


Graeme Wood

SEPTEMBER 16, 2014

On a clear morning this past February, the inmates in the B Yard of Pelican Bay State Prison filed out of their cellblock a few at a time and let a cool, salty breeze blow across their bodies. Their home, the California prison system’s permanent address for its most hardened gangsters, is in Crescent City, on the edge of a redwood forest—about four miles from the Pacific Ocean in one direction and 20 miles from the Oregon border in the other. This is their yard time.

Most of the inmates belong to one of California’s six main prison gangs: Nuestra Familia, the Mexican Mafia, the Aryan Brotherhood, the Black Guerrilla Family, the Northern Structure, or the Nazi Lowriders (the last two are offshoots of Nuestra Familia and the Aryan Brotherhood, respectively). The inmates interact like volatile chemicals: if you open their cells in such a way as to put, say, a lone member of Nuestra Familia in a crowd of Mexican Mafia, the mix can explode violently. So the guards release them in a careful order.

“Now watch what they do,” says Christopher Acosta, a corrections officer with a shaved head who worked for 15 years as a front-line prison guard and now runs public relations for Pelican Bay. We are standing with our backs to a fence and can see everything.

At first, we seem to be watching a sullen but semi-random parade of terrifying men—heavily tattooed murderers, thieves, and drug dealers walking past one of five casual but alert guards. Some inmates, chosen for a strip search, drop their prison blues into little piles and then spin around, bare-assed, to be scrutinized. Once inspected, they dress and walk out into the yard to fill their lungs with oxygen after a long night in the stagnant air of the cellblock. The first Hispanic inmate to put his clothes on walks about 50 yards to a concrete picnic table, sits down, and waits. The first black inmate goes to a small workout area and stares out at the yard intently. A white guy walks directly to a third spot, closer to the basketball court. Another Hispanic claims another picnic table. Slowly it becomes obvious that they have been moving tactically: each has staked out a rallying point for his group and its affiliates.

Once each gang has achieved a critical mass—about five men—it sends off a pair of scouts. Two of the Hispanics at the original concrete picnic table begin a long, winding stroll. “They’ll walk around, get within earshot of the other groups, and try to figure out what’s going down on the yard,” Acosta says. “Then they can come back to their base and say who’s going to attack who, who’s selling what.”

Eventually, about 50 inmates are in the yard, and the guards have stepped back and congregated at their own rallying point, backs to the fence, with Acosta. The men’s movements around the yard are so smooth and organized, they seem coordinated by invisible traffic lights. And that’s a good thing. “There’s like 30 knives out there right now,” Acosta says. “Hidden up their rectums.”

A corrections officer at Pelican Bay conducts a search for contraband in an inmate’s cell.

Understanding how prison gangs work is difficult: they conceal their activities and kill defectors who reveal their practices. This past summer, however, a 32-year-old academic named David Skarbek published The Social Order of the Underworld, his first book, which is the best attempt in a long while to explain the intricate organizational systems that make the gangs so formidable. His focus is the California prison system, which houses the second-largest inmate population in the country—about 135,600 people, slightly more than the population of Bellevue, Washington, split into facilities of a few thousand inmates apiece. With the possible exception of North Korea, the United States has a higher incarceration rate than any other nation, at one in 108 adults. (The national rate rose for 30 years before peaking, in 2008, at one in 99. Less crime and softer punishment for nonviolent crimes have caused the rate to decline since then.)

If your name is on a Bad News List, gang members attack you on sight—but remove your name when your debts are paid.

Skarbek’s primary claim is that the underlying order in California prisons comes from precisely what most of us would assume is the source ofdisorder: the major gangs, which are responsible for the vast majority of the trade in drugs and other contraband, including cellphones, behind bars. “Prison gangs end up providing governance in a brutal but effective way,” he says. “They impose responsibility on everyone, and in some ways the prisons run more smoothly because of them.” The gangs have business out on the streets, too, but their principal activity and authority resides in prisons, where other gangs are the main powers keeping them in check.

Skarbek is a native Californian and a lecturer in political economy at King’s College London. When I met him, on a sunny day on the Strand, in London, he was craving a taste of home. He suggested cheeseburgers and beer, which made our lunch American not only in topic of conversation but also in caloric consumption. Prison gangs do not exist in the United Kingdom, at least not with anything like the sophistication or reach of those in California or Texas, and in that respect Skarbek is like a botanist who studies desert wildflowers at a university in Norway.

Skarbek, whose most serious criminal offense to date is a moving violation, bases his conclusions on data crunches from prison systems (chiefly California’s, which has studied gangs in detail) and the accounts of inmates and corrections officers themselves. He is a treasury of horrifying anecdotes about human depravity—and ingenuity. There are few places other than a prison where men’s desires are more consistently thwarted, and where men whose desires are thwarted have so much time to think up creative ways to circumvent their obstacles.

Because he is a gentleman, Skarbek waited until we’d finished our burgers to illustrate some of that ingenuity. “How can you tell what type of cellphone an inmate uses,” he asked, “based on what’s in his cell?” He let me think for about two seconds before cheerily giving me the answer: you examine the bar of soap on the prisoner’s sink. The safest place for an inmate to store anything is in his rectum, and to keep the orifice supple and sized for the (contraband) phone, inmates have been known to whittle their bars of soap and tuck them away as a placeholder while their phones are in use. So a short and stubby bar means a durable old dumbphone; broad and flat means a BlackBerry or an iPhone. Pity the poor guy whose bar of soap is the size and shape of a Samsung Galaxy Note.

The prevalence of cellphones in the California prison system reveals just how loose a grip the authorities have on their inmates. In 2013, the California Department of Corrections and Rehabilitation confiscated 12,151 phones. A reasonable guess might be that this represented a tenth of all cellphones in the system, which means that almost every one of the state’s 135,600 inmates had a phone—all in violation of prison regulations. “Prison is set up so that most of the things a person wants to do are against the rules,” Skarbek says. “So to understand what’s really going on, you have to start by realizing that people are coming up with complicated ways to get around them.” Prison officials have long known that gangs are highly sophisticated organizations with carefully plotted strategies, business-development plans, bureaucracies, and even human-resources departments—all of which, Skarbek argues, lead not to chaos in the prison system but to order.

Craig Canary, an inmate in Pelican Bay’s Security Housing Unit, in his solitary-confinement cell

Skarbek trained in an economic school of thought known as rational-choice theory, which aims to explain human behavior as the product of reasonable decisions by economic actors. In many cases, rational-choice theory has shown behaviors to be rational that at first appear wild, irrational, or psychopathic. When people are encouraged or forced to act against their economic interest, they find work-arounds as surely as water blocked by a boulder in a stream finds a way to flow around it.

In 1968, one of the founders of rational-choice theory, Gary Becker, wrote a pioneering paper, “Crime and Punishment: An Economic Approach,” premised on the idea that the prevailing view of crime required revision. According to prior dogma, criminals were best understood as mental defectives, crazy people who couldn’t control their impulses. Becker, who won a Nobel in economics in 1992 and died this past May, suggested instead that criminals offend because they make careful calculations of the probability and likely cost of getting caught—and then determine that the gamble is worthwhile. This insight, Skarbek says, opened the study of crime up to economic theory.

Skarbek attended graduate school at George Mason University, a bastion of rational-choice theory. Its faculty is also friendly to unorthodox subject matter: Robin Hanson has published papers about using betting markets to augment democratic government, and has proposed that it is rational to freeze one’s head after death; Peter Leeson wrote The Invisible Hook, a 2009 account of the economics of piracy. Skarbek’s doctoral adviser, Peter Boettke, showed how the behavior of the Soviet economy actually made sense if you viewed it as controlled not just by the government but also by the black- and gray-market activities of citizens.

Prison, Skarbek claims, is the ultimate challenge for a rational-choice theorist: a place where control of the economic actors is nearly total, and where virtually any transaction requires the consent of the authorities. The Soviets had far less control over their people’s economic activity than prison wardens do over the few dollars available for prisoners’ commissary purchases. Both settings have given rise to alternate currencies and hidden markets. Most famously, cigarettes have become the medium of exchange in many prisons. But when they are banned, other currencies take their place. California inmates now use postage stamps.

A scene from general-population housing

Among the fundamental questions about prison gangs—known in California-corrections argot as “Security Threat Groups”—is why they arise in the first place. After all, as Skarbek notes, California had prisons for nearly a century before the first documented gang appeared. Some states don’t have prison gangs at all. New York has had street gangs for well over a century, but its first major prison gang didn’t form until the mid-1980s.

The explanation, Skarbek says, can be found in demographics, and in inmate memoirs and interviews. “Before prison gangs showed up,” he says, “you survived in prison by following something called ‘the convict code.’ ” Various recensions of the code exist, but they all reduce to a few short maxims that old-timers would share with first offenders soon after they arrived. “It was pretty simple,” he explains. “You mind your own business, you don’t rat on anyone, and you pretty much just try to avoid bothering or cheating other inmates.”

If a white guy keeps everyone awake, the Aryan Brothers will discipline him to avoid having blacks or Hispanics attack one of their members.

But starting in the 1950s, things changed: The total inmate population rose steeply, and prisons grew bigger, more ethnically and racially mixed, and more unpredictable in their types of inmate. Prisons faced a flood of first offenders, who tended to be young and male—and therefore less receptive to the advice of grizzled jailbirds. The norms that made prison life tolerable disappeared, and the authorities lost control. Prisoners banded together for self-protection—and later, for profit. The result was the first California prison gang.

That moment of gang genesis, Skarbek says, forced an arms race, in which different groups took turns demonstrating a willingness to inflict pain on others. The arms race has barely stopped, although the gangs have waxed and waned in relative power. (The Black Guerrilla Family has been weakened, prison authorities told me, because of leadership squabbles.) The Mexican Mafia was the sole Hispanic gang until 1965, when a group of inmates from Northern California formed Nuestra Familia to counter the influence of Hispanics from the south. Gang elders—called maestros—instruct the youngsters in gang history and keep the enmity alive.

What’s astonishing to outsiders, Skarbek says, is that many aspects of gang politics that appear to be sources of unresolvable hatred immediately dissipate if they threaten the stability of prison society. For example, consider the Aryan Brotherhood—a notoriously brutal organization whose members are often kept alone in cells because they tend to murder their cell mates. You can take the Brotherhood at its word when it declares itself a racist organization, and you can do the same with the Black Guerrilla Family, which preaches race war and calls for the violent overthrow of the government. But Skarbek says that at lights-out in some prisons, the leader of each gang will call out good night to his entire cellblock. The sole purpose of this exercise is for each gang leader to guarantee that his men will respect the night’s silence. If a white guy starts yelling and keeps everyone awake, the Aryan Brothers will discipline him to avoid having blacks or Hispanics attack one of their members. White power is one thing, but the need to keep order and get shut-eye is paramount.

Another common misconception about prison gangs is that they are simply street gangs that have been locked up. The story of their origins, however, is closer to the opposite: the Mexican Mafia, for example, was born at Deuel Vocational Institution, in Tracy, California, in 1956, and only later did that group, and others, become a presence on the streets. Today, the relation of the street to the cellblock is symbiotic. “The young guys on the street look to the gang members inside as role models,” says Charles Dangerfield, a former prison guard who now heads California’s Gang Task Force, in Sacramento. “Getting sentenced to prison is like being called up to the majors.”

But Skarbek says the prison gangs serve another function for street criminals. In a 2011 paper in American Political Science Review, he proposed that prison is a necessary enforcement mechanism for drug crime on the outside. If everyone in the criminal underworld will go to prison eventually, or has a close relationship with someone who will, and if everybody knows that gangs control the fate of all inmates, then criminals on the street will be afraid to cross gang members there, because at some point they, or someone they know, will have to pay on the inside. Under this model, prison gangs are the courts and sheriffs for people whose business is too shady to be able to count on justice from the usual sources. Using data from federal indictments of members of the Mexican Mafia, and other legal documents, Skarbek found that the control of prisons by gangs leads to smoother transactions in the outside criminal world.

Gangs effect this justice on the inside in part by circulating a “bad-news list,” or BNL. If your name is on a BNL, gang members are to attack you on sight—perhaps because you stole from an affiliate on the outside, or because you failed to repay a drug debt, or because you’re suspected of ratting someone out. Skarbek says one sign that the BNL is a rationally deployed tool, rather than just a haphazard vengeance mechanism, is that gangs are fastidious about removing names from the list when debts are paid.

An inmate of the Security Housing Unit at Pelican Bay is flanked by corrections officers as he is transported from one area of the unit to another.

No scholar writing in the law-abiding world, I was told by guards at Pelican Bay, can capture the reality of prison life in all its brutality. I was prepared for that to be true, even just based on my own reading. In 2005, Don Diva magazine interviewed a former guard at Rikers Island, who described the conditions of prison life in vivid terms. “[In each cell] you have a filthy toilet with no cover, a rusty sink, and a metal frame they call a bed,” he told the magazine. “Inmates use the toilet as a refrigerator in the summer to keep milk cool.” More vivid still was his description of inmate survival tactics:

Inmates are legendary for keeping razors in their mouths. Being able to “spit out a razor” is like a magic trick in jail. You could be in the mess hall, get into an altercation with another inmate, and the next thing you know he’s spit out two razors from both sides of his mouth and your face is slashed up … A nigga will become Houdini when it comes to survival. Spitting razors became such a problem that inmates immediately punched other inmates in the mouth as soon as an argument began. This was so that if the other inmate did have razors in his mouth, he would cut his own mouth up before even getting the opportunity to spit them out.

But I found that the staff at Pelican Bay had already been thinking about prisons the way Skarbek does. While I was there, Lieutenant Jeremy Frisk, the prison’s Institutional Gang Investigator, delivered a half-hour PowerPoint presentation focused on the managerial ingenuity of the gang leaders. One of the last slides featured a picture of the Chrysler chairman and 1980s business icon Lee Iacocca. “He was a very good manager,” Frisk said, “and turned Chrysler around from the brink of bankruptcy. And he could do that just from his management strategy: he never turned a wrench on a car, never assembled a door. But because of his ideas, they could make millions of dollars.” Frisk said gang leaders are the Lee Iacoccas of the prison world: brilliant managers of violence. (Since that presentation, I have found it impossible to look at a picture of Iacocca without imagining him stuffing his cheeks and rectum with razor blades.)

The safest place for an inmate to hide a phone is in his rectum. To keep the orifice sized and supple, inmates have been known to tuck a bar of soap away as a placeholder.

Pelican Bay opened in 1989 as an upgraded version of two famous old California prisons, San Quentin and Folsom, both of which still house inmates but function, as they always have, like enormous holding pens, hardly optimal for supervising a population of violent psychopaths who plot constantly to subvert the rules of the institution. Even the most secure housing at San Quentin, says Pelican Bay’s acting warden, Clark Ducart, was built so prisoners could all go from their cells to the yard together, with 50 men moving as an ungovernable mass. The walkways were narrow, and exposed prisoners to each other in ways that encouraged attacks. “As you walked guys to the shower,” he told me, “they’d get stabbed or speared.” Pelican Bay, by contrast, allows much greater levels of control, and a much more oppressive existence for anyone trying to plot a crime. The population is sectioned into yards and blocks that might have little contact with one another, and that allow the inmates to be managed with special attention to their gang affiliation. Upon identifying a gang member, the prison can modulate his location, freedom, and level of surveillance, to a degree that inmates have called stifling and inhumane.

On every cellblock at Pelican Bay, the guards post plastic identity cards on the wall, to keep track of which inmate is in which cell. These cards include each inmate’s name and photo. But the most-important information is conveyed by the cards’ color, which roughly correlates with probable gang affiliation: green for northern Hispanics, pink for southern Hispanics, blue for blacks, white for whites, and yellow for others, including American Indians, Mexican nationals, Laotians, and Eskimos. The information is crucial to the smooth running of the institution. Maintaining balance in a cellblock, and not putting a lone gang member in a situation where he might be surrounded by members of a rival gang, requires constant attention on the part of the corrections officers.

Out in the yard, when Acosta and I watched the inmates gather by gang, the guards knew exactly what was happening, and they could have intervened and broken up obvious gang activity. And it was obvious: nearly all gang members have gang tattoos across their torsos, and some have markings on their faces too. As Robert Mitchum growled in the remake ofCape Fear: “I don’t know whether to look at him or read him.”

Each interaction we observed between a correctional officer and a prisoner resembled bargain more than diktat. Before yard time finished, the guards let me inspect cells with them. The cells were livable, especially in comparison to the Rikers Island ones I had read about, even if the whole block had a dank locker-room smell. When I peeked in an inmate’s cell, I saw a dirty metal object in the sink. It was blunt and had a wire attached. “Stinger,” Acosta said. “Inmates use it to boil water. It’s illegal, but if the inmate isn’t doing anything wrong, a guard might let it pass.” He said that if a guard discovered a contraband item during an inspection, he might place it on the inmate’s bunk, just to show that he knew about it and could confiscate it at any time, if the inmate didn’t behave.

The guards asked inmates to show me a technique called “fishlining,” which involves attaching an object to one end of a string, sliding it out of a cell and into the hallway, and then using the other end of the string to yank it across the floor, this way and that, until it slides in front of the desired cell. A shatter-toothed Aryan Brother smiled at me and said he could send a book to an adjacent cell this way. (On his shelf: a single-volume edition of The Chronicles of Narnia and a Teach Yourself book on German.) The fishlines work as a way to distribute contraband, but are also used, Skarbek told me, as a sort of corporate communications system—like pneumatic tubes for prisoners.

The messages inmates send include extensive questionnaires for new arrivals. Nuestra Familia is particularly sophisticated, and, in a sure sign of bureaucratization, the gang even has an initialism for its new-arrival questionnaire: NAQ. “When you get put in your cell, and the door slams shut, you might get a fishline with a piece of paper on it,” Skarbek says. “And you’ll be expected to answer the questions in full.” The survey might include questions about your offense, your judge, and your relatives in other prisons. But it could also ask where you lived on the outside and what resources you have that could be valuable to the gang. The questionnaires are collated and checked. At some prisons, inmates use their cellphones to confirm details on Facebook, and Skarbek says they have been known to open LexisNexis accounts. Gang members are trained in micrography—the writing and decipherment of very tiny letters—so they can produce tightly rolled pieces of paper, called “kites,” to be transported from prison to prison in the usual orifice. These activity reports circulate around the prison system. Christopher Acosta showed me a kite that had been intercepted at Corcoran State Prison, reporting on a gang’s battle with a rival there.

An inmate doing push-ups in the SHU’s exercise yard, a small concrete room with an overhead skylight where inmates are allowed to spend an hour and a half a day and receive their only exposure to sunlight

Finding kites is difficult, because guards cannot cavity-search every inmate every day. The only way to control known gang members is to confine them under strict conditions that make communication almost, but not quite, impossible—no freedom of movement or circulation with the general prison population, for example, and only rare, carefully monitored visits.

Over the years, California has tried two broad strategies for gang management. The first was to break up gangs and scatter their members to distant prisons where their influence would be divided and diluted. That strategy too frequently allowed gangs to metastasize, effectively seeding the whole prison system, and even other states’ and the federal system, with gang activity. The current strategy, implemented in the 1990s, is to identify high-level gang members (a process called “validation”) and bring most of them to Pelican Bay.

Pelican Bay is far from the gangs’ strongholds of Los Angeles and the Central Valley. In every direction there is little more than redwoods, marijuana farms, and seacoast. More important, Pelican Bay has the facilities and knowledge necessary to isolate and neutralize gang members. In Sacramento, the Department of Corrections and Rehabilitation has posters on the wall showing mug shots of all the major gang leaders—the Lee Iacoccas, Steve Jobses, and Henry Fords of the underworld—grouped by the prisons they live in. Most are at Pelican Bay, probably for life, in a snowflake-shaped building called the Security Housing Unit, or SHU (pronounced “shoe”).

Of course, there are ways to control inmates that American prisons have never tried on a large scale. Skarbek points out that the gay-and-transgender unit of the Men’s Central Jail in Los Angeles County is safe and gang-free—so much so that prison officials have had to screen out straight Angelenos who play gay just to keep away from gangs. That jail is simply small and well administered, argues Sharon Dolovich, a UCLA professor who studied it, and it’s not clear that its methods could scale up. We could easily replicate less enlightened penal practices, too. “In other countries, they can use corporal punishments not available to authorities in American prisons,” Skarbek says—a bullet in the back of the neck is a strong deterrent to any Chinese gang that might form behind bars. Within the bounds of American civil rights, though, we are left with prisons whose smooth operation relies in part on the predatory activities of gangs—and with facilities like the SHU, which is California’s effort to control the gangs by subjecting their leaders to levels of surveillance and restriction far beyond what most American inmates face.

The gay-and-transgender unit is safe and gang-free—so much so that prison officials have had to screen out straight Angelenos who play gay just to keep away from gangs.

Walking into the SHU feels like entering a sacred space. After the clanging of doors behind you, a monastic silence reigns. The hallways radiate from the command center at the hub of the SHU snowflake, and each one has chambers on either side that sprout chambers of their own. The hallways echo with footsteps when you walk down them. There are no prison noises: no banging of tin cups, no screaming of the angry or insane. The silence is sepulchral, and even when you get to branches of the snowflake, where the inmates actually live, it seems as if everyone is in suspended animation, on one of those interstellar journeys that last multiple human lifetimes.

In fact, many are just watching television while wearing headphones. In the company of Christopher Acosta, I visited a cellblock where fewer than a dozen cells held men, most of them living without cell mates. Before entering, I met a female security guard who, after demanding that I display my identification card more prominently, showed me a board with inmates’ pictures on it, each color-coded. Hispanics and whites predominated. She showed me the slips of paper indicating that a couple of inmates wanted halal food, although she said she suspected the meal requests were a way to break monotony and create work for the staff, rather than as an expression of any authentic religious conviction. She said the inmates were allowed televisions with the speakers disabled, as well as 10 books at a time.

The other Pelican Bay inmates were enjoying time together in the main yards, but these hard-core gang members didn’t have that option. Instead, they could go to a large, featureless concrete room at the end of the block for daily solitary exercise. The “yard” had a plexiglass roof that allowed them to see the sky above, and a small drainage hole in the floor, through which they could sometimes communicate faintly with other inmates on other cellblocks. Last year, gang members used the drainage pipes of their in-cell toilets to communicate clandestinely across cellblocks and coordinate a hunger strike by inmates statewide, to protest the conditions in the SHU.

With a buzz and a clang, the guard opened the last door, and Acosta and I entered the cellblock. He warned me that no one would talk. We had spent much of the day discussing the violent proclivities of the men under lockdown at Pelican Bay—how they became experts at weapons craftsmanship, for example, and could fashion the metal post of a bunk bed or the edge of a cell door into a spear, known as a “bone crusher,” that could be flung from inside a cell and penetrate a man’s neck or liver. So I expected hostile interviews, if any at all.

One of the first men I saw turned out to be genial but squirrely. He was Hispanic, refused to give his name, and babbled away about how prison gangs are “just a thing,” never quite articulating what that meant. The only sentence he said that made any sense was that he was in for life for killing two people. The door that separated him from me was a steel plate with small holes in it. After just a few seconds of his talking, I got a headache, partly from his mad monologue and partly from the odd moiré effect of looking at him through the screen.

As I passed down the line of cells, I tried talking to everyone but got little response. One heavily tattooed Hispanic man flicked his hand at me from behind the steel door, as if to shoo away a flea. Most ignored me, and the few who paid any attention just stared at me like I was prey and said nothing other than “no.” Finally one man with large glasses and a thick black mustache said, “Prison gangs? There ain’t no prison gangs here.” He then turned to a blank wall and started doing calisthenics.

When I emerged, and the door had clanged again behind me, I told the guard I hadn’t managed to talk with anyone. She was not surprised. Any conversation they attempted, she said, might be overheard and used against them.

But there are limits to what even the most carefully designed prisons can constrain. The guard and I were talking in library voices, and no sounds came from the row of cells nearby. “It’s quiet,” I said, lowering my voice. “Can they hear what we’re saying?”

“Every word,” she said. “Every single word.”