book

Beautiful Data

by Toby Segaran, Jeff Hammerbacher

July 2009

Beginner to intermediate

384 pages

12h 56m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Beautiful Data
SPECIAL OFFER: Upgrade this ebook with Oâ€™Reilly
A Note Regarding Supplemental Files
Preface
How This Book Is Organized
Conventions Used in This Book
Using Code Examples
How to Contact Us
SafariÂ® Books Online
1. Seeing Your Life in Data
Personal Environmental Impact Report (PEIR)
your.flowingdata (YFD)

Personal Data Collection
Working Data Collection into RoutineAsynchronous data collection
Data Storage
Data Processing
Data Visualization
PEIRMapping location-based dataExperimenting with visual cuesMapping multivariate location tracesChoosing a color schemeMaking trips interactiveDisplaying distributionsSharing personal dataYFD
The Point
How to Participate
2. The Beautiful People: Keeping Users in Mind When Designing Data Collection Methods
Introduction: User Empathy Is the New BlackWhat Is UX?The Benefits of Applying UX Best Practices to Data Collection
The Project: Surveying Customers About a New Luxury Product
Specific Challenges to Data Collection
Challenges of AccessibilityChallenges of PerceptionBuilding trustLength of surveyAccurate data collectionMotivation
Designing Our Solution
Design PhilosophyDesigning the Form LayoutWeb form typography and accessibilityGiving them some spaceAccommodating different browsers and testing for compatibilityInteraction design considerations: Dynamic form lengthDesigning trustDesigning for accurate data collectionMotivationReporting the live data results
Results and Reflection
3. Embedded Image Data Processing on Mars
Abstract
Introduction
Some Background
To Pack or Not to Pack
The Three Tasks
Slotting the Images
Passing the Image: Communication Among the Three Tasks
Getting the Picture: Image Download and Processing
Image Compression
Downlink, or, It's All Downhill from Here
Conclusion
4. Cloud Storage Design in a PNUTShell
Introduction
Updating Data
The ChallengeOur ApproachMore on mastershipSupporting ordered dataTrading off consistency for availability
Complex Queries
The ChallengeOur Approach
Comparison with Other Systems
Google's BigTableAmazon's DynamoMicrosoft Azure SDSOther Related SystemsOther Systems at Yahoo!
Conclusion
Acknowledgments
References
5. Information Platforms and the Rise of the Data Scientist
Libraries and Brains
Facebook Becomes Self-Aware
A Business Intelligence System
The Death and Rebirth of a Data Warehouse
Beyond the Data Warehouse
The Cheetah and the Elephant
The Unreasonable Effectiveness of Data
New Tools and Applied Research
MAD Skills and Cosmos
Information Platforms As Dataspaces
The Data Scientist
Conclusion
6. The Geographic Beauty of a Photographic Archive
Beauty in Data: Geograph
Visualization, Beauty, and Treemaps
What Is Beauty in Visual Data Exploration?Making Treemaps Beautiful: A Geographic Perspective
A Geographic Perspective on Geograph Term Use
Representing the Term HierarchyRepresenting Absolute Location with ColorRepresenting Relative Location with Spatial TreemapsRepresenting Location Displacement
Beauty in Discovery
Reflection and Conclusion
Acknowledgments
References
7. Data Finds Data
Introduction
The Benefits of Just-in-Time Discovery
Corruption at the Roulette Wheel
Enterprise Discoverability
Federated Search Ain't All That
Directories: Priceless
Relevance: What Matters and to Whom?
Components and Special Considerations
The Existence of, and Availability of, ObservationsThe Ability to Extract and Classify Features from the ObservationsThe Ability to Efficiently Discover Related Historical ContextThe Ability to Make Assertions (Same or Related) About New ObservationsThe Ability to Recognize When New Observations Reverse Earlier AssertionsThe Ability to Accumulate and Persist This Asserted ContextThe Ability to Recognize the Formation of Relevance/InsightThe Ability to Notify the Appropriate Entity of Such Insight
Privacy Considerations
Conclusion
8. Portable Data in Real Time
Introduction
The State of the Art
TransportXMPPBitTorrentProprietary/P2PFormatsAPIsPollingRate limitingGetting it rightZero miles per gallon efficiencyEventsHTML 5 eventsWAN Scale Events
Social Data Normalization
Business Value of DataPublic versus private
Conclusion: Mediation via Gnip
9. Surfacing the Deep Web
What Is the Deep Web?
Alternatives to Offering Deep-Web Access
Basics of HTML Form ProcessingQueries and Query TemplatesSelecting Input CombinationsQuality of query templatesInformativeness testSearching for informative query templatesPredicting Input ValuesGeneric text inputsTyped text inputs
Conclusion and Future Work
References
10. Building Radiohead's House of Cards
How It All Started
The Data Capture Equipment
Velodyne LidarGeometric Informatics
The Advantages of Two Data Capture Systems
The Data
Capturing the Data, aka "The Shoot"
The Outdoor Lidar ShootThe Indoor Lidar ShootThe Indoor GeoVideo Shoot
Processing the Data
Post-Processing the Data
Launching the Video
Conclusion
11. Visualizing Urban Data
Introduction
Background
Cracking the Nut
Making It Public
Revisiting
Conclusion
12. The Design of Sense.us
Visualization and Social Data Analysis
Data
Visualization
Design ConsiderationsFoster personal relevanceProvide effective visual encodingsMake each display distinctSupport intuitive explorationBe engaging and playfulVisualization DesignsJob VoyagerBirthplace VoyagerU.S. census state map and scatterplotPopulation pyramidImplementation details
Collaboration
View SharingDoubly Linked DiscussionPointing via Graphical AnnotationCollecting and Linking ViewsAwareness and Social NavigationUnobtrusive Collaboration
Voyagers and Voyeurs
Hunting for PatternsMaking Sense of It AllCrowd Surfing
Conclusion
References
13. What Data Doesn't Do
When Doesn't Data Drive?1. More Data Isn't Always Better2. More Data Isn't Always Easy3. Data Alone Doesn't Explain4. Data Isn't Good for a Single Answer5. Data Doesn't Predict6. Probability Isn't Intuitive7. Probabilities Aren't Intuitive8. The Real World Doesn't Create Random Variables9. Data Doesn't Stand Alone10. Data Isn't Free from the Eye of the Beholder
Conclusion
References
14. Natural Language Corpus Data
Word Segmentation
Secret Codes
Spelling Correction
Other Tasks
Language IdentificationSpam Detection and Other Classification TasksAuthor Identification (Stylometry)Document Unshredding and DNA SequencingMachine Translation
Discussion and Conclusion
Acknowledgments
15. Life in Data: The Story of DNA
DNA As a Data StoreDNA Makes RNA Makes ProteinsHacking Your DNA Data Store with DrugsCancerReplicationCracking the CodeDNA As Digital StorageEvolution As an Algorithm
DNA As a Data Source
A Quantum Leap"My God, It's Full of Basesâ€¦"
Fighting the Data Deluge
The Sanger Institute's Sequencing PlatformProject managementFlexible Data CaptureInstrument and Data Management
The Future of DNA
How to Become a Genetic HackerNext Next-GenThe Era of Big Data
Acknowledgments
16. Beautifying Data in the Real World
The Problem with Real Data
Providing the Raw Data Back to the Notebook
Validating Crowdsourced Data
Representing the Data Online
Unique Identifiers for Chemical EntitiesOpen Data and Accessible Services Enable a Wide Range of Visualization and Analysis OptionsIntegrating Data with a Central Aggregation ServiceEnabling Data Integration via Unique Identifiers and Self-Describing Data Formats
Closing the Loop: Visualizations to Suggest New Experiments
Building a Data Web from Open Data and Free Services
Acknowledgments
References
17. Superficial Data Analysis: Exploring Millions of Social Stereotypes
Introduction
Preprocessing the Data
Exploring the Data
Age, Attractiveness, and Gender
Looking at Tags
Which Words Are Gendered?
Clustering
Conclusion
Acknowledgments
References
18. Bay Area Blues: The Effect of the Housing Crisis
Introduction
How Did We Get the Data?
Geocoding
Data Checking
Analysis
The Influence of Inflation
The Rich Get Richer and the Poor Get Poorer
Geographic Differences
Census Information
Exploring San Francisco
Conclusion
References
19. Beautiful Political Data
Example 1: Redistricting and Partisan Bias
Example 2: Time Series of Estimates
Example 3: Age and Voting
Example 4: Public Opinion and Senate Voting on Supreme Court Nominees
Example 5: Localized Partisanship in Pennsylvania
Conclusion
References
20. Connecting Data
What Public Data Is There, Really?
The Possibilities of Connected Data
Within Companies
Impediments to Connecting Data
The Representation ProblemShared Nouns and Shared VerbsThe Same Thing with Different NamesDifferent Things with the Same Name
Possible Solutions
Matching on Multiple FieldsCollective Reconciliation
Conclusion
A. Contributors
Index
About the Authors
COLOPHON
SPECIAL OFFER: Upgrade this ebook with Oâ€™Reilly

Overview

In this insightful book, you'll learn from the best data practitioners in the field just how wide-ranging -- and beautiful -- working with data can be. Join 39 contributors as they explain how they developed simple and elegant solutions on projects ranging from the Mars lander to a Radiohead video. With Beautiful Data, you will:

Explore the opportunities and challenges involved in working with the vast number of datasets made available by the Web
Learn how to visualize trends in urban crime, using maps and data mashups
Discover the challenges of designing a data processing system that works within the constraints of space travel
Learn how crowdsourcing and transparency have combined to advance the state of drug research
Understand how new data can automatically trigger alerts when it matches or overlaps pre-existing data
Learn about the massive infrastructure required to create, capture, and process DNA data

That's only small sample of what you'll find in Beautiful Data. For anyone who handles data, this is a truly fascinating book. Contributors include:

Nathan Yau
Jonathan Follett and Matt Holm
J.M. Hughes
Raghu Ramakrishnan, Brian Cooper, and Utkarsh Srivastava
Jeff Hammerbacher
Jason Dykes and Jo Wood
Jeff Jonas and Lisa Sokol
Jud Valeski
Alon Halevy and Jayant Madhavan
Aaron Koblin with Valdean Klump
Michal Migurski
Jeff Heer
Coco Krumme
Peter Norvig
Matt Wood and Ben Blackburne
Jean-Claude Bradley, Rajarshi Guha, Andrew Lang, Pierre Lindenbaum, Cameron Neylon, Antony Williams, and Egon Willighagen
Lukas Biewald and Brendan O'Connor
Hadley Wickham, Deborah Swayne, and David Poole
Andrew Gelman, Jonathan P. Kastellec, and Yair Ghitza
Toby Segaran

Become an Oâ€™Reilly member and get unlimited access to this title plus top books and audiobooks from Oâ€™Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on Oâ€™Reilly

Oâ€™Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the Oâ€™Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

Iâ€™ve been on the Oâ€™Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.