Command Line Basics

If you really want to master the command line, Zed Shaw’s Command Line Crash Course is probably indispensible. I’ve never actually taken it but I’ve taken other Zed Shaw classes and he’s good at what he does.

By default, OSX’s Terminal (and Ubuntu) both use Bash. Bash is actually just one of many command line interpreters out there, but comparing the relative merits of shell environments is well, well beyond the scope of our class. You probably won’t use the command line much in your career as a journalist but if you’re going to work with data, there are times when it can save you from tearing your hair out. If you go to Socrata you can download a 2.24 GB file that contains a record of every single call to 311 since 2010. You can’t open it in Excel, however — it is too big.

So if you want to get some kind of handle on what is in the file, you need some way to examine it without opening it. Enter the shell. Continue reading Command Line Basics

Troubleshooting Highcharts

When you’re ready to embed a chart, JSFiddle’s build in iframe tool is handy, but in the long run it isn’t what you want. Embedding Fiddles is fiddly, so instead of mastering JS Fiddle learn how to put a chart on your own page.

I put together some very useful Bootstrap templates including two Highcharts templates that are your best starting points. If your chart isn’t showing up where you think it ought to, try checking the following:

  • Did you place a <div..>…</div> on the page where you want the chart to appear?
  • Does your <div> have a descriptive, one word id? id="container" is not descriptive. id="time_to_leak" is.
  • Is your Highcharts function looking for the right id?
    $(function () {$('#oscar_night').highcharts({ is looking for a div with id="oscar_night"
  • Is your jquery call above your function? (By “jquery call” I mean the script tag that includes a minified copy of jQuery, probably from a content delivery network somewhere. Something like <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
  • Is your highcharts call below your function? (That’s the line that pulls in Highcharts’ scripts <script src="//code.highcharts.com/highcharts.js"></script>)

Festival of data – Jillian Eugenios

Check out this interactive piece combining data and food – http://issuu.com/ryanmaceachern/docs/designxfood

I love it because it is instantly immersive and interactive. The color palette is consistent, and it is visually engaging.

A big problem here is the teeny tiny text. The opening text, which tells the story, is white on a light blue background. It’s tough to read. Having a first page like that could deter some readers.

I like the flip-book style, and the attempt it makes at a narrative. And all pie charts should be in cereal bowl form. 🙂

But look at the little text! How can we appreciate it when we can’t see it?

I would also really appreciate a birds-eye view of all of them to be able to compare.

 

 

Homework Week 8 (Due March 28)

Everyone should clean up their Bootstrap demo page and be working on Story 1 Rough Drafts.

I’ll be in the newsroom from 2-7 on Wednesday to help you over any last hurdles, but you should also feel free to email me if anything comes up. Just be sure to keep the principles of “Good Questions” in mind: http://hickman.spring-2014.dataviz.journalism.cuny.edu/2014/02/07/asking-good-questions/

Your rough draft should be complete, reporting-wise. If you want the layout to work a bit differently or would like to fine tune the visual feel of your maps or charts, that’s something we can work on in class. Your rough draft should be on an HTML page and uploaded to Digital Storage.

Send me the URL for your Rough Draft by 8 AM on Friday March 28.

HTML Fundamentals

We’re using bootstrap to manage most page layout needs, but you still need a little bit of HTML to place images, link to other pages or add breaks between paragraphs.

I like ReText for Ubuntu Linux or Mou for OSX.

Both programs allow you to write your text in a very simplified language called “markdown” — and they’ll produce clean HTML from it. By default a blank Mou document has everything you need to know about markdown right inside it.

I like a lot of things about writing in Markdown: you can stay focused on the content because it is super readable. Markdown enforces structured, heirarchical HTML which will make the web team swoon when you move to a working newsroom.

You can’t do much formatting in Mou, but you shouldn’t be doing much formatting anyway: focus on your reporting.

If you really want to start to understand how HTML works, these readings might help:

Why can’t I just use Dreamweaver?

You might have figured out by now that you can save a Word document as HTML. You might even have a “What You See is What You Get” (or WYSIWYG, prounounced whizzy wig) editor like Dreamweaver Handy. The problem is that both of these will do their utmost to replicate exactly what you’ve got on the screen in the HTML they produce. You won’t be able to read the results and you’ll have a hard time wrestling them into a different template.

Trust me: if you want more fine grained control than you can get in Bootstrap, a WYSIWIG editor is not the path to follow.

Think of it this way: you can make spaghetti sauce from a diced onion, two cloves of garlic and a can of tomatoes. Someone says to add a bay leaf, some thyme and oregano. Then your overbearing gourmet friend says you must start with a soffritto and … at this point no one would blame you for eying the jarred sauce aisle. If you’re just starting out as a cook, you are far, far better off making a super simple sauce from scratch so you can build on it. (There is a Cosby Show episode about what happens if you heat up a jar of sauce and try to pass it off as homemade: You will not impress Clair Huxtable.)

Plus, the truth is that you can read the ingredients on a jar of Ragu, but you can’t read the ingredients on generated HTML and you definitely can’t get the sugar out and the time you’d spend trying to figure out how to make the sauce taste less tinny would be far better spent dicing an onion and getting on with it.

Instead, use a Markdown editor to compose your text and break it into paragraphs, lists and blockquotes, and then use Bootstrap to lay it out.

Bootstrap and SimpleHTTP

Launching a webserver

It seems like an extra hurdle, but getting a little web server running goes a long way towards being able to test out your code before you upload it to a real web server.

Step 0: Start to organize the files that you’re putting on Digital Storage into a single directory so you can keep track of them.
Step 1: Enable New Terminal at Folder
Step 2: Navigate to the folder that contains your HTML files. Right click on the folder in your Finder window and select “New Terminal at Folder” — it might be in a “Services” sub-menu.
Step 3: Check your directory with pwd (Just type pwd at the command prompt and hit return)
Step 4: Launch Python’s Simple HTTP Server with python -m SimpleHTTPServer

Now you can visit http://localhost:8000 and see your html files served over a little local web server.

Bonus: in another terminal window try running python --help — see if you can figure out what the -m flag is doing.

Troubleshooting

Read the error messages you see and think about what they’re saying. If you see Firefox can't establish a connection to the server at localhost:8000. that means there’s no server running on port 8000. You might have the port number wrong, or SimpleServer might have stopped running. Luckily, it is easy enough to restart. If you see :

Error response
Error code 404.
Message: File not found.
Error code explanation: 404 = Nothing matches the given URI.

that is a clue that either you tried to open a page that doesn’t exist (check the spelling?) or your web server is running from the wrong folder.

Bootstrap

The secret truth is that you can load Bootstrap’s CSS library without a web server, but getting used to testing things over a server will pay off in the long run. I promise.

The reason we’re doing it now is so we can include script and CSS files with protocol-relative URLs. Protoco-huh? For now, just trust me. Brocolli is better for you than brownies even though brownies won’t kill you. Drink lots of water, get plenty of sleep and at least a little vigorous exercise, even if you don’t really understand.

We walked through some basics of Bootsrap: you need to call their CSS in the page header, and if you’re using any javascript you need to call their javascript before you close the body of your page. You need a valid doctype definition and a few particular meta tags.

Get started with my template, and remember to keep an eye on the heirarchy. Everything but the doctype goes in either the head or the body (and head and body belong inside of html tags). The head includes a few meta tags, a title and any link tags that call style sheets. Sometimes you’ll put script tags in the head, sometimes you want them right before the closing body tag. Why? That’s complicated. Almost everything else will go inside the body tag.

If you’re using Bootstrap, start by putting everything inside a container div:

<div class="container">
    (stuff goes here)
</div>

You can start adding other elements — paragraphs, a jumbotron or a set of tabs — inside that container.

Quickfloats are handy for pullquotes. You can let them find their own size or use the grid to set their width.

Bootstrap does a lot of fancy footwork to customize layouts on tiny, small, medium and large screens. For now, I suggest that you stick to the col-md- classes. They’re the most universal. Our goal here is to tell good stories, not master the finer points of CSS layout, with or without Bootstrap.

We didn’t talk much about paragraphs, blockquotes or anchors, but you’ll need them if you don’t want your content to run together in a great heap. You can wrap paragaphs in <p> tags manually, but if you want some help with the basic pieces, I’m a huge fan of Mou. You can compose in Markdown, a heavily simplified syntax, and then copy the resulting HTML into a Bootstrap template. Markdown is a very basic language — you can’t do a lot with it, so you can focus on the basics.

Johnson/Sharif: Politician Salaries

A proposed title or headline

What’s your City Official’s Payroll?

A story slug — up to three words that capture the essence of your story

Politician salaries

Key elements

New York City officials are often paid $80,000-$500,000 a year.

At least 10 officials have been found on misusing taxpayer’s money in the past two years.

A news hook, or explanation of why this story matters now

The Queens Library director was charged with using taxpayer’s money for personal use. The story will focus on how this alleged corruption could happen how much of New York taxes go to salaries, and how are the salaries determined and distributed.

Links to data

Payroll for all branches and public offices

http://seethroughny.net/payrolls/city-of-ny/

NYC City Council Districts

https://www.google.com/fusiontables/DataSource?dsrcid=1768167

Ivanova/Willens: Medical Data Breaches

A proposed title or headline

Who’s losing your data?

A story slug — up to three words that capture the essence of your story

Data breaches

Key elements

– Since 2009, more than 2.5 million medical records have been lost or stolen in New York State. That means one in ten New Yorkers has had their data compromised.

– Institutions reporting breaches of medical data run the gamut from insurance providers to public hospitals to prestigious university-affiliated medical centers.

– The overwhelming majority of these breaches occur with electronic (not paper) records.

A news hook, or explanation of why this story matters now

In brief, medical data means big money–both for its legitimate users and to people selling it on the black market.

The New York State Department of Health is on the verge of rolling out the State Health Information Network of New York (SHIN-NY), a network that allows patients’ medical information to travel more easily between providers. While medical providers stand to gain “bonuses” in the form of federal money for converting to medical records, patient advocates have raised concerns with the project, including lack of transparency and patient consent.

We want to show the extent to which data breaches exist in the current system, and the overlap between institutions that can’t secure their data now with those who will be part of the SHIN-NY (spoiler: there’s a lot of them.)

Links to data

Data breaches in New York state since 2009: http://www.hhs.gov/ocr/privacy/hipaa/administrative/breachnotificationrule/breachtool.html

A cleaned-up file is here (that we’re trying to geocode):

https://docs.google.com/a/journalism.cuny.edu/spreadsheet/ccc?key=0Alfw-9hKKmAHdHJMMVFUZWg3c2YyblQ3MjJoc3dRMFE&usp=drive_web#gid=0

We’re also thinking about ways to map overlaps in patients of the various entities on teh “breach list.” In other words, how likely is it that someone who’s a patient at Doctor X’s also has a health plan with carrier y? Possibility here for an intelligent guess about how many people have had their medical records compromised more than once.

Sources

Dr. Latanya Sweeney, Harvard/FTC — technologist & privacy expert. latanya@fas.harvard.edu

Dr. Deborah Peel, founder, Patients’ Privacy Rights. dpeelmd@patientprivacyrights.org

Corinne Carey, New York Civil Liberties Union; advisor to SHIN-NY. ccarey@nyclu.org.

Eric Fader, health care lawyer, Day Pitney.  efader@daypitney.com // (212) 297 2413