Gawker, Graydon Carter & Robots.txt
2009, Daniel Stewart
![]() Kottke, Jason 2002 Gawker Logo |
} |
GawkerW is a well-trafficked blog that trades in “Gossip from Manhattan and the Beltway to Hollywood and the Valley.” It is the flagship property of the Gawker Media EmpireW, which is owned by Nick DentonW. |
![]() Editor’s Letter, November 2009 |
} |
Graydon CarterW is the editor of Vanity Fair magazineW, a property owned by Condé Nast PublicationsW. To say he has had a successful career in publishing might be considered an understatement. Graydon Carter embodies the swagger and style of what a prominent Manhattan magazine editor should be, an influential individual who co-owns two restaurants that you will never be allowed to eat at. |
|
They say, “Never pick a fight with anyone who buys ink by the barrel,” or for that matter, “bandwidth by the terabyte.” Nick Denton, Gawker, and Graydon Carter are giants with which one might be wise not to engage. I tread lightly henceforth: | ||
*** | ||
![]() http://gawker.com/robots.txt |
} |
Robots.txtW is a file often included in the root directory of most website domains. It is a protocol that tells search engines, like Google, where to go via the Allow: function, and where not to go via the Disallow: function. Like many websites, the server that hosts Gawker’s content contains the robots.txt protocol. A quick glance, and Gawker’s robots.txtM looks fairly typical. Like many sites, Gawker uses the Disallow: function to prevent search engines from crawling certain parts of their site that might mislead some search results. |
![]() Disallow: /322813/ # Remove post |
} |
However, one particular Disallow: function of Gawker’s robots.txt stands out from the others: Disallow: /322813/ # Remove postThis Disallow: instructs search engines to ignore http://gawker.com/322813/ while not actually removing the article and its content from Gawker.com. This number, 322813, references a PostID. Most individual blog posts are assigned a unique numerical identifier, or PostID. If you’ve ever visited an individual blog post on Gawker.com, notice the address. It might look similar to: http://gawker.com/322813/title-of-article This address, or Permalink, or URL will usually include a protocol (http://), the domain (gawker.com), the PostID (322813), and the title of the article converted to the URL format (title-of-article), sometimes truncated. The domain and the PostID are the most important identifiers for a Permalink, the title URL, which gives Permalinks an alphabetical identifier, can actually be manipulated by the end-user or removed from the link altogether without affecting its destination. The Permalink title can contain just about any string of alphanumeric characters and dashes, for example: http://gawker.com/322813/ =
|
*** |
![]() http://gawker.com/322813/ |
} |
Still with me? Upon visiting http://gawker.com/322813/ we are met with an article about the son of the editor of Vanity Fair, entitled “The Strokes Really Speak To Graydon Carter’s Emo Kid Spike.” The article goes on to review and re-publish a semi-autobiographical video slideshow narrated by Graydon Carter’s son, Spike. Then Gawker editor, Maggie Shnayerson—whose brother Michael is a contributing editor at Vanity Fair—authors a fairly straightforward post on November 14, 2007, mostly quoting the young Carter directly from his narration, and labeling him an “Emo Kid.” Ms. Shnayerson does not clearly indicate from where or when the video originated. While the article itself isn’t particularly harsh, given the context of Gawker, it kindof is. |
http://gawker.com/322813/#c2995961http://gawker.com/322813/#c2980917 |
} |
The Internet—especially sites like Gawker—is a shooting gallery of negotiable value, depending on which end of the gun barrel you may find yourself. The commenters, sitting in the upper-balcony, are out for blood. While mostly critical of the video and its narrator, few offer support and context. weknowmorethanyouaboutthis, in defense of Spike, is awarded approval from Gawker Media owner, Nick Denton. Another commenter, gauche, offers further context however, this comment will be hidden to anyone visiting the site without a commenter login. |
http://bg-video.cp.motionbox.com/motionboxons/flash/VideoPlayer.swf?video_uid=4c96d5b51e16efc1c3&type=sd&security_token=prod3.6edf3d5dd0b941d5 |
} |
The videoM—it is anticlimactic—is still embeded onto the page and accessible from Motionbox, its host, however, the object appears to be hidden behind a large black jpegM, stored under the Gawker asset directory /2009/11/ which may or may not indicate the year and month of its creation. I take no particular joy in re-publishing a personal video that may have been published by its author, possibly to satisfy an educational requirement, re-published onto a gossip website, un-published by its original author and intentionally concealed inside the gossip website, while not entirely deleted. However, having come this far, I feel doing so serves this investigation. |
*** | ||
|
So, a heavily trafficked New York media blog published a personal video made by the son of an influential magazine editor, quite possibly, against the son’s wishes. The article on which the video appears, has been singled out and hidden from internet search engines while not removed from the site entirely, and the video itself has been covered up by a large black jpeg, although not deleted from its host. Why does any of this matter?
It doesn’t, nor should it. A young man reflecting upon his life over the narration of a video slideshow, a father looking out for his son, journalists playing ball, someone calling in a favor, and a writer—me—with a little too much spare time who stumbled upon something clever. |
||







