Perl and LWP

Sean M. Burke9780596001780, 0-596-00178-9

With the LWP (Library for WWW in Perl) suite of modules, your programs can download and extract information from the Web. Written by a contributor to LWP, and with a foreword by one of LWP’s creators, Perl & LWP is the authoritative guide to using this powerful and popular toolkit.You’ll learn how to navigate the Web with LWP, fetching web pages, submitting forms, managing cookies, and accessing authenticated web sites. The book also demonstrates using three different techniques, regular expressions, tokens, and tress, to extract information from HTML. The many detailed case studies show you how to apply this knowledge to real world situations, from downloading news stories to e-commerce. Perl & LWP is a must have for Perl programmers who want to automate and mine the Web.

---

Table of contents :
Local Disk……Page 0
Perl & LWP……Page 1
JObjects QuestAgent – “Search by Field” Applet……Page 2
Copyright (Perl & LWP)……Page 3
Preface (Perl & LWP)……Page 4
Perl & LWP: Index……Page 5
Preface (Perl & LWP)……Page 6
The Gory Details (Perl & LWP)……Page 7
Colophon (Perl & LWP)……Page 8
Index: Symbols & Numbers……Page 9
Index: A……Page 10
Index: B……Page 11
Index: C……Page 12
Index: D……Page 14
Index: E……Page 15
Index: F……Page 16
Index: G……Page 18
Index: H……Page 19
Index: I……Page 21
Index: J……Page 22
Index: K……Page 23
Index: L……Page 24
Index: M……Page 26
Index: N……Page 28
Index: O……Page 29
Index: P……Page 30
Index: Q……Page 32
Index: R……Page 33
Index: S……Page 35
Index: T……Page 37
Index: U……Page 39
Index: V……Page 41
Index: W……Page 42
Index: X……Page 43
Index: Y……Page 44
Index: Z……Page 45
Structure of This Book (Perl & LWP)……Page 46
User’s View of Object-Oriented Modules (Perl & LWP)……Page 48
So Why Do Some Modules Use Objects? (Perl & LWP)……Page 49
POST Example: ABEBooks.com (Perl & LWP)……Page 50
Converting Relative URLs to Absolute (Perl & LWP)……Page 57
Converting Absolute URLs to Relative (Perl & LWP)……Page 58
Installing LWP (Perl & LWP)……Page 60
User Agents (Perl & LWP)……Page 65
Example: AltaVista (Perl & LWP)……Page 74
Automating Form Analysis (Perl & LWP)……Page 77
Rewrite for Features (Perl & LWP)……Page 79
Modifying HTML with Trees (Perl & LWP)……Page 85
URLs (Perl & LWP)……Page 89
Processing (Perl & LWP)……Page 95
LWP in Action (Perl & LWP)……Page 100
Authentication (Perl & LWP)……Page 104
An HTTP Authentication Example:The Unicode Mailing Archive (Perl & LWP)……Page 107
Example: Babelfish (Perl & LWP)……Page 110
Token Sequences (Perl & LWP)……Page 113
Example: BBC News (Perl & LWP)……Page 118
Example: Extracting Linksfrom a Bookmark File (Perl & LWP)……Page 122
Introduction to Web Automation (Perl & LWP)……Page 125
Idiosyncrasies of HTML Forms (Perl & LWP)……Page 128
Individual Tokens (Perl & LWP)……Page 134
HTML Processing with Trees (Perl & LWP)……Page 136
Cookies, Authentication, and Advanced Requests (Perl & LWP)……Page 138
The LWP Class Model (Perl & LWP)……Page 142
HTML::TreeBuilder (Perl & LWP)……Page 144
Detaching and Reattaching (Perl & LWP)……Page 148
Basic HTML::TokeParser Use (Perl & LWP)……Page 151
Attaching in Another Tree (Perl & LWP)……Page 155
Regular Expression Techniques (Perl & LWP)……Page 161
Example: A Link-Checking Spider (Perl & LWP)……Page 165
Creating New Elements (Perl & LWP)……Page 179
HTTP::Response Objects (Perl & LWP)……Page 183
Words of Caution (Perl & LWP)……Page 188
Simple HTML Processing with Regular Expressions (Perl & LWP)……Page 190
Troubleshooting (Perl & LWP)……Page 193
Tokenizing Walkthrough (Perl & LWP)……Page 195
Example: Extracting Linksfrom Arbitrary HTML (Perl & LWP)……Page 196
Fetching Documents Without LWP::Simple (Perl & LWP)……Page 199
Inside the do_GET and do_POST Functions (Perl & LWP)……Page 201
HTML Processing with Tokens (Perl & LWP)……Page 203
More HTML::TokeParser Methods (Perl & LWP)……Page 205
Using Extracted Text (Perl & LWP)……Page 210
File Uploads (Perl & LWP)……Page 212
LWP and GET Requests (Perl & LWP)……Page 216
Forms (Perl & LWP)……Page 219
POST Example: License Plates (Perl & LWP)……Page 221
Limits on Forms (Perl & LWP)……Page 225
Relative URLs (Perl & LWP)……Page 226
Example: Fresh Air (Perl & LWP)……Page 228
Web Basics (Perl & LWP)……Page 231
LWP::Simple (Perl & LWP)……Page 233
Narrowing In (Perl & LWP)……Page 237
Adding Extra Request Header Lines (Perl & LWP)……Page 239
An HTTP Transaction (Perl & LWP)……Page 243
Deleting Images (Perl & LWP)……Page 246
When Regular Expressions Aren’t Enough (Perl & LWP)……Page 248
HTTP POST (Perl & LWP)……Page 249
History of LWP (Perl & LWP)……Page 251
Example: Extracting Temperatures from Weather Underground (Perl & LWP)……Page 253
A User Agent for Robots (Perl & LWP)……Page 255
Programming with LWP Classes (Perl & LWP)……Page 257
Spiders (Perl & LWP)……Page 259
Order of Chapters (Perl & LWP)……Page 261
LWP Modules (Perl & LWP)……Page 262
HTTP Status Codes (Perl & LWP)……Page 268
Common MIME Types (Perl & LWP)……Page 269
Language Tags (Perl & LWP)……Page 272
Common Content Encodings (Perl & LWP)……Page 275
ASCII Table (Perl & LWP)……Page 278
Modules and Their Functional Interfaces (Perl & LWP)……Page 300
Modules with Object-Oriented Interfaces (Perl & LWP)……Page 302
What Can You Do with Objects? (Perl & LWP)……Page 304
What’s in an Object? (Perl & LWP)……Page 306
What Is an Object Value? (Perl & LWP)……Page 308
Important Standards Documents (Perl & LWP)……Page 312
Acknowledgments (Perl & LWP)……Page 313
LWP Classes: Behind the Scenes (Perl & LWP)……Page 314
Getting the Data (Perl & LWP)……Page 315
Inspecting the HTML (Perl & LWP)……Page 317
First Code (Perl & LWP)……Page 319
Alternatives (Perl & LWP)……Page 321
Ideas for Further Expansion (Perl & LWP)……Page 322
200s: Successful (Perl & LWP)……Page 323
300s: Redirection (Perl & LWP)……Page 324
400s: Client Errors (Perl & LWP)……Page 325
500s: Server Errors (Perl & LWP)……Page 326
Comments & Questions (Perl & LWP)……Page 327
Conventions Used in This Book (Perl & LWP)……Page 328
Perl & LWP……Page 329
http://www.oreilly.com/catalog/perllwp/errata/perllwp.confirmed……Page 330
http://www.oreilly.com/catalog/perllwp/errata/perllwp.unconfirmed……Page 341