- Scrapy Tutorial
- Scrapy - Home
- Scrapy Basic Concepts
- Scrapy - Overview
- Scrapy - Environment
- Scrapy - Command Line Tools
- Scrapy - Spiders
- Scrapy - Selectors
- Scrapy - Items
- Scrapy - Item Loaders
- Scrapy - Shell
- Scrapy - Item Pipeline
- Scrapy - Feed exports
- Scrapy - Requests & Responses
- Scrapy - Link Extractors
- Scrapy - Settings
- Scrapy - Exceptions
- Scrapy Live Project
- Scrapy - Create a Project
- Scrapy - Define an Item
- Scrapy - First Spider
- Scrapy - Crawling
- Scrapy - Extracting Items
- Scrapy - Using an Item
- Scrapy - Following Links
- Scrapy - Scraped Data
- Scrapy Built In Services
- Scrapy - Logging
- Scrapy - Stats Collection
- Scrapy - Sending an E-mail
- Scrapy - Telnet Console
- Scrapy - Web Services
- Scrapy Useful Resources
- Scrapy - Quick Guide
- Scrapy - Useful Resources
- Scrapy - Discussion
Scrapy - Telnet Console
Description
Telnet console is a Python shell which runs inside Scrapy process and is used for inspecting and controlling a Scrapy running process.
Access Telnet Console
The telnet console can be accessed using the following command −
telnet localhost 6023
Basically, telnet console is listed in TCP port, which is described in TELNETCONSOLE_PORT settings.
Variables
Some of the default variables given in the following table are used as shortcuts −
Sr.No | Shortcut & Description |
---|---|
1 | crawler This refers to the Scrapy Crawler (scrapy.crawler.Crawler) object. |
2 | engine This refers to Crawler.engine attribute. |
3 | spider This refers to the spider which is active. |
4 | slot This refers to the engine slot. |
5 | extensions This refers to the Extension Manager (Crawler.extensions) attribute. |
6 | stats This refers to the Stats Collector (Crawler.stats) attribute. |
7 | setting This refers to the Scrapy settings object (Crawler.settings) attribute. |
8 | est This refers to print a report of the engine status. |
9 | prefs This refers to the memory for debugging. |
10 | p This refers to a shortcut to the pprint.pprint function. |
11 | hpy This refers to memory debugging. |
Examples
Following are some examples illustrated using Telnet Console.
Pause, Resume and Stop the Scrapy Engine
To pause Scrapy engine, use the following command −
telnet localhost 6023 >>> engine.pause() >>>
To resume Scrapy engine, use the following command −
telnet localhost 6023 >>> engine.unpause() >>>
To stop Scrapy engine, use the following command −
telnet localhost 6023 >>> engine.stop() Connection closed by foreign host.
View Engine Status
Telnet console uses est() method to check the status of Scrapy engine as shown in the following code −
telnet localhost 6023 >>> est() Execution engine status time()-engine.start_time : 8.62972998619 engine.has_capacity() : False len(engine.downloader.active) : 16 engine.scraper.is_idle() : False engine.spider.name : followall engine.spider_is_idle(engine.spider) : False engine.slot.closing : False len(engine.slot.inprogress) : 16 len(engine.slot.scheduler.dqs or []) : 0 len(engine.slot.scheduler.mqs) : 92 len(engine.scraper.slot.queue) : 0 len(engine.scraper.slot.active) : 0 engine.scraper.slot.active_size : 0 engine.scraper.slot.itemproc_size : 0 engine.scraper.slot.needs_backout() : False
Telnet Console Signals
You can use the telnet console signals to add, update, or delete the variables in the telnet local namespace. To perform this action, you need to add the telnet_vars dict in your handler.
scrapy.extensions.telnet.update_telnet_vars(telnet_vars)
Parameters −
telnet_vars (dict)
Where, dict is a dictionary containing telnet variables.
Telnet Settings
The following table shows the settings that control the behavior of Telnet Console −
Sr.No | Settings & Description | Default Value |
---|---|---|
1 | TELNETCONSOLE_PORT This refers to port range for telnet console. If it is set to none, then the port will be dynamically assigned. |
[6023, 6073] |
2 | TELNETCONSOLE_HOST This refers to the interface on which the telnet console should listen. |
'127.0.0.1' |
To Continue Learning Please Login
Login with Google