AIMech AI
Home / Curated / Algorithms / Chakrabarti classifier-guided focused crawling

Chakrabarti classifier-guided focused crawling

Algorithms AIMech:ChakrabartiFocusedCrawler DRAFT
Aliases: Chakrabarti focused crawler, Taxonomy-guided focused crawling
A focused web-acquisition procedure that uses a trained classifier over a supplied topic taxonomy to decide or prioritize hyperlink expansion, with hard and soft focus modes and periodic relevance-weighted hub discovery to promote promising crawl locations.
Identity Notes & Concept Scope Compared all 442 pinned AIO and 679 native IDs, names, aliases and scoped synonyms before either cohort write; no normalized identity match. Ignored- and hidden-inclusive searches of three checkouts returned 26,0,0 lines, all read: prior deduplication logs and the source lead, not these maintained identities. Semantic search covered 127 AIO and 439 native definitions: 70 AIO and 318 native rows match previously inspected unchanged context; all 57 newly selected AIO and 121 new native definitions were read. Eleven complete AIO nodes, incident edges and generated records were assessed, with full native NaiveBayes, BestFirstSearch, DINOSelfDistillation and FlakeFocusedCrawl context. The classifier-guided acquisition family and relevance-weighted ranking component were compared directly. The concept is the 1999 classifier-guided acquisition family with hard and soft focus modes and periodic distiller integration. A human taxonomy is an input, not a learned hierarchy. It differs from Flake flow grouping, general best-first search and the naive Bayes component alone.

Claims & Grounded Evidence

6 Claims
Mechanism #taxonomy-and-classifier
The user marks good categories in an existing topic tree, with no good category an ancestor of another. The intended classifier model assigns documents to exclusive leaf topics and represents text as a multinomial bag of words with class priors. Posterior mass for the selected categories supplies relevance; the taxonomy is an input rather than a hierarchy learned by the crawling procedure.
Mechanism #focus-policies
Hard focus finds the maximum-posterior leaf and permits expansion only if a selected good category is its ancestor. Soft focus sums the posterior mass of the selected categories, assigns each unvisited neighbor the parent relevance, uses the maximum over multiple referring parents and updates the score after retrieval. Soft focus does not discard pages a priori. A document can have substantial total good-category mass while its most probable leaf lies elsewhere; these policies are not interchangeable threshold tests.
Mechanism #frontier-and-distiller
The soft crawl frontier sorts first by increasing fetch-attempt count and then by decreasing relevance; hard focus orders surviving URLs by attempt count with arbitrary remaining ties. Periodically the crawler stops for relevance-weighted hub/authority analysis, promotes top hubs for revisiting and prepares unvisited pages cited by those hubs. The cited relational-database/concurrent redesign is described as ongoing work, not assumed to be this implementation.
Limitation #probability-equation-boundary
Equation 1 includes the class prior in its numerator but omits prior factors from the sibling-likelihood denominator, despite the surrounding intended Bayes model. Literal values therefore need not normalize: priors 1/4 and 3/4 with equal likelihoods 1/2 give 1/8 and 3/8, summing to 1/2. The record preserves the intended probabilistic model and this printed discrepancy; it does not silently repair or diagnose an uninspected author implementation.
Limitation #evaluation-boundary
The historical comparison of unfocused, hard and soft crawling uses the same classifier for crawl guidance and acquired-page relevance scoring. The authors frame this as evaluation of the neighborhood-based crawling heuristic. It is not independently human-labeled relevance accuracy or classifier calibration. Plotted progression uses fetched-page counts, and overlap from disjoint seed subsets is indirect evidence of coverage, not exhaustive recall or general invariance to initialization.
Limitation #failure-and-scope
Text-poor pages can be misclassified, hard pruning can lose useful paths, and soft focus can diffuse into other topics. The mutual-funds example motivates broadening selected categories; automatic detection and adaptation are future work. The stated average-relevance goal is not accompanied by a global optimum guarantee. Neither historical topical labels nor link-based examples establish medical conclusions or causal claims about the represented communities.

⚠️ Curated Limitations & Negative Results

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes
AIO Target Class Mapping Relation Mapping Rationale
AIO:Classification
Classification
 related_match Trained topic classification directly determines link expansion and priority. The acquisition algorithm is related to this classification task without being identical to the classifier, a learned taxonomy or an asserted AIO subclass.

Evidence Sources & Primary Literature

1 Source
Source ID Publication / Document Type / Access Reference & Checksum
chakrabarti1999
Soumen Chakrabarti, Martin van den Berg, Byron Dom
primary paper
full_text
ID: doi:10.1016/S1389-1286(99)00052-3
sha256:bd6570f71558…

Curation Provenance

Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-14T02:43:28Z
Updated: 2026-10-14T02:43:28Z
Source File: data/curated/ChakrabartiFocusedCrawler.yaml