Home / Curated / Algorithms / Flake focused community crawling

Flake focused community crawling

Algorithms AIMech:FlakeFocusedCrawl DRAFT

Aliases: APPROXIMATE-FLOW-COMMUNITY, Flake focused-crawl algorithm

A flow-based focused-crawling family that alternates gathering a local web graph around positive seed pages, extracting a group through seeded maximum flow, promoting selected discovered pages to seeds and recrawling, with version-specific graph augmentation and seed-ranking rules.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 677 native IDs, names, aliases and scoped synonyms before either cohort write; no normalized identity match. Ignored- and hidden-inclusive searches of all three checkouts returned 20,0,0 lines, all read: prior source-deduplication logs and the Flake source lead, not either maintained concept. Semantic search covered 70 AIO and 318 native definitions; 64 AIO and 290 native rows match prior inspected context, and all six newly selected AIO and 28 native definitions were read. Eleven complete AIO node/edge/generated contexts were assessed, five verified unchanged and six newly read. The full EM record and previously inspected GN/Radicchi records provide native identity context. Both drafts were compared for a fixed-graph exact core versus a versioned crawl/reseed family. Retain one focused-crawling family with explicitly distinguished2000 and2002 versions. Both alternate flow-based grouping, seed promotion and recrawling, but their graph augmentation and seed-ranking rules differ. The family is distinct from the one-graph exact core and from likelihood-based expectation maximization; no extra duplicate is created for each historical wrapper.

Claims & Grounded Evidence

7 Claims

Mechanism #acquisition-and-reseeding

The procedure uses an explicitly supplied seed set, gathers outgoing links and searches for incoming links, performs flow-based grouping on the resulting graph, promotes selected nonseed pages and repeats for a prescribed number of rounds. Graph acquisition and seed expansion are part of the algorithm; the wrapper is distinct from solving a single fixed augmented graph.

supports Source: Self-Organization and Identification of Web Communities · Locator: Figure 2b and approximate-flow discussion, printed p. 68 / PDF p. 3
supports Source: Efficient Identification of Web Communities · Locator: Sections 3.2–3.3 and Table 1, printed pp. 153–154 / PDF pp. 4–5

Mechanism #kdd2000-variant

The KDD2000 implementation symmetrizes links within the seed and immediate-neighbor region while preserving directed boundary-reference links toward a virtual sink. It explicitly scales only seed-to-neighbor capacities, deviating from the uniform-scaling construction discussed in its theorem. Table 1 promotes community pages tied to selected high in-degree and out-degree scores measured on the induced graph, then recrawls.

supports Source: Efficient Identification of Web Communities · Locator: Section 3.2, Figure 2 and implementation-deviation paragraphs; Section 3.3 and Table 1, printed pp. 153–154 / PDF pp. 4–5

Mechanism #ieee2002-variant

The 2002 wrapper describes a fixed-depth crawl in both hyperlink directions, sets k to the seed-set size, invokes EXACT-FLOW-COMMUNITY on the induced graph, ranks returned pages by internal community links and adds highest-ranked nonseeds. These choices differ from the earlier boundary wiring and induced-graph in/out-degree rule; the versions are retained explicitly rather than presented as identical pseudocode.

supports Source: Self-Organization and Identification of Web Communities · Locator: Figure 2b and approximate-flow discussion, printed p. 68 / PDF p. 3
supports Source: Efficient Identification of Web Communities · Locator: Table 1 and preceding implementation-deviation paragraphs, printed p. 154 / PDF p. 5

Limitation #em-and-convergence-boundary

The earlier source describes reseeding as inspired by expectation maximization, but supplies no statistical expected-log-likelihood objective or corresponding parameter-estimation E-step. It explicitly lacks a proof of convergence to a fixed point and a bound on deviation from the ideal community procedure. A prescribed round count and empirical stabilization are not such guarantees.

supports Source: Efficient Identification of Web Communities · Locator: Section 3.3, printed p. 154 / PDF p. 5

Limitation #coverage-and-theorem-boundary

Exact flow on a sampled graph does not establish a result for the complete Web. Missing incoming links, crawl depth, boundary treatment, seed selection and heuristic capacity scaling affect the represented problem. The 2000 paper’s Theorem 2 concerns a specified uniformly scaled virtual-sink graph under a stated inequality, while the authors explicitly describe deviations in the implemented crawler; it is not an unconditional guarantee for either wrapper.

supports Source: Efficient Identification of Web Communities · Locator: Section 3.2, Theorem 2 and implementation deviations, printed pp. 153–154 / PDF pp. 4–5
supports Source: Self-Organization and Identification of Web Communities · Locator: Approximate-flow discussion, printed p. 68 / PDF p. 3

Scope #historical-evaluation

The KDD2000 examples concern support-vector-machine research, the Internet Archive and Ronald Rivest, with different seeds, iteration counts and handling of same-domain links. The authors report false positives and say network retrieval dominated runtime. The 2002 examples use three individual-scientist seeds and roughly two hundred returned pages per run; these are conditional historical demonstrations, not comparable modern accuracy or throughput benchmarks.

supports Source: Efficient Identification of Web Communities · Locator: Section 4, Tables 2–4 and runtime discussion, printed pp. 154–158 / PDF pp. 5–9
supports Source: Self-Organization and Identification of Web Communities · Locator: Experimental Results and Figure 3, printed p. 69 / PDF p. 4

Limitation #interpretation-and-ranking

Internal-link counts rank pages rather than estimate calibrated membership probabilities. The 2002 paper’s keyword features and exhaustively selected three-term rules characterize the returned groups after collection; their reported match rates are not a held-out evaluation of a learned predictive model. Link-based retrieval does not establish bias-free or universally correct semantic communities.

supports Source: Self-Organization and Identification of Web Communities · Locator: Experimental Results, Figure 3 and Table 1 discussion, printed pp. 69–70 / PDF pp. 4–5

⚠️ Curated Limitations & Negative Results

This family record preserves the distinct 2000 and 2002 graph and ranking conventions. It does not combine their details into a new purported author implementation.
Exact-flow core limitations remain relevant: preserving seeds and optimizing a cut do not certify every seed’s degree criterion or recover any arbitrarily preselected community. The separate exact-flow record and finite calculations document those boundaries.
The 2002 Figure 2a all-E rescaling ambiguity is inherited by its called core. The prose preexisting-edge convention is the recorded interpretation; author code was not audited.
Tie handling, the precise number of promoted pages in the 2002 description and all exceptional candidate-pool cases were not resolved from implementation evidence. No fixed-point or approximation guarantee is asserted.
Both complete primary papers, displayed algorithms and result tables were inspected. Their historical URLs were not visited and their descriptions were not independently revalidated. No live Web crawl, author-code execution, model training or benchmark replication was performed; DRAFT remains.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:Clustering` Clustering	related_match	The crawler repeatedly groups linked pages around positive examples. The general clustering task is related without asserting that the seeded acquisition algorithm is purely unsupervised, a hierarchical clustering method or a trained predictor.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`flake2002`	Self-Organization and Identification of Web Communities Gary William Flake, Steve Lawrence, C. Lee Giles, Frans M. Coetzee	primary paper full_text	ID: doi:10.1109/2.989932 sha256:104f1461e300…
`flake2000`	Efficient Identification of Web Communities Gary William Flake, Steve Lawrence, C. Lee Giles	primary paper full_text	ID: doi:10.1145/347090.347121 sha256:016ffe67913e…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-14T02:24:51Z

Updated: 2026-10-14T02:24:51Z

Source File: data/curated/FlakeFocusedCrawl.yaml