This Crawler and Scraper will search each file
and clone all files matching your searchterm
using an API from scraping-bot.io
🚀This web-scraping tool will scrawl through the content of an entire domain of your choice. It will extract data without getting blocked. Users can configure the following scrawl-settings before they start scrawling:
- five different CrawlingDepth's
- the Domain / Webpage they want to scrawl
- a searchterm to search for specific content inside of the Domain
All pages containing the searchstring will be considered relevant, and thus copied and saved to the server. PS: The maximum-crawling-depth is limited to a depth of 5 levels.⚡️
First Demo
The first Example will search all across the domain: nevelingreply.de.
It will scrape all pages containing the term "Dienstleister" achross all pages. Matching pages will be cloned to our server.
nevelingreply.de/competence
Crawler and scraper demo with reference-links.
- Crawler Startingpoint
- nevelingreply.de/competence
- CrawlingDepth Limit:
- Level 2
- Scraper-Searchterm
- 2
- IF Searchterm is true
- store all result-pages to server (JSON-File)
- About
- The Crawler will begin at the Startingpoint and crawl as deep as defined per Crawling Depth. The JSON FILE stores all Pages containing the searchterm.
- Reference Links
- start the scraperAttention: The scraper will start to run!
Wait for up to 60 seconds
until the result will show up!!run the scraping-service - downloadClick here for an instant result.Download JSON FILE
https://crawl-and-scrape.vercel.app/api/CrawlingDepth (Param)2/?crawlURL=CrawlURL (Param)https://nevelingreply.de/competence?searchString=searchString (Param)DienstleisterSecond Demo
The second Example searches all across the domain: scraping-bot.io.
It scrapes all pages containing the term "my paragraph of". All pages containing that searchstring-term will be stored to our server.
scraping-bot.io/crawler/second-page.html
Crawler and scraper demo with reference-links.
- Crawler Startingpoint
- scraping-bot.io/crawler/second-page.html
- CrawlingDepth Limit:
- Level 5
- Scraper-Searchterm
- 5
- IF Searchterm is true
- store all result-pages to server (JSON-File)
- About
- The Crawler will begin at the Startingpoint and crawl as deep as defined per Crawling Depth. The JSON FILE stores all Pages containing the searchterm.
- Reference Links
- start the scraperAttention: The scraper will start to run!
Wait for up to 60 seconds
until the result will show up!!run the scraping-service - downloadClick here for an instant result.Download JSON FILE
https://crawl-and-scrape.vercel.app/api/CrawlingDepth (Param)5/?crawlURL=CrawlURL (Param)https://www.scraping-bot.io/crawler/second-page.html?searchString=searchString (Param)my paragraph of the second pageThird Demo
The third Example scrapes across: triplesensereply.de
and find all pages containing the term "Frontend Developer".
All pages containing that searchstring-term will be found and stored to the server.
triplesensereply.de/agentur
Crawler and scraper demo with reference-links.
- Crawler Startingpoint
- triplesensereply.de/agentur
- CrawlingDepth Limit:
- Level 1
- Scraper-Searchterm
- 1
- IF Searchterm is true
- store all result-pages to server (JSON-File)
- About
- The Crawler will begin at the Startingpoint and crawl as deep as defined per Crawling Depth. The JSON FILE stores all Pages containing the searchterm.
- Reference Links
- start the scraperAttention: The scraper will start to run!
Wait for up to 60 seconds
until the result will show up!!run the scraping-service - downloadClick here for an instant result.Download JSON FILE
https://crawl-and-scrape.vercel.app/api/CrawlingDepth (Param)1/?crawlURL=CrawlURL (Param)https://www.triplesensereply.de/agentur/?searchString=searchString (Param)Frontend DeveloperThis service is currently saving the HTML-markup of all matching pages and saves them all in just one json file as strings.
Built with NextJS and Tailwind CSS.
Feel free to contact me In case of any questions.