A competitor price parser is software designed to collect data on the cost of goods or services from the pages of certain sites - online stores, price aggregators, marketplaces. The price parser marketplaces settings allow you to scan data at any frequency required by the customer. Parsing results can be generated into tables in Excel or CSV format or transmitted via API.
Parsing the prices of competitors is the collection of prices for similar goods or services from the pages of websites of online stores or price aggregators. Competitor prices are parsed regularly, according to a schedule. The purpose of such parsing is to obtain up-to-date information about the prices of competitors in order to create your own pricing system and develop a marketing strategy.
Содержание
- 1 Parsing competitor prices and its problems
Price parsing
Parsing of prices is one of the main and effective marketing tools in the e-commerce segment. Let's figure out together what kind of animal it is, and why without it the path to success will be long and thorny
To begin with, let's turn to the definition of the concepts "price parser" and "price parsing"
Price parsing is...
.
A price parser is an IT product (a specially designed computer program) designed to collect data on the cost of goods or services from the pages of sites visited by the parser. This program is able to bypass crawling protection, and has fine settings to collect exactly the information and in the form specified by the customer.
Price parsing is the process of monitoring the prices of competitors or partners, which is performed regularly, according to a schedule. The purpose of price parsing is to obtain up-to-date information to control prices on the market and help the customer optimize their own pricing policy.
Simply put, a parser is software that works under the control of the user, which searches, selects, collects and saves information in a form convenient for viewing and analyzing by the user, and parsing is the automatic collection of the information required by the user from certain Internet resources selected by him.
Most companies actively use scraping, but hide this fact. They treat scraping in the business environment with hostility: "reputable companies do not do such a shameful business." And they themselves quietly buy parsers and keep a whole team of IT specialists to set up and work with this software. The largest players in the Internet sales market, for which, it seems, there are no competitors, regularly monitor the market - parsing rivals and therefore remain in the lead.
If this is acceptable for mastodons, then why are less successful companies embarrassed to use site scraping in their work? It's most likely a misperception. It is perceived not as a gathering of publicly available information useful for business, but as a peek through a keyhole. This phenomenon arises due to the fact that such a tool as parsing, companies began to implement relatively recently, and many still do not fully understand what kind of process it is.
Recently, more and more entrepreneurs have been interested in parsing Internet resources as an effective tool for collecting databases and developing business. Extracting data from competitors' websites will allow you to achieve an advantage in your niche, learn about market trends and consumer demands. First of all, everyone is interested in the prices that are presented on the Internet sites of competitors. To do this, use the price parser.
Often from the pages of competitors' sites companies collect content for their own online resources. For these purposes, content grabbers* are used.
Parsing is not just data collection, but one of the methods of business promotion, since in addition to the prices themselves, the parser can collect promotional offers, the availability of goods from competitors or partners, product reviews and any other information, depending on the software settings. Parsing prices from sites is most effective when the assortment is stable, and Internet sites do not use aggressive methods of protection against crawling.
In other words,
automatic price collection (price parsing) is the optimal software solution for collecting marketing information about the prices of competitors and partners on the Internet. Automatic monitoring of competitors' prices is more expedient than manual collection, since it saves all the company's resources.
For parsing prices from sites, specially developed software is used - price parsers written in various programming languages. In automatic mode, programs bypass websites to collect information about the prices of competitors and partners. The efficiency of the parsers is not comparable to the department of workers involved in monitoring prices manually.
An automated competitor price monitoring program is more beneficial for an online reseller, as it allows you to compare the prices of online stores selling the same products as your company in real time. How your price is more (or less) attractive to customers. In addition, parsing prices in online stores will allow suppliers to control the MAP/ MSRP and promptly respond to dumping.
Price parser in online stores - automated software that monitors prices on online trading platforms in order to control manufacturers and suppliers compliance with recommended retail prices.
Parsing the prices of online stores is monitoring the prices of partners for manufacturers and suppliers in order to compare them with the MAP/ MSRP. Monitoring results are generated into reports in Excel or CSV formats. Deviation from the recommended retail price up or down is usually highlighted in different colors, which greatly simplifies the process of analyzing the report.
What is parsing for?
Business development tasks for which parsing can be used.
- Competitive analysis. Manually browsing dozens or hundreds of competitor sites is quite expensive, it is much easier to run automated scraping.
- Own pricing. By parsing prices from competitors' websites, you can track changes in the cost of goods and form your own pricing.
- Audit of your own site. You can parse not only other people's sites, but also your own to clean up deleted pages, incorrect information and any other "garbage".
- Copying content. If you need to fill out the product cards of your own online store, you can speed up the process by parsing information from other sites. Of course, all content must be adapted to your own site so as not to get accusations of plagiarism.
- Lead generation. With the help of site parsing, you can collect and replenish the customer base.
- Collection of databases of installers. When it becomes necessary to find specialists to install equipment in different regions, you can parse advertisements of installers or installers of these regions. Bulletin boards, search results, search engine ad blocks are well suited for parsing.
Parsing functions can be useful for every business.
Automatic price parsing
Automatic price parsing is the collection of prices from sites selected by the user using software that works in automatic mode. The purpose of this procedure is to control prices on the websites of customers and competitors in order to maintain the optimal own price.
To automate the work of marketers in collecting prices from partners and competitors, various types of parsers are used - computer programs written specifically for this purpose.
Parsers in a matter of minutes can bypass hundreds of sites and collect any information of interest to you from their pages. They require a highly technical staff to set up, but this tool is a huge time saver for regular data collection.
Now there is a process of rapid reproduction of trading platforms on the Internet. One well-tuned price parser can replace the work of several employees engaged in manual price monitoring.
There are several ways to automate using parsing.
• Buy a ready-made parser.
• Write a parser for your project.
• Pay for the services of cloud services for parsing prices.
Each of these methods has its pros and cons. I will try to evaluate each of them.
Market price parser - buy ready
There are situations when it was necessary to parse the prices of competitors “for yesterday”. The best solution in this case would be to buy a price parser and customize it for your tasks.
https://parserok.ru/
The site parser is designed to collect prices and then display them in an Excel spreadsheet. It was created on VBA (Visual Basic for Applications) and presented as an add-in for MS Excel. To parse prices, you need to write a special macro management subroutine.
The site scraper can be used for the following:
1. Parse prices and other information from websites.
2. Output data to a table for later export.
3. Implement an algorithm for transferring data to e-mail or Telegram.
The “pluses” of the parser include the ability to set up a schedule, the ability to parse several data sources at the same time. It is a one-time purchase, but upgrades are available for free. The "cons" include the need for self-configuration of macros, the output of a large array of data only in a tabular format.
https://excelvba.ru/
This parser also collects information using Excel macros. On the site, you can select and download a ready-made site parser, or order a parser setup to collect data from your source.
The advantages of this solution include a one-time payment for the parser, the ability to process a large amount of information, the ability to collect and display additional data (promotions, availability, etc.), the ability to collect only selected information (you can update prices for one store, for all at once, or selected ones only). The disadvantages include the need to independently search for links for parsing, paid setup of each subsequent site for data collection, the only format for storing data is an Excel file.
hotline parser or universal parser - write it yourself
If "time suffers" you can order writing your own parser. This can be done by the company's own specialists, or you can resort to the help of freelancers or enter into an agreement with a company that specializes in writing custom parsers.
If you opted for freelancers, I recommend using the services of freelance exchanges. Their big advantage is the security of transactions and available information about the past achievements of the specialist you have chosen.
https://freelancehunt.com/project/napisat-parser-dlya-polucheniya-dannyih/468443.html
If you prefer working with companies, below are examples of some of them with extensive experience writing scrapers.
https://iparser.ru/
https://catalogloader.com/documentation/eprice
Involving freelancers or third-party companies in writing a parser has one, but a very significant minus. The parser requires regular reconfiguration, since online store sites are dynamic and can make changes to the layout. Practice shows that after the delivery of the order, freelancers evaporate and do not want to bother with its maintenance. In this case, relevant data will be collected by such a parser for no more than 2-3 weeks.
Cloud services for parsing prices or SaaS service
Another option for collecting prices online is to purchase licenses to use cloud-based price parsers.
Price parsing using a SaaS service is a service of a company that provides price monitoring as a ready-made software solution for collecting data from sites of any complexity.
Such services involve flexible settings for data collection, comparison - matching of goods or services from competitors' or partners' websites, generation of reporting and analytical data. Companies that provide this service have extensive experience in collecting any data from Internet sites. They can also collect prices for competitors' products for you. Below I will give some examples of such companies.
Datacol-powered online store price parser can be tested before purchase. It can be customized directly to your needs. After setting up, you can check the collected data for uniqueness and export it to various CMS formats.
The disadvantages include the rather high complexity of setting up and the cost of setting up parsing if this is done by a Datacol specialist.
The advantages of this price parser include high performance, low requirements for computer resources, the ability to set a parsing schedule for several months in advance. It is possible to write a parser for your needs.
The disadvantages include the rather high cost of acquiring a parser and the need to pay extra for updates separately.
Parsing prices from sites on your own - 9 obstacles
Every seller on the Internet wants to have a price for their goods that will make them attractive in the eyes of potential buyers. I have already noted above that most often, this is the lowest price on the market. It is important for online stores to keep the price low, but make it inaccessible to competitors for automatic parsing. After all, as soon as they see these prices, they will immediately reprice their own goods. That is why most online sales companies use some form of protection against automatic parsing of their prices. They are trying to build comprehensive protection that a competitor's marketer cannot bypass without good special training. Here are some of the possible ways to protect.
1. Ban on parsing.
Imagine a situation. You have selected the target sites from which you plan to collect information. But when running the collection program, it turned out that sites forbid parsing through their robots.txt. In this case, you need to get permission from site owners to parse. If permission is denied, it is better to choose other, similar Internet resources. However, it is worth remembering that data posted in the public domain rarely falls under the protection of copyright or related rights. Their privacy can also be questioned. It makes sense to consider whether the ban is legitimate.
2. Updates of site page design.
If the site is created using HTML (HyperText Markup Language - Hypertext Markup Language), the web designer can model the pages according to his own vision, which leads to differences in the structure of sites. Therefore, to collect information from resources with different structures, you will have to create several parsers. Updating the design or adding new features to the site will also require reconfiguring the parser.
If the parser is configured for a certain structure, then after updating it, it will not be able to collect information from the site page. The situation will be even worse if the price parser continues to collect data, but pulls prices from other fields. For example, instead of the current price, it will pull up the old crossed-out price or the price when applying for a loan…
Parsing of prices will provide information, but the conclusions you draw from it will turn out to be false. You may lose profit.
3. Blocking IP addresses.
This is one of the most common methods of protection against parsing. The site turns on IP address blocking when it detects a large number of requests from the same IP. In this case, access to the site may be completely blocked or restricted, which will not allow you to collect all the necessary information. This problem can be overcome with the help of IP proxy services that integrate with automated parsers. The situation will be even more “interesting” if, instead of blocking your IP address, the platform includes issuing deliberately false information to you.
For example, you want to parse the current prices of a competitor, and he issues the recommended retail prices (MAP/ MSRP) of the supplier to your IP address. Price parsing is done, and repricing (repricing your products based on competitor prices) will not give you the expected increase in sales.
4. Access after entering captcha.
Everyone is familiar with the situation when, before entering the desired resource, it is proposed to enter an illegible word or numbers in the field, select one type of image, solve a logical problem, etc. to confirm that you are not a robot. In other words, enter the captcha. CAPTCHA is a fully automated public Turing test to distinguish between computers and humans. People solve test tasks easily, but parsers do not.There are many technologies for bypassing captcha, but the parsing process can slow down. We described in detail all the methods of dealing with automatic price parsing by online sellers in our article “Methods for Monitoring Prices on the Internet”.
5. Trap for the bot (Honeypot trap). Sometimes site owners install special software to obtain information about intruders, the so-called honeypot traps (collectors of information about intruders, which is then used to fight them). As traps, there can be links that a person does not see, but the parser reads. When the parser is trapped, the site receives information, such as an IP address, and can block the attacker. It can also significantly slow down the delivery of its content to the detected bot. It can mix in the result of price parsing a price prepared for such a case (price with MAP/ MSRP or in another currency).
6. Slow site loading speed. Too many access requests to a site can slow down its loading speed. If a person is browsing the site and the download speed has slowed down, it is enough to refresh the page. But the parser in such cases does not know what to do and the parsing process stops. This can lead to a failure to re-evaluate your price list or its complete blocking. In online trading, there are several peaks in sales during the day. At this moment, the maximum number of potential buyers are looking on the Internet for the best offer for the product they are interested in. If your parsing of competitors' prices is late by this moment, repricing of your products is not done, your price does not correspond to the market, customers will go shopping to other resources.
7. Interactive content. Many sites have dynamic content that is developed and embedded using AJAX technologies. It is also called smart content, as it adapts exclusively to the interests and behavioral factors of users. The dynamic content built into the site affects the speed of loading images and scrolling pages, so parsing such resources requires additional settings. To solve such a problem, the site parser must have logic that allows it to imitate the behavior of real users of the Internet site. The developer of such a grabber must be highly qualified, and the marketer who configures it must be familiar with a large number of features of such parsing.
8. Authorization on the site. On some online resources, before you receive information, you will be asked to enter your login credentials - register. After authorization, your web browser creates a cookie value and adds it to your requests on other sites. Thus, the user is identified and gets access to information and services on various resources.
To parse sites that require authorization, you must send cookies along with requests. This is not easy, it requires a high-quality parser and careful and lengthy settings, but it is simply necessary for some parsing tasks. For example, you need to parse prices from a private section of your supplier's portal, or you want to find out the current discounts and promotions of your competitors.
9. Parsing in real time. Monitoring prices and stocks of competitors involves real-time parsing. Frequent data changes can lead to huge profits for some and, conversely, to losses for others. The parser must constantly collect and update data from competitor sites. But the request and delivery of data occurs over a period of time. Parsing a large amount of information in real time can also become a problem.
It is not difficult even for the simplest parser to collect the prices of several hundred products from a dozen online stores. Problems begin when you need to parse the prices of many thousands of items from hundreds of competitor sites. To collect such data arrays, prices are parsed into dozens of streams simultaneously. To receive, store and process these arrays, specialized databases with high speed and high-quality interface are needed.
I am sure that there is no such protection that cannot be bypassed with the experience and skill of parsing. My task was to show what difficulties marketers will have to face when they parse competitors' prices on their own.
Specialized online price monitoring services
There are many monitoring services available online. To use them, the nomenclature must be presented in XML or CSV format and certain parameters must be set (periodicity and regionality). You will receive a report in the form of a table, in which it is convenient to track any price fluctuations and analyze data. Such services help to obtain information about competitors, do not waste the potential of employees on hard routine work, update data at any time and quickly respond to price fluctuations in the market.
Below I will give examples of several services that do automatic online price monitoring.
Price monitoring service "Price Controls"
The "Price Control" project allows you to find all online platforms that sell the product you are interested in, which is important for protecting your brand on the Internet. It can collect prices from resources of any complexity - online stores, marketplaces, price aggregators. In addition to prices, you can collect data about promotions, discounts, delivery, product availability... It is possible to set up the distribution of information to your partners and customers.Convenient private office.
Convenient private office.
The advantages of the service include the possibility of parsing prices both on the links already collected by customers, and the provision of this service by the service. Price Control can perform a comparison - matching - of your products with an assortment of online sites.
What other price monitoring services exist and what are their differences, you can read in our article “Top 5 effective price monitoring services for brands and online retailers.”
In addition, one of the advantages of this service is the monthly updating of all data collected from online stores for corporate clients. This allows you to find new products that stores have introduced to their pages this month and collect prices for them.
This service provides the Test Purchase in online stores and marketplaces. With the help of a control purchase, it turns out to find out the channel for the receipt of goods on the Internet. This can be very useful for identifying unscrupulous suppliers in the online sales channel.
If you buy monitoring services from the Price Control service, then the Control Purchase service will be provided to you at a significant discount.
The advantages of this service include the ability to automatically compare customer products with products on websites, bypass captcha, the availability of an API, and the availability of notifications when prices change. The disadvantages include the need to pay separately for setting up the parsing of each store, which, with a large number of competitors, seriously increases the price of parsing.
The advantages of the service include the presence parsing prices of online stores, comparing them with the MAP and sending letters to violators. The history of price changes is stored in your personal account. Z-PRICE collects data about promotions, discounts and product availability. You can set up a price monitoring schedule, you can transfer monitoring data from the service to the client's accounting system. The disadvantages of the service include an incomprehensible price for monitoring services. The price is set for monitoring 200 online sites, and 150 sites are displayed in reports. Some customers complain about the poor quality of matching their products with the products of online stores.
The advantages of the Competera service include the possibility of pricing for offline and online sellers. They not only monitor prices marketplaces and online stores, but also help customers make pricing decisions.
The disadvantages are the lack of ready-made solutions for clients in Competera. They customize their platform for each client individually. This is quite long, complicated, requires the collection of a large amount of data from customers and is expensive. Their price tag for the client starts from $1,000 US.
How Competitor Price parser Helps Your Business
Timely and high-quality information about the prices of competitors can seriously increase the competitiveness of any business. Regular collection of prices online makes it possible to timely reprice your goods offered for sale online. Analysis of monitoring prices over long periods of time will allow you to determine their pricing strategies and timely adjust your own. Monitoring the availability of goods on the pages of online stores will allow you to identify limitations in their assortment and earn more on your balances.
The Price Control service is ready to help you in carrying out both price monitoring and copyright protection for unique content. Use the services of professionals and do not waste your time and money on solving scraping problems. All the obstacles associated with parsing, our specialists have learned to bypass, collect high-quality data and to the extent that the client needs to solve the tasks.
Alexander Glygalo
Senior software engineer of the Price Control project. Higher education. Studied programming at CNU. Experience in IT - 8 years.
- info@pricecontrol.biz
- 10:00 — 19:00 ( Mon-Fri) — Оffice
- 10:00 — 21:00 (Mon-Sun) — technical support
- +38 093 234 55 03
We are in social networks
For clients
About