Data Collection for Competitive Analysis

Data collection is key. Healthcare now relies on increasingly large amounts of data for maximizing both artificial intelligence (AI) and revenue cycle management (RCM) systems. BHM Healthcare Solutions understands the value of data in getting healthcare organizations to next-gen analytics.

Today’s blog begins at the beginning: collecting data. BHM published this article, written by our Senior Data Analytics Consultant, David Shaffer, as the first tool in your data collection toolbox. While there are many reasons to collect data, David was working on a project with a client to gather data for competitive analysis — enabling them to stay ahead of their competition without hours of manual data searching, entry, cleaning, and review.

As your IT teams amass more and more data, look to BHM as your independent healthcare analytics consulting resource. For more information on our services, contact BHM using our services form or call 888-831-1171. Ask our representative for more information about our business intelligence services.

Before getting to David’s blog, the links below provide some background on today’s topic:

• How to Extract Web Content
• Web Data Extraction: Beginner to Advanced
• Web Data Collection Wiki

Web Data Collection

Web data collection, sometimes called web scraping, is the process by which we obtain information from the HTML (HyperText Markup Language) of a web page. In this article, we’re going to talk about why we might want to do this and walk through the steps involved from a coding perspective.

Why Collect Data from the Web?

Did you know there are around two hundred million active websites on the internet today? That’s two hundred million sources of data we can collect from. You could argue that the quality of the data we might gather from the majority of these is questionable at best — but nonetheless, it is still data.

Companies you’ve probably heard of, like Kayak, deploy bots at scale to capture data from websites and perform price comparisons for you, saving you the time of visiting each company’s page and comparing prices yourself. E-commerce is another popular space where web scraping is used to analyze who is buying what, when, and for how much, in order to gain an advantage over competitors.

In this article, we are going to scrape data from Amazon, with the goal of capturing the prices for office chairs sold by Amazon Basics. We won’t cover the steps involved to store, manipulate, and analyze that data after we scrape it — that will be for another article. However, we will use a bot to open Amazon’s web page using Selenium for Python and navigate to the page we want automatically, without the click of a mouse. Then, we will scrape the page for the prices and cleanse the data.

Using a Bot to Open and Navigate Through a Website

Very simply, Selenium automates browsers. It’s primarily used for testing, but in many cases it serves as a useful tool for gaining efficiency with repetitive processes.

Step 1: Import the webdriver package.

Step 2: Download the Selenium webdriver.

Make sure you choose the right driver for your specific browser and operating system. I am using Chrome on Windows, so I’ve downloaded the Chrome driver for Win32 and placed it in my Documents folder. I’ll then refer to that location when defining my browser variable.

Step 3: Open the browser with the bot.

Use the function get and pass the web page address as the argument.

Step 4: Allow the page to load.

When you open a web page and it isn’t loaded instantly, that’s because it’s taking time for the HTML to load. This is a great time to use the sleep function to allow the page to load. If we don’t do this, Python will move to the next line in the script and try to execute it — and when acting on HTML that hasn’t yet loaded, your program will throw an error.

Step 5: Enter the product you want to search for in the search box.

In this case, we are going to search for “office chair.” This is where Selenium can get tricky. To enter text into the search box, we must first find a way to identify the element from the HTML. You can press F12 to open the browser page’s HTML.

Notice the notification along the top of the page that says, “Chrome is being controlled by automated test software.” In many cases this can be detrimental to our efforts, as many websites — such as Zillow — block automation tools like Selenium. When you try to navigate through their page with a bot, it will take you to a page that requires you to pass a test to prove you are human.

Now that we’ve opened the HTML, the fastest way to get to the element we want is by clicking the arrow-inside-a-box icon along the top of the inspection pane. Then click on the element on the website that you want to access — this will bring you to the element’s HTML.

Identifying an element by its id is always a good place to start, if the id appears to be unique to that element. So I am going to find the element by its id and use the send_keys function to type into the search box.

Step 6: Search!

Identify the element — in this case, the magnifying glass — and click it.

Step 7: Allow the page a few seconds to load.

Step 8: Filter for products sold by Amazon Basics.

In many cases, we are unable to identify an element by its id. In this instance, I am identifying it by its xpath. To learn about all the ways to identify an element, be sure to check out this page and save it to your favorites — I reference it all the time while writing Selenium scripts.

Step 9: Allow the page a few seconds to load.

Step 10: Filter for products with a 4-star rating or higher.

Using the Beautiful Soup Package to Extract Data from the HTML

So far, we’ve used Selenium to access the HTML and navigate through web pages, but we can’t use Selenium to extract the text within the HTML. That’s where Beautiful Soup comes in!

Step 11: Define your Beautiful Soup variable.

Step 12: Extract prices.

This step can be the most difficult of all the steps we’ve covered up to this point. Using the find_all function, we can identify the values inside all the price elements. But we must first identify the tag used to house the prices. After reviewing a few of the prices, I see that they are all tagged in a similar manner, so I write code to extract all of them. I must define the attrs argument to tell my code in more detail which span tags to pull information from, because the prices aren’t the only HTML lines that use a span tag.

Step 13: Manipulate the output.

Using a standard Python list comprehension, I want to end up with just a list of the prices. Then, using the len function and the mean function from the statistics package, we can see the quantity and the average price of Amazon Basics office chairs with a 4-star rating or higher that are currently listed for sale on Amazon.

You can see how something like this, done with many bots on a regular basis, can obtain useful data for analyzing your competition’s prices — all without you having to click a single button!

---

Data and security go hand-in-hand within healthcare. Download our HITRUST Origin Story to learn how security and healthcare came together.

Editor’s Note: BHM Healthcare Solutions understands the value of data in getting healthcare organizations to next-gen analytics. For more information on our services, contact BHM using our services form or call 888-831-1171 and ask our representative for more information about our business intelligence services.