# Web Scraping and Mapping ## Creating a Map of Major Ports in China using Python and Mapbox's Map *Sedar Sahin, May 2023* ### Introduction In this tutorial, we will learn how to create an interactive map showcasing the major ports in China. Port locations become especially important if you ship goods to or from China. Therefore, for those you who (will) work as a Data Scientist (or Analyst) in a logictics, trading, or manufacturing company, it is good to know where the these ports are. These skills you will learn in this practice will help you visualize not only (and taking the step the further and apply geospatial analyses) the locations of ports, but any kind of establishment with latitude and longitude info for stakeholders. We will use Python's libraries like Requests, BeautifulSoup, Pandas, and Folium to gather the data, manipulate it, and visualize it on the map. Initially, we'll encounter a small challenge with the map's language, but we'll overcome it by utilizing Mapbox's English base map. Prerequisites: * Basic knowledge of Python * Familiarity with web scraping using BeautifulSoup * Understanding of data manipulation with Pandas * Basic awareness of Folium library for map visualization ### Step 1: Installing Required Libraries First things first, ensure you have Python installed on your system. Then, install the necessary libraries using pip if you haven't already: ```bash pip install requests beautifulsoup4 pandas folium ``` ### Step 2: Web Scraping to Get Major Ports Data Once we have the necessary pakages install, we can now start working on our favorite Python interface, any IDE of your choice or a Jupyter Notebook. Because we don't have access to such data with port names along with their latitude and longitude, we need to create it. We will fetch the list of major ports in China from the [Wikipedia page](https://en.wikipedia.org/wiki/List_of_ports_in_China) (see below) using Python's **requests** and **BeautifulSoup** libraries. To do this, we'll write a Python script to scrape the data.

Lisf of Ports in China (Wikipedia: https://en.wikipedia.org/wiki/List_of_ports_in_China)

Here is the list of Ports in the tabular form:

NR	Port
1	Dalian
2	Yingkou
3	Jinzhou
4	Qinhuangdao
5	Tianjin
6	Yantai
7	Weihai
8	Qingdao
9	Rizhao
10	Lianyungang
11	Nantong
12	Zhenjiang
13	Jiangyin
14	Nanjing
15	Shanghai
16	Ningbo
17	Zhoushan
18	Jiujiang
19	Taizhou (North of Wenzhou)
20	Wenzhou
21	Taizhou (South of Wenzhou)
22	Changle
23	Quanzhou
24	Xiamen
25	Shantou
26	Jieyang
27	Guangzhou
28	Zhuhai
29	Shenzhen
30	Zhanjiang
31	Beihai
32	Fangchenggang
33	Haikou
34	Basuo

To retrieve coordinates we need to click on each one of the ports on the page. The below screenshot shows the details about the *Dalian Port,* and its coordinates in two different places on the page both of which highlighted with red rectangles:

Port of Dalian (Wikipedia: https://en.wikipedia.org/wiki/Dalian)

{% hint style="info" %} The way Wikipedia structured its url is: "[https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/Dalian)" + **\** Using this information we wil be creating urls to access to each port's page in order to extract their coordinates. We can use apps like Excel, Numbers, Google Sheets or simply a text editor to achieve this. {% endhint %} The following table shows the ports with their corresponding Wikipedia pages:

NR	Port	URL Wiki
1	Dalian	https://en.wikipedia.org/wiki/Dalian
2	Yingkou	https://en.wikipedia.org/wiki/Yingkou
3	Jinzhou	https://en.wikipedia.org/wiki/Jinzhou
4	Qinhuangdao	https://en.wikipedia.org/wiki/Qinhuangdao
5	Tianjin	https://en.wikipedia.org/wiki/Tianjin
6	Yantai	https://en.wikipedia.org/wiki/Yantai
7	Weihai	https://en.wikipedia.org/wiki/Weihai
8	Qingdao	https://en.wikipedia.org/wiki/Qingdao
9	Rizhao	https://en.wikipedia.org/wiki/Rizhao
10	Lianyungang	https://en.wikipedia.org/wiki/Lianyungang
11	Nantong	https://en.wikipedia.org/wiki/Nantong
12	Zhenjiang	https://en.wikipedia.org/wiki/Zhenjiang
13	Jiangyin	https://en.wikipedia.org/wiki/Jiangyin
14	Nanjing	https://en.wikipedia.org/wiki/Nanjing
15	Shanghai	https://en.wikipedia.org/wiki/Shanghai
16	Ningbo	https://en.wikipedia.org/wiki/Ningbo
17	Zhoushan	https://en.wikipedia.org/wiki/Zhoushan
18	Jiujiang	https://en.wikipedia.org/wiki/Jiujiang
19	Taizhou,_Zhejiang	https://en.wikipedia.org/wiki/Taizhou,_Zhejiang
20	Wenzhou	https://en.wikipedia.org/wiki/Wenzhou
21	Taizhou (South of Wenzhou)	NA
22	Changle	https://en.wikipedia.org/wiki/Changle
23	Quanzhou	https://en.wikipedia.org/wiki/Quanzhou
24	Xiamen	https://en.wikipedia.org/wiki/Xiamen
25	Shantou	https://en.wikipedia.org/wiki/Shantou
26	Jieyang	https://en.wikipedia.org/wiki/Jieyang
27	Guangzhou	https://en.wikipedia.org/wiki/Guangzhou
28	Zhuhai	https://en.wikipedia.org/wiki/Zhuhai
29	Shenzhen	https://en.wikipedia.org/wiki/Shenzhen
30	Zhanjiang	https://en.wikipedia.org/wiki/Zhanjiang
31	Beihai	https://en.wikipedia.org/wiki/Beihai
32	Fangchenggang	https://en.wikipedia.org/wiki/Fangchenggang
33	Haikou	https://en.wikipedia.org/wiki/Haikou
34	Basuo	https://en.wikipedia.org/wiki/Basuo

Please note that the Port #19 was replaced by "Taizhou,\_Zhejiang" in order to retrieve its coordinates and #21 was skipped because there is no geographical coordinate data available on Wikipedia for this port with this name. {% hint style="info" %} Keep in mind extracting URLs can be automated via scraping as well. For the sake of this tutorial, I wanted to use scraping only for one job, which is to extracting coordinates of the ports. {% endhint %} We have links of the ports, it is time to retrieve their coordinates. To do that we first "inspect" the HTML (no need to worry about it for this tutorial) to see via which element/attribute the latitude and longitute information is being stored.

The image below shows that latitude and longitudes are stored in HTML attribute (i.e. class) latitude and longitude respectively.

HTML attibutes for latitude and longitude information

Now we have all the information, including the names of the ports, their urls, and know how to access to their coordinates, we can now roll up our sleeves and work on our Python script to combine them. ```python # Import Libraries # Web Scraping import requests from bs4 import BeautifulSoup # Data Manipulation import pandas as pd # Regular Expression Operations (to be used in degree conversion) import re # Maps import folium ``` Since we will be scraping multiple (33 to be exact) pages it is good to create a function to do the heavy work: ```python def get_latlon_from_wikipedia(url): # access to the url response = requests.get(url) if response.status_code != 200: raise Exception("Failed to fetch the page.") # parse the page soup = BeautifulSoup(response.text, 'html.parser') # retrieve latitude and longitude info latitude_span = soup.find('span', {'class': 'latitude'}) longitude_span = soup.find('span', {'class': 'longitude'}) if not latitude_span: raise Exception("Latitude data not found on the page.") if not longitude_span: raise Exception("Longitude data not found on the page.") # assign the coordinates to latitude and longitude variables latitude = latitude_span.text.strip() longitude = longitude_span.text.strip() # store the latitude and longitude info in a list latlon = [latitude, longitude] return latlon ``` Next, we need to define a list which will store all ports' urls that we have extracted earlier. This is simply a copy and paste step. ```python url_list=[ 'https://en.wikipedia.org/wiki/Dalian', 'https://en.wikipedia.org/wiki/Yingkou', 'https://en.wikipedia.org/wiki/Jinzhou', 'https://en.wikipedia.org/wiki/Qinhuangdao', 'https://en.wikipedia.org/wiki/Tianjin', 'https://en.wikipedia.org/wiki/Yantai', 'https://en.wikipedia.org/wiki/Weihai', 'https://en.wikipedia.org/wiki/Qingdao', 'https://en.wikipedia.org/wiki/Rizhao', 'https://en.wikipedia.org/wiki/Lianyungang', 'https://en.wikipedia.org/wiki/Nantong', 'https://en.wikipedia.org/wiki/Zhenjiang', 'https://en.wikipedia.org/wiki/Jiangyin', 'https://en.wikipedia.org/wiki/Nanjing', 'https://en.wikipedia.org/wiki/Shanghai', 'https://en.wikipedia.org/wiki/Ningbo', 'https://en.wikipedia.org/wiki/Zhoushan', 'https://en.wikipedia.org/wiki/Jiujiang', 'https://en.wikipedia.org/wiki/Taizhou,_Zhejiang', 'https://en.wikipedia.org/wiki/Wenzhou', 'https://en.wikipedia.org/wiki/Taizhou (South of Wenzhou)', 'https://en.wikipedia.org/wiki/Changle', 'https://en.wikipedia.org/wiki/Quanzhou', 'https://en.wikipedia.org/wiki/Xiamen', 'https://en.wikipedia.org/wiki/Shantou', 'https://en.wikipedia.org/wiki/Jieyang', 'https://en.wikipedia.org/wiki/Guangzhou', 'https://en.wikipedia.org/wiki/Zhuhai', 'https://en.wikipedia.org/wiki/Shenzhen', 'https://en.wikipedia.org/wiki/Zhanjiang', 'https://en.wikipedia.org/wiki/Beihai', 'https://en.wikipedia.org/wiki/Fangchenggang', 'https://en.wikipedia.org/wiki/Haikou', 'https://en.wikipedia.org/wiki/Basuo', ] ``` Time for running our function to extract coordinates from each port's page. We will be storing them in a dictionary called lat\_log\_dict

# create a dictionary to store the coordinates
lat_log_dict = {}

# counter
cnt = 0

# loop through the list of urls and extract and store the coordinates

for url in url_list:
    # copy url
    wikipedia_url = url
    
    # extract the name of the port
    port = wikipedia_url.split("/")[-1]
    
    try:
        latlon = get_latlon_from_wikipedia(wikipedia_url)
        # print(f" {cnt} - Lat/Lon for {port}:", latlon) # to display current port and its coordinates
        
        lat_log_dict[port] = latlon
        
    except Exception as e:
        print("Error:", str(e))
    cnt+=1
        
lat_log_dict

Output: ``` {'Dalian': ['38°54′N', '121°36′E'], 'Yingkou': ['40°37′30″N', '122°13′08″E'], 'Jinzhou': ['41°07′44″N', '121°08′53″E'], 'Qinhuangdao': ['39°53′18″N', '119°31′13″E'], 'Tianjin': ['39°08′01″N', '117°12′19″E'], 'Yantai': ['37°27′53″N', '121°26′52″E'], 'Weihai': ['37°30′48″N', '122°07′14″E'], 'Qingdao': ['36°04′01″N', '120°22′58″E'], 'Rizhao': ['35°25′01″N', '119°31′37″E'], 'Lianyungang': ['34°35′48″N', '119°13′17″E'], 'Nantong': ['31°58′52″N', '120°53′38″E'], 'Zhenjiang': ['32°11′17″N', '119°25′26″E'], 'Jiangyin': ['31°50′20″N', '120°17′42″E'], 'Nanjing': ['32°03′39″N', '118°46′44″E'], 'Shanghai': ['31°13′43″N', '121°28′29″E'], 'Ningbo': ['29°51′37″N', '121°37′28″E'], 'Zhoushan': ['29°59′08″N', '122°12′27″E'], 'Jiujiang': ['29°39′40″N', '115°57′14″E'], 'Taizhou,_Zhejiang': ['28°39′21″N', '121°25′15″E'], 'Wenzhou': ['27°59′38″N', '120°41′57″E'], 'Changle': ['25°55′N', '119°33′E'], 'Quanzhou': ['24°52′28″N', '118°40′33″E'], 'Xiamen': ['24°28′47″N', '118°05′20″E'], 'Shantou': ['23°21′14″N', '116°40′55″E'], 'Jieyang': ['23°33′04″N', '116°22′22″E'], 'Guangzhou': ['23°07′48″N', '113°15′36″E'], 'Zhuhai': ['22°16′18″N', '113°34′37″E'], 'Shenzhen': ['22°32′29″N', '114°03′35″E'], 'Zhanjiang': ['21°16′12″N', '110°21′27″E'], 'Beihai': ['21°28′52″N', '109°07′12″E'], 'Fangchenggang': ['21°41′12″N', '108°21′17″E'], 'Haikou': ['20°01′07″N', '110°20′56″E'], 'Basuo': ['19°05′31″N', '108°40′16″E']} ``` ### Step 3: Storing Data with Pandas With the data extracted, we'll use the Pandas library to organize and store the port details in a structured format, making it easier to work with the data. ```python # create a dataframe df = pd.DataFrame(data=lat_log_dict) df = df.T df.reset_index(inplace=True) # update the column names df = df.rename(columns={'index':'Port', 0:'Lat (DMS)',1:'Lon (DMS)'}) # diplay first 3 rows df[:3] ``` Output:

We now have coordinates for all the Major ports in China. All the coordinates are degree-minutes-seconds format. Python's mapping package, Folium, however, works with decimal degress. Therefore, before we proceed to map the ports we need to convert the latitude and longitudes values to decimal degrees {% hint style="info" %} Conversion Formula: **decimal degrees = degrees + (minutes / 60) + (seconds/3600)** {% endhint %} To do that we will create another function called **dms2dd** that will convert each coordinate to decimal degrees. ```python def dms2dd(s): # example: s = """ 0°11'23.29"S """ if '″' not in s: s = s[:-1] + '0″'+ s[-1] degrees, minutes, seconds, direction = re.split('[°′″\"]+', s) dd = float(degrees) + float(minutes)/60 + float(seconds)/(60*60); if direction in ('S','W'): dd*= -1 return dd ``` We will now apply the function using **pandas' apply** method to latitude and longitude data: ```python df['Lat'] = df['Lat (DMS)'].apply(dms2dd) df['Lon'] = df['Lon (DMS)'].apply(dms2dd) df.head(10) ``` Output:

### Step 4: Setting Up Folium and OpenStreetMap (OSM) We'll begin visualizing the port locations using the Folium library with the 'OpenStreetMap (OSM)' base map. It will give us a basic map view but may present location names in Chinese, which can be a bit challenging for non-Chinese speakers. ```python # Tiles from OpenStreetMap (it does not offer English names for local cities, use MapBox instead) tiles = 'OpenStreetMap' # put geo. locations to a list location_list_df = df[['Lat','Lon']].values.tolist() # Display the ports # create basemap map_obj = folium.Map(location=[df.Lat.mean(), df.Lon.mean()], tiles=tiles, zoom_start=4) # place ports on the map for loc in range(len(location_list_df)): folium.Marker(location=location_list_df[loc], popup=df.iloc[loc].Port, # icon = folium.Icon(color='blue', icon=f'info-sign') # if wants to use an icon icon = folium.DivIcon( html=('') ) ).add_to(map_obj) # save the map # map_obj.save('china_ports_osm.html') # display the map map_obj ``` Output:

> Due to HTML limitations the map pasted above is not interactive. Once you run the code you will be able interact with it on your computer as shown below {% embed url="" %} Interactive Map of China via [OpenStreetMap](https://www.openstreetmap.org/#map=5/29.693/117.706) {% endembed %} As you can see in the map that all names are in local language, meaning unless you speak Chinese, we have no way of telling where, for instance, the city of Dilian is. Therefore we need to translate the names into English. At the time of this writing (May 2023), OpenStreetMap does not support/offer English translations for countries where English is not the official language. ### Step 5: Creating a Mapbox Account To overcome the language issue, we'll create an [account on Mapbox](https://account.mapbox.com). Mapbox offers an English base map, which will make our map more user-friendly non-Chinese speakers. {% embed url="" %} ### Step 6: Generating a Mapbox Public Key Once you have your Mapbox account set up, you will generate a public key, or used the default, to access the Mapbox maps in our Python script. For this tutorial, I created an access token called *china\_ports*

**Step 7: Using Mapbox's English Base Map** We'll modify our Python script to use the Mapbox's Map with the English base layer. This will allow us to display location names in English, and enhance the user experience. The map will have interactive features (once run), allowing users to zoom in, zoom out, and click on port markers to get additional information. ```python # Tiles from MapBox mapbox_access_token = '' # example: mapbox_access_token='pk.eyJ1I...' # Select a tile style for Mapbox map tileset_ID_str = "streets-v12" # Mapbox Styles: https://docs.mapbox.com/api/maps/styles/ tilesize_pixels = "256" tiles = f"https://api.mapbox.com/styles/v1/mapbox/{tileset_ID_str}/tiles/{tilesize_pixels}/{{z}}/{{x}}/{{y}}@2x?access_token={mapbox_access_token}" # Display the ports # create basemap location_list_df = df[['Lat','Lon']].values.tolist() # place ports on the map map_obj = folium.Map(location=[df.Lat.mean(), df.Lon.mean()], tiles=tiles, zoom_start=4, attr='Mapbox') for loc in range(len(location_list_df)): folium.Marker(location=location_list_df[loc], popup=df.iloc[loc].Port, # icon = folium.Icon(color='blue',icon=f'info-sign') icon = folium.DivIcon( html=('') ) ).add_to(map_obj) # save the map map_obj.save('china_ports_mbx.html') # display the map map_obj ``` Output:

Once again due to HTML limitations the map pasted above is not interactive. Once you run the code you will be able interact with it on your computer. For interactive Mapbox Map please click the following button {% embed url="" fullWidth="false" %} ## Conclusion Congratulations! We've successfully created an interactive map displaying major ports in China using Python, BeautifulSoup, Pandas, and Folium with Mapbox's English base map. You can now explore and share this map with others, enhancing their understanding of the major ports in China. \
--- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://dshub.gitbook.io/ds-hub/geospatial-analysis/gsa-at-work/web-scraping-and-mapping.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.