# Pagination Tutorial source: https://developer.mastercard.com/bin-lookup/documentation/tutorials-and-guides/pagination-tutorial/index.md ## Overview {#overview} This tutorial covers how to use pagination to call the API in sequence, and pull all of the data. It also demonstrates how to store the results of each call in memory and also to a CSV file. Python is the language used in this tutorial but the steps, and code, will be similar for other languages. At the end of this tutorial you will have a simple Python script that when run will call the API several times in a paginated fashion until it has received all of the data. The code will store the results in an array and also convert them into a CSV file. ### Prerequisites {#prerequisites} For this tutorial you must ensure you have `Python` and `pip` installed on your system. If you don't already have Python installed, check out the [Python documentation](https://www.python.org/doc/). Tip: If you have followed any of the other BIN Lookup tutorials in this section, you can skip steps 1 and 2 because they are the same for every tutorial. If you want to get straight to the code, all example files are available on [Github](https://github.com/Mastercard-Samples/bin-lookup-sample-code) ## 1. Create the File \& Download Dependencies {#1-create-the-file--download-dependencies} First, we need to ensure to have the dependencies needed for our code. We install the `mastercard-oauth1-signer` and `requests` packages via the following command line: ```terminal pip install mastercard-oauth1-signer requests ``` Next, create a file called `pagination-example.py` and add the following imports at the top of the file: ```python import requests from requests.auth import AuthBase import oauth1.authenticationutils as authenticationutils from oauth1.signer import OAuthSigner import csv ``` ## 2. Configuring Authentication {#2-configuring-authentication} Next, create some variables to hold the base URL, which is the url for the API we will be calling, and for your consumer key. You can get your consumer key from your project in the projects dashboard. ```python BASE_URL = 'Add Sandbox or Production BASE URL here' CONSUMER_KEY = 'Add your project consumer key here' ``` Following that, use the this code to create a simple class that will sign all HTTP requests we send using the Python requests library: ```python # MCSigner # Helper class for signing request objects class MCSigner(AuthBase): def __init__(self, consumer_key, signing_key): self.signer = OAuthSigner(consumer_key, signing_key) def __call__(self, request): self.signer.sign_request(request.url, request) return request ``` Finally, use the Mastercard OAuth library to create a signing key using the .p12 cert that you downloaded when you created your project, and the keystore password that you set: ```python # Generate a signing key and use it, and consumer key, with the signer class signing_key = authenticationutils.load_signing_key('./certs/sandbox.p12', 'keystorepassword') signer = MCSigner(CONSUMER_KEY, signing_key) ``` ## 3. The Pagination Loop {#3-the-pagination-loop} Now that we have our credentials configured and our requests handler setup for signing, we can begin calling the paginated API. The API provides metadata to tell us what page we are on, how many records there are in total, and how many pages in total too. Lets create a method that we can call that will sequentially call the API to download all account range records. ```python def fetch_data_from_api(base_url, initial_page=1, post_payload={}, signer=None, data=None): all_items = [] current_page = initial_page total_items_downloaded = 0 total_items = None while True: # Update the payload to include the current page number for pagination post_payload.update({"page": current_page, 'size': '10000'}) # Perform a POST request to the API response = requests.post(base_url, params=post_payload, auth=signer, json=data) response_data = response.json() # Extract metadata current_page_number = response_data['currentPageNumber'] total_pages = response_data['totalPages'] total_items = response_data['totalItems'] # Extract the actual items items = response_data.get('items', []) # Add the items from the current page to the master list all_items.extend(items) # Update the total number of items downloaded total_items_downloaded += len(items) # Print the current progress for reference print(f"Downloaded {len(items)} items from page {current_page_number}/{total_pages}.") # Check if we have reached the last page if current_page_number >= total_pages: break # Move to the next page current_page += 1 # After the loop, verify that the number of downloaded items matches the totalItems value if total_items_downloaded == total_items: print(f"Successfully downloaded all {total_items_downloaded} items.") else: print(f"Warning: Downloaded {total_items_downloaded} items, but expected {total_items} items.") return all_items ``` Note: You can increase or decrease the size of the records you want by changing the size value. We recommend a number between 5000 and 10000, less records per page mean more API calls, more means longer response times. And to call this method we use the following code, once the download is complete you can then store the results: ```python all_records = fetch_data_from_api(base_url=f'{BASE_URL}/bin-ranges', initial_page=1, post_payload={}, signer=signer) ``` ## 4. Storing in a CSV {#4-storing-in-a-csv} You now have all records stored in the items variable, but if you want to store them persistently here is some sample code to save the data to a CSV file: ```python # Set up a file to store the results from the API data_file = open('account_ranges.csv', 'w', encoding='utf-8') csv_writer = csv.writer(data_file) # A for loop to go through the JSON objects and convert them into CSV rows count = 0 for item in all_records: if count == 0: # Writing headers of CSV file header = item.keys() csv_writer.writerow(header) count += 1 # Writing data of CSV file csv_writer.writerow(item.values()) data_file.close() ``` To execute the Python script, use the following command: ```terminal python pagination-example.py ``` Once the script completes you will have a CSV called `account-ranges.csv` showing all of the records from the BIN Lookup service. ## Code {#code} Here is the full Python source code: ```python import requests from requests.auth import AuthBase import oauth1.authenticationutils as authenticationutils from oauth1.signer import OAuthSigner import csv BASE_URL = 'Add Sandbox or Production BASE URL here' CONSUMER_KEY = 'Add you project consumer key here' # MCSigner # Helper class for signing request objects class MCSigner(AuthBase): def __init__(self, consumer_key, signing_key): self.signer = OAuthSigner(consumer_key, signing_key) def __call__(self, request): self.signer.sign_request(request.url, request) return request # Generate a signing key and use it, and consumer key, with the signer class signing_key = authenticationutils.load_signing_key('./certs/sandbox.p12', 'keystorepassword') signer = MCSigner(CONSUMER_KEY, signing_key) def fetch_data_from_api(base_url, initial_page=1, post_payload={}, signer=None, data=None): all_items = [] current_page = initial_page total_items_downloaded = 0 total_items = None while True: # Update the payload to include the current page number for pagination post_payload.update({"page": current_page, 'size': '10000'}) # Perform a POST request to the API response = requests.post(base_url, params=post_payload, auth=signer, json=data) response_data = response.json() # Extract metadata current_page_number = response_data['currentPageNumber'] total_pages = response_data['totalPages'] total_items = response_data['totalItems'] # Extract the actual items items = response_data.get('items', []) # Add the items from the current page to the master list all_items.extend(items) # Update the total number of items downloaded total_items_downloaded += len(items) # Print the current progress for reference print(f"Downloaded {len(items)} items from page {current_page_number}/{total_pages}.") # Check if we have reached the last page if current_page_number >= total_pages: break # Move to the next page current_page += 1 # After the loop, verify that the number of downloaded items matches the totalItems value if total_items_downloaded == total_items: print(f"Successfully downloaded all {total_items_downloaded} items.") else: print(f"Warning: Downloaded {total_items_downloaded} items, but expected {total_items} items.") return all_items all_records = fetch_data_from_api(base_url=f'{BASE_URL}/bin-ranges', initial_page=1, post_payload={}, signer=signer) # Set up a file to store the results from the API data_file = open('account_ranges.csv', 'w') csv_writer = csv.writer(data_file) # A for loop to go through the JSON objects and convert them into CSV rows count = 0 for item in all_records: if count == 0: # Writing headers of CSV file header = item.keys() csv_writer.writerow(header) count += 1 # Writing data of CSV file csv_writer.writerow(item.values()) data_file.close() ```