Python Data Stream Retrievals

Python can be useful for retrieving data from different sources in different formats. The fact that Python is a script makes it flexible in dealing with varying formats, requirements, and scenarios. Below are some examples of use cases that are best addressed by using a Python-based connector:

1. Retrieving data via an API connection where multiple HTTP calls are required in order to handle any of the following cases:
   • Obtaining an OAuth 2.0 token.
   • Retrieving Entities.
   • Looping Through Calls.
   • Paginating through the response.
2. Parsing the data during the retrieval process itself (prior to having it available in the mapping screen) so that it’s easier to work with within Marketing Cloud Intelligence Data Pipelines.

Marketing Cloud Intelligence Data Pipelines will successfully run your Python script as long as the following criteria are met:

1. Script must be written in Python 3.6.
2. Script must only use packages supported by Marketing Cloud Intelligence Data Pipelines (See the supported packages section below).
3. The dataset generated by the script must be one that can be uploaded to a Data Stream.
4. Script processing time must not exceed 10 minutes.
5. Script memory consumption does not exceed 1.5 GB.

There are two different ways to write Python scripts:

• Internal - The Python module is read via Marketing Cloud Intelligence Data Pipelines data streams.
• External - The script is run via a local IDE.

The examples below show how to write internal scripts. If you are writing external scripts, you must write the scripts as follows:

• Call the datorama module via “from datorama.datorama import *” instead of “import datorama”
• Remove “dataorama.” from the beginning of each function.

Here you can see a live example of how this looks:

from datorama.datorama import *
 
#Generating a list of values for our headers' row
row_values_list = ['date','campaign','impression','clicks']
 
# Writing a new row to the csv input file from values list
add_row(row_values_list)

Example 1 - using add_row() and save():

import datorama 
#Generating a list of values for our headers' row 
row_values_list = ['date','campaign','impression','clicks'] 
# Writing a new row to the csv input file from values list 
datorama.add_row(row_values_list) 
#Generating two more lists containing row values 
#Note that the values of a specific column are in the same position in their row's list as the position of their header in the headers' row 
row2 = ['01-12-2017','Campaign 1',15,6] row3 = ['02-12-2017','Campaign 2',20,10] 
# Creating a list of the two rows we want to write at once 
multiple_rows = [row2,row3] 
# Writing several rows at once to the csv by sending a list of multiple values' lists 
datorama.add_rows(multiple_rows) 
# Saving the csv file to upload it to our data stream 
datorama.save()

Example 2 - using save_csv():

import datorama 
#Generate the exact same file by providing a csv formatted string of the file's content 
csv_string = '''"date","campaign","impression","clicks"\n\ "01-12-2017","Campaign 1", 15, 6\n\ "02-12-2017","Campaign 2", 20, 10'''
#Saving the csv file, based on the input string 
datorama.save_csv(csv_string)

The Python Retrieval Method supports a limited list of Python Packages, which are commonly used for handling data retrieval. Importing and using a non-supported Package will result in in script failure. If your needs cannot be met by using one of the existing Supported Packages, and can only be met by using a Package that is not currently supported, please log a Feature Request with your CSM so that the requested package can be considered to be added to the Supported Packages list in the future. The Supported Packages are those supported by ‘AWS Lambda’ as well as the ones shown below:

• Requests
• re
• ftplib
• Pandas
• oauthlib
• cloudstorage
• boto3
• requests_oauthlib
• webapp2
• boto
• Numpy
• logging
• json
• tweepy
• os
• datetime
• oauth2client
• googleapiclient
• apiclient
• httplib2
• Googleads
• pyasn1
• Facebookads
• tabula-py (the syntax to import this package is: 'import tabula')
• xlrd
• mlxtend
• Selenium
• simple_salesforce
• scrapy
• time
• base64
• crypt

This script retrieves data about Demographic Statistics, By Zip Code, published by data.cityofnewyork.us

import datorama import urllib2
     import jsonurl = "https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.json?accessType=DOWNLOAD"
     req = urllib2.Request(url)
     response = urllib2.urlopen(req)
     response_str = response.read().decode('utf-8')
     response_json = json.loads(response_str)
     columns_js = response_json["meta"]["view"]["columns"]
     rows_list = response_json["data"]
     headers = []
     for column in columns_js: headers.append(column["name"])
     datorama.add_row(headers)
     datorama.add_rows(rows_list)
     datorama.save()

This script retrieves publicly available data from Twitter for different Screen Names, with Metrics such as Follows, Comments, etc. Using it requires having a valid App registered with Twitter:

import datoramaimport tweepyimport datetimeimport jsonimport 
     
     pandas as pdscreen_names = ["NASCAR", "F1", "PGATOUR", "MotoGP", "IndyCar", "ufc", "WWE"]
     report_fields = ["screen_name", "statuses_count","followers_count","friends_count","listed_count","favourites_count" ]
     #Twitter API 
     credentialsconsumer_key = ""consumer_secret = ""access_key = ""access_secret = "" 
     #Calculating the date of the run
     now = datetime.datetime.now()
     date = now.strftime("%d-%m-%Y")
     #authorize twitter, initialize 
     tweepyauth = tweepy.OAuthHandler(consumer_key, consumer_secret)
     auth.set_access_token(access_key, access_secret)
     api = tweepy.API(auth)
     full_report = pd.DataFrame()
     #Pulling the report from Twitter per screen namefor screen_name in screen_names: 
     get_user = api.get_user(screen_name = screen_name) 
     #Using pandas DataFrame to manipulate the report 
     df = pd.DataFrame(get_user._json) 
     #Filtering the report only on requested fields 
     requested_df = df[report_fields] 
     #Adding the screen name to the data set 
     requested_df["account"] = screen_name 
     #Appending the current screen name's data set to the full report of all screen names 
     full_report =full_report.append(requested_df, ignore_index=True)
     #Adding the run date in the "date" column of the data 
     setfull_report["date"] = date
     #Creating a csv string of the data 
     setcsv_string = full_report.to_csv()
     #Saving the csv 
     datorama.save_csv(csv_string)