An Idiot’s Guide to RPMA Development – Part 3
In the third and final part of this blog series, I’ll focus on building a parser to grab our data from Intellect and send it on up to AT&T’s M2X platform.
So now that our Arduino is pulling some measurements and sending them to a properly configured RPMA DevKit board (rACM), it’s time to pull that data out of Intellect and send it on to M2X. This part got pretty tricky for me, as it was my first real foray into Python. Plus, M2X is really picky about the data you feed it.
Our Starter Pack includes a handful of REST examples that you can use to read device data, send data downstream to a device from the cloud, and so on. The readmeConsole.txt doc is surprisingly helpful here. Read it! Our engineers have crafted a few parsers that work with M2X, but the serial example is limited to a single value. Since we’re sending two values in one string, we’ll have to split it up with a custom parser before sending it up to M2X. There are three files that we’re going to modify in our M2M_REST_Examples folder: createDevices.py, parsers.py, and restm2x.py.
createDevices.py
First, let’s update createDevices.py with our device info and some API keys from M2X. You created an account on M2X, right? Open createDevices.py with the editor of your choice (I just use TextEdit.app on my Mac). You’ll likely be greeted by a bewildering array of various devices. Time to add ours to the mix! Delete all but one, which we’ll modify to meet our needs.
| devices = [ | |
| {‘desc’:’Arduino-rACM’, // whatever you feel like | |
| ‘nodeId’:’0x3451f’, // add your Node ID here | |
| ‘parser’:’serial_m2x’, // let’s reference the parser we’ll create | |
| ‘m2x_device_id’: ‘z547acf8d450fc94863a5b7b498b6135’, // grab from M2X | |
| ‘m2x_primary_key’:’n0b9b4d39c1bb86e733f37b62c666128′, // same | |
| ‘alarm_email_enabled’:0, | |
| ‘alarm_email_list’:[]}, | |
| ] |
by Make yours like mine. Your nodeId can be found by running that GET_DEVICE_STATUS command from Part 2, or peering at the top of your development board. It’s the many-digits-long value under MAC on the module sticker.
After creating an M2X account, you’re offered the chance to create a device. Make yours numerical! If it prompts you to create your first stream, name it temp_f and then name a second stream humidity. Then copy-paste Device ID and Primary API keys from M2X into createDevices.py. Save your changes and you’re done there. Take care throughout this whole process that your text editor doesn’t change apostrophes and quotation marks from straight-up-and-down marks to fancy curved open and close marks. TextEdit.app does this by default and it is absolutely maddening.
parsers.py
Next up we’ll need to add our new parser to parsers.py. I ended up modifying the existing Serial Uplink parser to handle an additional sensor, and it did the trick. The changes I made should be pretty self-explanatory. I’m not going to get into the different sensorIds and sensorTypes: they’re explained over in readmeConsole.txt. Go ahead and add the following code somewhere amongst the existing parsers and save the document.
| #—————————————————————————— | |
| # Temp/Humidity Arduino | |
| # APP_INTF7 Application UART A_UART_RX J207 pin 7 | |
| # Sample of a serial-based Arduino-to-rACM setup. Add additional sensor inputs | |
| # under ‘expectedSensors’. Since we’re sending all sensor data to rACM as a single | |
| # serial string all sensorIds are ‘6’ and sensorType is ‘0xFFF’. sensorName should | |
| # correspond with your M2X stream names. | |
| #—————————————————————————— | |
| def parser_serial_m2x(raw_hex): | |
| expectedSensors = [{‘sensorId’: 6, ‘sensorType’:0xFFF, ‘sensorName’:’humidity’, | |
| ‘sensorDesc’:’Relative Humidity, Arduino’}, | |
| {‘sensorId’: 6, ‘sensorType’:0xFFF, ‘sensorName’:’temp_f’, | |
| ‘sensorDesc’:’Temperature (deg F), Arduino’}] | |
| expectedAlarmTypes = [] # no alarms from this device | |
| msg = RacmUlMssg(raw_hex, expectedSensors, expectedAlarmTypes) | |
| if msg.getStatus() == ‘OK’: | |
| if msg.getMsgType() == ‘Alarm’: | |
| return(msg.getMsgType(), msg.getAlarmData()) | |
| else: | |
| return(msg.getMsgType(), msg.getData()) | |
| else: | |
| # Debug | |
| print ‘ERROR: ‘, msg.getError() | |
| print ‘raw_hex = ‘, raw_hex | |
| return(‘ERROR’,”) |
rest2m2x.py
Modifying rest2m2x.py was kind of a beast for me, but I’m going to put most of the blame on M2X for being such a stickler about the data it can process. I began the whole thing by modifying the existing serial example provided in the script. I’m absolutely sure that someone with more experience could have handled parsing the Arduino’s string into two separate numerical values more efficiently than I. Bear with me, please.
| # ————————————————————————————————— | |
| # Temp/Humidity (ARDUINO Serial) | |
| # This bit calls the new parser and also parses the serial payload into segments for M2X. It grabs the serial | |
| # string from an arduino and parses it into chunks delineated by the ‘_’ character. Then it formats/sanitizes | |
| # the separate strings into numeric characters for M2X ingest. | |
| # Each segment can be called by using text[0] (first segment) text[1] (second segment) and so on. | |
| # NOTE: You can set the M2X stream data type to numeric with this parser. | |
| # ————————————————————————————————— | |
| elif dev_info[‘parser’] == ‘serial_m2x’: # Referencing the appropriate parser over in /parsers.py | |
| print ‘nBegin parsing ULSDU at %s with: serial_m2x, sdu_id %s nnodeId %s rx_timestamp %s payload %s’ | |
| % (datetime.datetime.now(), sdu_id, hex(int(node_id_hex, 16)), rx_timestamp, payload) | |
| (msgType, data) = parsers.parser_serial_m2x(payload) | |
| # Note we do not send the test button alarm to M2x, so we only accept one message type | |
| if msgType == ‘Serial’: # We’re still pulling a serial-type message from Intellect | |
| allMeasAccepted = 1 # Default means M2X accepted all measurements | |
| for i in range(len(data)): | |
| timestamp = convertToDateTime(rx_timestamp).strftime(“%Y-%m-%dT%H:%M:%SZ”) # Timestamp format m2x expects | |
| serialDATA = hex2text(data[i][‘data’]) # Converting hex to ASCII. | |
| text = serialDATA.split(“_”); # Splitting ASCII string into chunks | |
| # Replace ‘temp_f’ below with the name of your first M2X stream | |
| # ‘re.sub(“[^0-9^.]”, “”, text[0])’ below sanitizes string for M2X by stripping out any non-numeric characters. M2X is picky. | |
| res = sendStream2M2x(dev_info[‘m2x_primary_key’], dev_info[‘m2x_device_id’], | |
| ‘temp_f’, timestamp, re.sub(“[^0-9^.]”, “”, text[0])) # Replace ‘temp_f’ with name of your first M2X stream | |
| if res == ‘{“status”:”accepted”}’: | |
| print “OK: M2X Stream Updated on ATT Servers”, ‘temp_f’,datetime.datetime.now(), | |
| ‘n text = ‘, text[0], hex(int(node_id_hex, 16)) | |
| else: | |
| print “ERROR: M2X Stream Failed to Update on ATT Servers reason: %s” % (res) | |
| print data[i][‘sensorName’], timestamp, data[i][‘data’], text[0], hex(int(node_id_hex, 16)), | |
| datetime.datetime.now() | |
| # We can’t let a misconfigured device block all other UL SDUs | |
| # so we drop this SDU and must go back and push it to M2x in the future | |
| incrementNextUlSduPointer(nextSduFile, last_sdu_id) | |
| allMeasAccepted = 0 | |
| # Now we run the same sequence again for our second M2X stream ‘humidity’. Rinse and repeat for each additional stream you want data sent to. | |
| res = sendStream2M2x(dev_info[‘m2x_primary_key’], dev_info[‘m2x_device_id’], | |
| ‘humidity’, timestamp, re.sub(“[^0-9^.]”, “”, text[1])) # Replace ‘humidity’ with name of your second M2X stream | |
| if res == ‘{“status”:”accepted”}’: | |
| print “OK: M2X Stream Updated on ATT Servers”, ‘humidity’,datetime.datetime.now(), | |
| ‘n text = ‘, text[1], hex(int(node_id_hex, 16)) | |
| else: | |
| print “ERROR: M2X Stream Failed to Update on ATT Servers reason: %s” % (res) | |
| print data[i][‘sensorName’], timestamp, data[i][‘data’], text[1], hex(int(node_id_hex, 16)), | |
| datetime.datetime.now() | |
| # We can’t let a misconfigured device block all other UL SDUs | |
| # so we drop this SDU and must go back and push it to M2x in the future | |
| incrementNextUlSduPointer(nextSduFile, last_sdu_id) | |
| allMeasAccepted = 0 | |
| # We only increment next SDU if the current SDU made it to M2X completely | |
| # or in the case of certain errors | |
| if allMeasAccepted: | |
| incrementNextUlSduPointer(nextSduFile, last_sdu_id) | |
| else: | |
| print ‘INFO: msgType will not be sent to M2x: ‘, msgType | |
| print ‘Data: ‘, data | |
| incrementNextUlSduPointer(nextSduFile, last_sdu_id) # increment if we discard and don’t send to M2X |
While this script is running, it pulls data from Intellect and parses data from our device, and then sends the data up to M2X. Since the Arduino is transmitting a text string with an underscore ‘_’ between the two values, we need to split the string up and send each value separately to M2X. Since M2X is particular about the sort of stuff we send over, I also have each string segment run through a filter that removes any hidden non-numerical characters, leaving the pure data behind. Then we send each value to separate M2X streams; in this case humidity and temp_f.
Running a Script, Parsing Data
We’re almost there! You have to update ATT_Devices.json by running the following command: ‘python createDevices.py’ which will populate ATT_Devices.json with your RPMA DevKit’s info. Now, if everything has been inputted correctly, you should be able to run rest2m2x.py and have your data kicked up into M2X. Log in to M2X and watch the data pour in!
That’s it! You’re done! What are you still doing here? Well, yes, I did learn a bunch of things from this exercise. Developing a sensor solution with RPMA connectivity in mind turns out to be super easy. Pair an Arduino or Raspberry Pi to one of our RPMA DevKits and you get a surprisingly potent proof of concept for your IoT application that doesn’t take a lot of technical know-how to get going. Request an RPMA DevKit below and start cracking!