Charge for Routing

[vstadmin]

When was this $0.0006 charge for routing implememented? I did not see that when we started using blues and I don’t remember seeing it recently.

I have a system that will be sending data every 10 sec to our servers that will require 172,800 “events” per month which comes to over $100 / device per month!

Totally unworkable. I would swear that charge was not there recently.

[vstadmin]

May 4, 2021:

June 3, 2021:

When I started this project in March, I was led to believe via the documentation that there was not siginficant charge for events. Now suddently this new pricing sheet tells me it will cost me OVER $100 / device per month! I might as well quit right now.

This is EXACTLY what Particle did to us that caused us to find Blues as an alternative.

[RobLauer]

Hi @vstadmin,

Unless your solution truly requires low latency event streaming (every 10 seconds as you said), you can dramatically lower the number of routed events by either:

• including more data (e.g. minutes of accumulated data) in a single Note;
• or adding a JSON array of accumulated data points (minutes? hours? depending on the data) to a single Note, like:

{"all_data":[    
    {"temp":23, "time":1000},    
    {"temp":24, "time":1001}, 
    {"temp":25, "time":1002}, 
    {"temp":26, "time":1003},   
]}

Let me know if you have any follow up questions!

Thanks,
Rob

[vstadmin]

I am making some changes to accommodate accumulating data on a different timer than sending the data to notehub.io.

Here is what I had before:

def fifteen_sec_updates():

    '''
    These are the things to do once every 15 Seconds

    Collect Data
    '''

    global cont

    to_modem = {key:cont[key] for key in ['pressure','runcycles','faults','mode','temp']}
          
    if DEBUGGING:
        logging.debug(f'Modem - Sending: {to_modem}')

    ### Output for blues modem

    req = {"req" :"note.add"}
    req["file"] = "vstdata.qo"
    req["sync"] = True
    req["body"] = to_modem

If I change it to this:

def fifteen_sec_updates():

    '''
    These are the things to do once every 15 Seconds

    Collect Data
    '''

    global cont

    to_modem = {key:cont[key] for key in ['pressure','runcycles','faults','mode','temp']}
          
    if DEBUGGING:
        logging.debug(f'Modem - Sending: {to_modem}')

    ### Output for blues modem

    req = {"req" :"note.add"}
    req["file"] = "vstdata.qo"
    req["sync"] = False
    req["body"] = to_modem

and then add:

def ten_minute_updates():

    '''
    These are the things to do once every 10 minutes

    Blues has changes the rules on talking to their HUB, so I have changed
    these routines to hit their hub once every 10 minutes instead of every 15 seconds
    but i'm accumulating the 15 second data
    
    '''

    global cont

    req["sync"] = True

    try:

        rsp = card.Transaction(req)

    except:
        
        logging.error('No response from modem.  Will keep trying...')

will this accomplish accumulating data every 15 seconds to then be later sent every 10 minutes? Of course I have a main timing loop that fires these functions off at the appropriate time.

[vstadmin]

unfortunately, I set this to be:

    req = {"req": "hub.set"}
    req["product"] = productUID
    req["mode"] = "periodic"
    req["sync"] = True
    req["outbound"] = 1

and it is accumulating every 15 seconds and syncing once every minute, but its registering a separate event for each record which defeats the purpose.

More ideas?

[vstadmin]

@RobLauer

• including more data (e.g. minutes of accumulated data) in a single Note;
• or adding a JSON array of accumulated data points (minutes? hours? depending on the data) to a single Note, like:

I really don’t know how to accumulate more data inside one note. What are the limitations on number of bytes / blocks, etc.

[gwolff]

I think I see what you’re getting at.

Essentially you want req["body"] to be set to an array of dicts.
It is the note.add that counts as the event, not the sync. (Admittedly a little confusing)

My Python is a little weak, so pardon me if there’s some syntax errors in my snippet here:

to_modem= []
MaxNumDataPointsBeforePushToNotecard = 40 # 4x per minute for 10 minutes
 global cont

    
    data_to_store = {key:cont[key] for key in ['pressure','runcycles','faults','mode','temp']}

   to_modem.append(data_to_store);

   if(to_modem.len() >= MaxNumDataPointsBeforePushToNotecard ):
      if DEBUGGING:
         logging.debug(f'Modem - Sending: {to_modem}')

       ### Output for blues modem

      req = {"req" :"note.add"}
      req["file"] = "vstdata.qo"
      req["sync"] = False
      req["body"] = {"data":to_modem}

      to_modem= []

[vstadmin]

Thank you, I will look at this in detail.

What is the maximum amount of data and overhead that can be sent in a note?

[vstadmin]

Is 250 bytes the limitation on the the body? So if my data was 25 bytes, I could only store 10 records before I ran out of space? This is going to be very difficult.

[gwolff]

The “payload” field byte limit is 250
The “body” doesn’t have that limit

The total size for the note is limited to ~ 4KB

[gwolff]

I strongly recommend using note.template to describe the structure of your Notes prior to adding Notes to the Notefile.

What are Note Templates and Why Should I Use Them

v20201208.1 What are Note Templates? Note Templates are a description of a prescribed data structure with pre-set sizes for data added to the body of Notes in a Notefile. A Note Template names each field of the Note Body. Each named field must also describe the datatype of the field value. For more information about how to create Note Templates, see the note.template request documentation. Why should you use them? Note Templates provide Notecard with way to store future Notes added to a No…

[vstadmin]

That is very good to know. How can you check to see the entire note size before you send it.

[RobLauer]

In Python you can always check the size of an object with sys.getsizeof.

[vstadmin]

I can get the size of the block I create, but can I get the size of the overall JSON that notehub will receive?

[RobLauer]

Checking the size of the JSON object you are sending is a good start, but you’re right it doesn’t tell 100% of the story.

The only way to perfectly measure the total amount of data sent to Notehub is to use the card.usage APIs, documented here: Low Bandwidth Design - Blues Developers

[vstadmin]

Good morning.

I am attempting to use note templates.

i have the following defined:

    modem_template = {"seq":11, "pressure":12.1, "runcycles":12, "faults":12, "mode": 11, "temp": 12.1 }

    req = {"req" : "note.template"}
    req["file"] = "vstdata.qo"
    req["body"] = {"data" : modem_template}

    rsp = card.Transaction(req)
    print(rsp)

then in my loop, I am attempting to concatenate 4 records with:

    data_to_store = {key:cont[key] for key in ['seq','pressure','runcycles','faults','mode','temp']}

    modem.append(data_to_store)
    
    if cont['seq'] == 3:
        req = {"req" :"note.add"}
        req["file"] = "vstdata.qo"
        req["sync"] = True
        req["body"] = {"data" : modem}
        rsp = card.Transaction(req)

        print(rsp)
        pprint(modem)
        modem = []

        cont['seq'] = 0
    else:
        cont['seq'] = cont['seq'] + 1

I get the correct data:

[{'faults': 196,
  'mode': 0,
  'pressure': -0.14,
  'runcycles': 20108,
  'seq': 1,
  'temp': 0.0},
 {'faults': 196,
  'mode': 0,
  'pressure': -0.14,
  'runcycles': 20108,
  'seq': 2,
  'temp': 0.0},
 {'faults': 196,
  'mode': 0,
  'pressure': -0.14,
  'runcycles': 20108,
  'seq': 3,
  'temp': 0.0}]

but when I print(rsp), I get:

{'err': 'error adding note: object expected because of template'}

I also tried just setting my template to:

req["body"] = modem_template

and got:

{'err': 'error adding note: body fields not found in template: data'}

as my error.

I feel i’m close, but 'm just spinning my wheels at this point.

[vstadmin]

Incidentally, my code DOES work without the template. I would love to get it working with the template if that will save on data or bandwidth.

[RobLauer]

Editing this response to refer folks to @gwolff’s response below.

Note templates actually aren’t appropriate for arrays of data (since the structure of the array is really unknown…you could send 1 element or 100). I would stick with the original method of adding notes without using a template.

[vstadmin]

Ok, I was just following earlier concepts and would really like to remove any redundancy. I guess I can shrivel by fieldnames down to a single character.

[gwolff]

I disagree on avoiding the templates in this case.

My concern has to do with the nuance of how templated Notefiles are handled in the sync process versus untemplated Notefile Notes. Untemplated Notes use more RAM in the sync process, as they must be serialized before the sync. For large Notes this could be a problem.

Templates permit JSON arrays, but they must be fixed length. However, they are compressed and sent as a binary blob, and then serialized back to JSON on the server, saving Notecard resources.

I will follow up with a template example.