Notecard: I2C error, not recoverable + can't connect: stuck in extended-network-failure

Troubleshooting
1

Hi,

I cannot connect with notecard. Updated to latest firmware v3. Stuck in extended-network-failure, with increasing delays:

TRACE hub.sync_status: SyncStatus { status: Some("network: can't connect (11.3 min remaining) {registration-failure}{network}{extended-network-failure}"), time: None, sync: Some(true), completed: Some(1), requested: None }

have the output of card.wireless as well.

The same location has worked flawlessly with good signal before.

(I’ve tried to power-cycle, reset, and software-reset)

Best regards, Gaute

2

Hey @gauteh can you capture a trace log using the steps described here so that we can take a deeper look?

3

Think the link may be wrong? But using the ftdi adapter? Don’t have access before Monday, will do then if issue still persists. If USB is ok, i can try later.

lør. 19. feb. 2022, 15:37 skrev Brandon Satrom via Blues Wireless Community <blues@discoursemail.com>:

4

Sorry about that! Try this one, and here for more information.

5

The issue is gone now when I tried again with tracing over USB. I just came across another issue though: the MCU lost connection over I2C to the modem, and no matter what the MCU did / restarting etc, nothing happend. Just could not get any response from the notecard (i2c NAK). I could not see anything in the trace output (not sure when it happened), but the notecard looked like it was doing its thing, trying to get a GPS fix and occasionally connecting.

Resetting the modem fixed the issue immediately.

Edit, seems to have happened again, trace:

G17:30:15.73 GPS search (21 sec, 33/33 dB SNR, 0/6 sats, HDOP 0)
R17:30:19.19 bulk: created file:data/axl-qo.001
G17:30:20.73 GPS search (26 sec, 33/34 dB SNR, 0/4 sats, HDOP 0)
G17:30:25.73 GPS search (31 sec, 34/34 dB SNR, 0/4 sats, HDOP 0)
G17:30:30.73 GPS search (36 sec, 33/34 dB SNR, 0/7 sats, HDOP 0)
G17:30:35.73 GPS search (41 sec, 34/34 dB SNR, 0/7 sats, HDOP 0)
G17:30:40.73 GPS search (46 sec, 34/35 dB SNR, 0/7 sats, HDOP 0)
G17:30:45.73 GPS search (51 sec, 34/35 dB SNR, 0/8 sats, HDOP 0)
G17:30:50.74 GPS search (56 sec, 36/36 dB SNR, 0/8 sats, HDOP 0)
G17:30:55.73 GPS search (61 sec, 36/36 dB SNR, 0/7 sats, HDOP 0)
R17:30:59.05 bulk: created file:data/axl-qo.002
G17:31:00.74 GPS search (66 sec, 34/36 dB SNR, 0/9 sats, HDOP 0)
G17:31:05.74 GPS search (71 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:10.74 GPS search (76 sec, 37/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:15.74 GPS search (81 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:20.74 GPS search (86 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:25.74 GPS search (91 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:30.74 GPS search (96 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:36.74 GPS search (102 sec, 37/37 dB SNR, 0/9 sats, HDOP 0)
R17:31:38.82 bulk: created file:data/axl-qo.003
G17:31:41.74 GPS search (107 sec, 37/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:46.74 GPS search (112 sec, 35/37 dB SNR, 0/9 sats, HDOP 0)
G17:31:51.74 GPS search (117 sec, 37/37 dB SNR, 0/8 sats, HDOP 0)
G17:31:56.74 GPS search (122 sec, 37/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:01.74 GPS search (127 sec, 37/37 dB SNR, 0/10 sats, HDOP 0)
G17:32:06.74 GPS search (132 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:32:11.74 GPS search (137 sec, 36/37 dB SNR, 0/9 sats, HDOP 0)
G17:32:16.74 GPS search (142 sec, 35/37 dB SNR, 0/9 sats, HDOP 0)
R17:32:18.86 bulk: created file:data/axl-qo.004
G17:32:22.74 GPS search (148 sec, 34/37 dB SNR, 0/5 sats, HDOP 0)
G17:32:27.74 GPS search (153 sec, 36/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:32.74 GPS search (158 sec, 35/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:37.74 GPS search (163 sec, 36/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:42.74 GPS search (168 sec, 35/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:47.74 GPS search (173 sec, 36/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:52.74 GPS search (178 sec, 35/37 dB SNR, 0/8 sats, HDOP 0)
G17:32:57.74 GPS search (183 sec, 35/37 dB SNR, 0/5 sats, HDOP 0)
R17:32:59.62 bulk: created file:data/axl-qo.005
G17:33:02.74 GPS search (188 sec, 35/37 dB SNR, 0/7 sats, HDOP 0)
G17:33:07.74 GPS search (193 sec, 34/37 dB SNR, 0/7 sats, HDOP 0)
G17:33:12.74 GPS search (198 sec, 30/37 dB SNR, 0/6 sats, HDOP 0)
G17:33:17.74 GPS search (203 sec, 33/37 dB SNR, 0/6 sats, HDOP 0)
G17:33:22.75 GPS search (208 sec, 32/37 dB SNR, 0/12 sats, HDOP 0)
G17:33:27.75 GPS search (213 sec, 32/37 dB SNR, 0/12 sats, HDOP 0)
G17:33:32.75 GPS search (218 sec, 32/37 dB SNR, 0/13 sats, HDOP 0)
G17:33:37.75 GPS search (223 sec, 32/37 dB SNR, 0/13 sats, HDOP 0)
G17:33:42.75 GPS search (228 sec, 32/37 dB SNR, 0/9 sats, HDOP 0)
G17:33:48.73 GPS search (234 sec, 33/37 dB SNR, 0/15 sats, HDOP 0)
G17:33:54.73 GPS search (240 sec, 33/37 dB SNR, 0/15 sats, HDOP 0)
G17:33:59.74 GPS search (245 sec, 28/37 dB SNR, 0/5 sats, HDOP 0)
G17:34:04.74 GPS search (250 sec, 30/37 dB SNR, 0/11 sats, HDOP 0)
G17:34:10.74 GPS search (256 sec, 28/37 dB SNR, 0/12 sats, HDOP 0)
G17:34:16.73 GPS search (262 sec, 28/37 dB SNR, 0/12 sats, HDOP 0)
G17:34:22.73 GPS search (268 sec, 28/37 dB SNR, 0/12 sats, HDOP 0)
G17:34:27.73 GPS search (273 sec, 29/37 dB SNR, 0/6 sats, HDOP 0)
G17:34:32.74 GPS search (278 sec, 29/37 dB SNR, 0/8 sats, HDOP 0)
G17:34:37.75 GPS search (283 sec, 28/37 dB SNR, 0/9 sats, HDOP 0)

a reset fixed it (pushing the reset button on sparkfun carrier board).

6

I also sometimes get bytes sent when querying for data (sent = 0xff), never seen this before v3.

7

The I2C issue keeps happening, it seems that two things are happening around the same time:

  • GPS is getting a connection / disconnection
  • Cellular is changing towers

notecard is in exact same position. I do not see any messages about brown-out or not enough power.

Only way to re-establish I2C is to hard-reset the notecard.

It seems that in these situations it manages to connect and sync a few notes, then connection is lost. None of these observations cause a consistent I2C failure, so I don’t know if they are related.

I don’t have access to FTDI cable yet.

8

Hey @gauteh thanks for providing this additional context. I think, at this point, someone on our end needs to repro your setup to determine the issue, which might be hardware configuration-related, or an I2C issue in your library or with the host. Can you detail your exact setup so that we can try to replicate it?

1 Like
9

Thanks. I’ll write something later (the source and schematics are here: GitHub - gauteh/sfy: low-cost cellular drifter). Note that I’ve set the card up to do periodic syncs every 10 minutes, and these still happen (but it’s only sending _track.qo, not any data from my device). I’ve hooked up the host using a debugger and I can see the error happening in the I2C library (NAK).

1 Like
10

The setup is as follows:

  • Initialize driver with handshake
  • Call hub.set periodic sync 10 minutes
  • Set card.location in continuous mode, start = true, sync = true
  • Set up a template
  • Do initial sync

then I read location, time, storage, status about every second, with at least 25 ms between operations. I poll using 25 ms delays.

Data is sampled and queued into the templated queue, one package about every 20 seconds. When the storage is above 50% a sync is initiated, usually the 10 minute periodic sync comes first.

The puzzling part is that I completely reset the host, try to reset the I2C module. But nothing helps. As soon as I reset the notecard I2C from the host works again.

Notecard is powered over 2x3 D-cells. As I mentioned it still manages to connect and sync _track every 10 minutes, so power should be sufficient.

It can function for hours and hours, and then get into this state where it works for maybe three minutes, or 20 minutes before going out. Right now it is failing.

I’d be happy to do a video-call or something to troubleshoot any further.

Regards, Gaute

11

Hey @gauteh if resetting the host doesn’t work and cycling the Notecard is the only thing that helps, I’m inclined to think that something in the I2C transaction between the host and Notecard isn’t getting cleaned-up and causing the bus to get in a bad state over time. I have zero familiarity with Rust, but did you write that library using what we’ve done in note-c and note-python as inspiration? If not, I would suggest taking a look and making sure you’re using a similar pattern of timings, delays, and message segments.

1 Like
12

Once it gets into this state even pinging over i2c doesn’t work. It’s just nak. So no bytes are transferred successfully at all.

man. 21. feb. 2022, 23:41 skrev Brandon Satrom via Blues Wireless Community <blues@discoursemail.com>:

13

Small update on this one, and possibly related: Base64 format details in note.add - #5 by gauteh. I changed GPS mode from continuous to periodic and set heartbeat to 1 hour. The notecard is sitting on the desk in a basement with no (or almost never) GPS signal, and usually no motion. This caused the I2C disconnect to happen significantly more seldom, but not exactly once every hour it seems.

Now testing with GPS off.

1 Like
14

Hi,

I have now tested with the notecarrier B. With 2.2k pull-ups on the i2c bus, and a desktop-power supply. I still get this error where it goes into non-recoverable state for i2c-connection. Resetting the host MCU does not work. In this case it seems that once the notecard was done syncing and transmitting notes to the hub the i2c-connection was possible to re-establish.

Regards, Gaute

15

Also occasionally happens with power from USB on a Notecarrier B (and different USB to MCU). Seems to happen during sync. The CHG led of my MCU board lights up.

Connections Notecarrier B:

SDA (tested both with and without 2.2k pull-ups)
SCL (tested both with and without 2.2k pull-ups)
GND (MCU ground)
USB to computer
VIO to 3v3 on MCU
MCU has USB to computer

16

Hello @gauteh,

I have most of the same parts as you, and I’m going to see if I can reproduce your issue today. I’ll leave notes in this thread if I have any questions about your setup.

Best,
Zak

1 Like
17

@gauteh

i’ve been reading back through the whole post and I’ve looked at the hardware folder in your repository. It sounds like you’ve tried several configurations of hardware. Also, you didn’t mention parts used for your cellular and GPS antennas.

Can you share a picture of your most recent setup? I would like for our hardware setup to be as similar as possible, so I can ensure I experience the same behavior as you.

Also, regarding your battery notation above; “2x3 D-cells”. So you are saying you have 2 sets (parallel) of 3 D-cells in series, right? Meaning the configuration should generate ~4.5V and TONS of current. I presume the battery array feeds the regulator, but how does the output of the regulator connect to the rest of the hardware?

In a picture can you make clear where the power comes into your solution especially the connections to the MCU and Notecarrier.

I’ve been rummaging through my bins, and I have the Sparkfun Qwiic Notecarrier, the Pololu regulator, a Sparkfun Artemis Plus (not Nano), and unfortunately I don’t have the Adafruit accelerometer that you have.

Once I have a little more information, then I will try to recreate exactly what you have on your bench.

Cheers,
Zak

2 Likes
18

Hi Zak,

thank you very much for looking into this. As you see I have tried out multiple configurations.

Today I started with fresh D-cells to test the issue. I started with 6xD-cells (9V), but that proved very unstable. And I decided to try again with the 2x3 D-cells (4.5V) as you describe, this works for a while and then errors occasionally. I disconnected the VIO <-> MCU 3V3 as described above and it seems to work slightly better, but I don’t know. I also disconnected the pull-ups to see if that made a difference, but it (maybe) worked better with those.

The setup I am now attaching photos from has worked pretty well for about 4 to 5 hours before I noticed that the MCU was locked up in its reboot / recovery cycle. It tries to restart the notecard (which wont work because I2C is not working), eventually it restarts itself and waits several seconds before trying to reinitialize peripherals.

The Pololu regulator (3V3 stepup/stepdown) only powers the MCU(3V3 pin) and IMU (STEVAL LSM6DSOX), while the batteries are directly connected to the Notecarrier V+ (On the 9V setup, I used a 5V Pololu regulator rated to 3A and used it to power both the modem (V+ and MCU Vin)).

The issue just happened (but somewhat more sporadic) on the 4.5V setup, so lets use that as a basis.

Notecard was updated to latest firmware yesterday.

I am traveling and only brought a minimal setup (including forgetting my laptops power adapter), but I would happy to explain in more detail in a quick call if that helps.

Now that the system in a stuck state I can hear the modem audible buzzing, not sure which component is doing that… but it doesn’t sound right :smiley: . After resetting (connecting yellow wire temporarily to RST on notecarrier) sound goes away and MCU manages to reboot and reinitialize.

20220304_193750
20220304_1937501920×2560 399 KB

20220304_193953
20220304_1939531920×2560 310 KB

20220304_193851
20220304_1938511920×2560 560 KB

20220304_193944
20220304_1939441920×2560 459 KB

19

@gauteh,

Thanks for the pictures. They will really help me get going!

Before I start wiring it up, can we try powering the Notecarrier from the regulator instead of the batteries directly? I’m concerned the voltage difference between the Notecarrier (~4.5V) and the MCU (3.3V) may cause issues when evaluating logic levels. The Notecarrier-B should work from the regulator without issue.

Thanks,
Zak

1 Like
20

Could you also paste the results of {"req":"card.version"}?

I want to make sure I’m using the exact same Notecard as you are.

Here’s mine for reference…

> {"req":"card.version"}
{
 "body": {
  "org": "Blues Wireless",
  "product": "Notecard",
  "version": "notecard-3.2.1",
  "ver_major": 3,
  "ver_minor": 2,
  "ver_patch": 1,
  "ver_build": 13982,
  "built": "Feb 3 2022 15:02:36"
 },
 "version": "notecard-3.2.1.13982",
 "device": "dev:864475044196755",
 "name": "Blues Wireless Notecard",
 "sku": "NOTE-NBGL-500",
 "board": "1.11",
 "api": 3
}