This is the end

This is the end, beautiful friend

This is the end, my only friend, the end

Of our elaborate plans, the end

Of everything that stands, the end

No safety or surprise, the end

I’ll never look into your eyes, again


Good night to everybody! 🙂

Day 51: DropFS and a big fear


Before the week-end and on Monday, I worked on DropFS, our flash memory management system for drop messages.

First, I had to establish precisely the byte (even bit!) mapping for each type of page in memory. I did it with packed structures and bitfields in order to simplify parsing. Consequently, I spent a few days on writing and rewriting lots of typedefs and function prototypes, because our model changed everyday and because I wanted data structures as easy to manipulate as possible. And when we discovered that our flash could be written bit per bit, I once again had to rethink all my structures! But now I think that the result is very near to its final state.

Then on Monday, I was able to begin programming DropFS’s actual functions. Right now the basis (creating, reading and deleting indexes, boxes and messages) is done, I only have to finish the write and edit functions for tonight. Actually, I didn’t have much time to work on DropFS today because we received our first drops and discovered that their radio wasn’t working! Hopefully, we solved all the problems we had (with a big help from Alexis and Sam!) and now each of us has a working drop in their pocket. To summarize, we were unlucky on several points: the nRF we received was a C0 revision instead of G0 (with a lot more bugs and differences for PCB design) and some components weren’t delivered in time, so we had to use similar but inadapted quartz, capacities and resistances for the first prototypes. With this special combination of changes, everything went wrong! We were in a configuration where our SoftDevice and SDK versions weren’t supported, the HFCLK was supported by the chip but not the radio… In the mean time, our git repository went half-crashed… For a few hours, we thought that maybe none of this year’s PCBs could work with BLE. So having them working tonight was quite a relief.

Solving these issues nevertheless took us the whole day and we weren’t able to do anything else. So I can’t wait for testing DropFS on an actual drop!

Day 44: hints on Android API and BLE

Since last week (after having rewritten our protocol with a single characteristic), I struggled a lot with Android API for GATT connections. To help you to establish clean connections, here is what I have found.

A BluetoothGatt object is a kind of socket. You open it, then you connect and disconnect it as many times as you want, and then you close it.

In the Android API, the connection occurs as soon as the GATT is opened: it is the purpose of the BluetoothDevice.connectGatt method. BluetoothGatt.connect is only used for reconnections after a disconnection. On the contrary, the disconnect and close operations shouldn’t be done at the same time. Closing a GATT means forgetting it (it would be done by the Garbage Collector, but it is better to do it as soon as you don’t need it anymore, so that ressources can be freed), so what will happen if you receive a “onDisconnected” event for a GATT that you have already closed? A NullPointerException is thrown (see the Android logcat). Then BluetoothGatt.close must be used only after having received the confirmation that the GATT is disconnected (typically, in the onDisconnected callback).

Please note that the S110 SoftDevice (which we use on the nRF51822 chip) only accepts one GATT connection at a time, but it is only a matter of implementation, and not a standard behaviour with Bluetooth Low Energy. Consequently, your phone is able to open and maintain several GATT connections in the same time. And these connections may be established with the same BLE device (even a nRF51822). Technically, what you will have are several BluetoothGatt objects with different UUIDs (each connectGatt creates a new instance). But this is a bad behaviour, and if you maintain several BluetoothGatt objects towards the same nRF51822, after a few iterations you won’t be able to connect anymore. So don’t forget to close GATTs you won’t reconnect with, and keep track of already created GATTs so that you can reconnect them instead of creating new instances.

By the way, the life time of an Android application is not the life time of  your main activity. Even when the application is not visible anymore in the “Recent apps” panel, it may survive, and the only way to kill it properly (without debug options) is to “Force quit” it in the Application options. Consequently, be cautious when recreating a GATT object if you are not sure that the previous one was destroyed (which happens systematically when the app is killed).

A GATT connection must not be confused with pairing and bonding. Here is an extract of nRF51 documentation:

Bonding is when two devices that connect and follow procedures that allow them to cache information inorder to avoid repeating certain set-up procedures on subsequent reconnection(s).

The caching of information can span across layers in the system, for example, keys needed to secure the link on reconnection, GATT Server Configuration. Information with respect to a bonded peer is expected to be retained on power cycle of the device, and is maintained persistently.

In contrast to bonding, the Bluetooth specification defines procedure needed to establish secure link for ongoing session only, called Pairing. No information is cached for paired devices. Please refer to Volume 3 Part H, Section 6.5 of Bluetooth Core Specification 4.0 for more information.

To put it in a nutshell:

– GATT connections/disconnections are short term, when you have a few pieces of data to send.

– GATT objects are middle term, typically for the life time of your Android application.

– Pairing is for establishing secure connections.

– Bonding is long term, when you known your devices will often interact.

I hope I was clear enough, don’t hesitate to ask me if you have questions left, or to tell me if I was wrong on some point.

Day 35: putting everything together

Hello everybody!

Since my last post, I did a lot of different things.

First, on Android:

– I completed the Drop Scanner app, with four activities dedicated to scanning for BLE devices, displaying services and characteristics, reading, writing on characteristics and asking for notifications. It is already sufficient for reading and writing messages! But the UI is not very user-friendly.

– I added three extra activities on the app to provide a nicer UI for the mailbox. It does not work yet (the notification request on Content is sent, the write request on ID is sent… and the write ACK never comes back) and I don’t understand why, because the message arrives with no problem when I perform these steps by hand with the GATT UI. Maybe there are some timing issues. But I stopped working on it, because we are going to change our protocol for reading messages very soon anyway.

– I discovered the Android version easter egg, thanks to Sam.

– I spent some time with Adele and with Matthieu to help them setting up their environment for Android+Scala+Scaloid. I also did a quick crash course on Scala for Adele, who wasn’t familiar at all with functional programming.

– I know that there are still a lot of issues with this app: bad GATT connection management (at which abstraction level do I have to manage it? how long should it be?), awful design… But right now it is functional enough for us to test any interaction we want with the nRF51822, so I stopped working on it and wait for the others to get familiar with Scaloid.

And for the nRF51822 part:

– I began truly reading the code Matthieu and Adèle have been working on for the nRF51822. With the help of git grep, I understood its architecture, the meaning of the most useful structures, and now I have a good idea of where I should change the code to add features.

– Considering that there was too much copy-paste in the code (which started as a customization of the led button demo), I worked on refactoring it. Tonight, there are no more duplicate lines in the services and characteristics initialization process.

– Now I start working on a new BLE architecture, with only one characteristic for reading and writing messages.

See you soon!

Day 24: unblocked, at last!

During the week-end, I established BLE communication between a nRF51822 chip flashed by Matthieu and my smartphone (with Scaloid), but I was constrained by the UI: the only display I was able to do was toasts.

So, at the beginning of the week, I decided to explore Android’s View and Adapter. It began well… and then I was stuck for two days on a LayoutParams cast error! After having read lots of source files of Android API and Scaloid, I finally understood what was happening: an AbsListView checks the LayoutParams type of its children, but only when ids are stable (and for an ArrayAdapter, they aren’t). This behaviour is quite strange, I am not sure if it was on purpose, it may be a bug in the API.

I was quite angry at Android for a moment, but I was relieved to have solved my bug, as well.

Then when Sam informed me that my app was flooding his phone, replicating the same device hundreds of times, I was confused, because it only appeared once on mine. I did experiments with all the phones available in the room (Nexus 4/5, Android 4.3/4.4/4.4.2, with or without CyanogenMod), and discovered that I had bought the wrong phone: for hardware reasons, the Nexus 4 receives 100 times less BLE advertising packets than the Nexus 5. Sad, isn’t it?

I lost a lot of time with these two issues, but now I can finally go on with my app development and start implementing services for drop messages. And even if I was a little angry, it was really interesting to solve my problems, since it allowed me to discover a lot more on Android.

Day 20: a nice morning

This morning… I slept!

For the first time in a few days, I had some time to rest, which I enjoyed a lot. Since my last post, I have worked a lot on my STM32 for the lab and the communication challenge. We went on until late yesterday (the door closed at 23:30…), but I did it! After exploring non-deterministic bugs, a lot of documentation, and with the help of my best friend Google, I finally reached the final step. There are so many reasons why I could hate myself yesterday (it’s amazing how many stupid mistakes one can make when working under pressure…), but overall it was a very good experience. I am quite proud of what we achieved (basically, audio streaming) and the whole principle of discovering our board features in an ouverture-facile manner was very fun. This challenge was really a platform game! So, thank you, everybody: Alexis and Sam for organizing it, the students for having suffered with me and the alumni for passing by and encouraging us (the cookies were delicious!).

Therefore, I allowed myself to take some rest as a reward. But as soon as the morning was over, I was back in A406! No rest for the brave. This afternoon, Matthieu and I worked again on establishing a first communication between the BLE kit and an Android phone. While he was working on the kit, I managed the Android part. We agreed to develop the apps in Scala (with the scaloid library), which is much more compact and flexible than Java. But first, I had to remember how to program in Scala! So I spent some time reading documentation on Scala, Android, Scaloid, installing Android SDK, Sbt, and so on. I have now a DropScanner app on my Nexus 4… which does nothing yet, except displaying a text and a button. But tomorrow I will try using the BLE API for scanning.

By the way, be careful if you plan on testing BLE apps with a Nexus 4, since before Android 4.4 it couldn’t unbound Bluetooth Smart devices. Of course, I had not upgraded my Android yet…

Day 17: the end is near!

Since yesterday, I worked a lot on my STM32 lab. I have now begun the TCP/IP part, and all the other parts are done! The end is near, I am now more confident in my ability to finish the lab before the communication challenge.

During these two days, I also worked a little on the nRF51822’s BLE stack, by reading parts of the API documentation. I haven’t seen everything yet, but I slowly begin to understand how the stack works. Hopefully, I will understand the whole of it in time to complete our PSSC (programming a scanner app for an advertising drop) before Sunday!

Day 14: so much to do, so little time…

I have a loooot of things to tell you tonight. Are you ready? Let’s go!

On Friday, Matthieu, Adèle and I spent a lot of time reading several datasheets to compare different SoCs. In the end, only the nRF51822 and the CSR101x series seemed flawless… until we discovered that the CSR101x was really difficult to get. As a result, we definitely chose the nRF51822. The timing was good, since we received our developement kits on the very same day! We also worked on the choice of other components, in particular the button cells, with Adèle spending a lot of time on Farnell. Having filled up a table of the characteristics of all components, we were able to choose. For the battery, we selected a few of them (CR2032 and CR3032, rechargeable and not rechargeable), but we still needed some information on the support availability and actual prices for big ordering.

Setting up a comfortable environment

I invested my Saturday to ensure a better productivity later: I configured my development environment. More precisely, I set up a .screenrc file which automatically opens windows in the right directory, for all the tasks I do when developing for embedded systems and runs the appropriate commands (openocd, arm-none-eabi-gdb ch.elf, git status, emacs -nw main.c, and so on) at startup, with appropriate key bindings in addition. It was a bit difficult at first to understand how to write this file, but now it is done, and I am very happy of the result! When I start my computer, all I have to do now is to plug my STM32, create (or attach) a screen… and program! It may not seem like such a big change, but I know that I was always spending much time opening, renaming and closing tabs, remembering where my openocd.cfg and .gdbinit files were, opening lots of buffers in emacs… And now I won’t have to anymore. Of course my .screenrc is specific to my directories and my habits, but I can help you setting your own environment if you want!

I also spent an important part of my weekend on my STM32 lab, and I have now working semaphores, threads and events!

Today, I worked with Matthieu and Adèle to complete today’s PSSCs: the choice of components and the BLE architecture for drops.

While Matthieu was crying in front of complex schemes of solar panel connectors and Adèle was losing hope to find a CR3032 support, I was busy searching for rechargeable batteries cheaper than button cells and for application notes about impedance matching in datasheets. However, I found neither of them. I guess that we will keep our first idea of rechargeable button cells!

BLE architecture

Afterwards, we considered our BLE architecture. I read the SoftDevice specification and Android API documentation for BLE. Then the three of us discussed about architecture questions: what is the best granularity level, from a consumption point of view? How to ensure atomicity for read and write operations? We designed four architectures :

  • A single service, very generic, with characteristics “OperationID”, “Input”, “Output”. The current operation (read, write, edit, list boxes…) is selected by the “OperationID” characteristics and the semantics of the other characteristics depends on the operation.
  • A single service Message (characteristics “id” and “content”, when the client changes one value, the server updates the other one accordingly) and a service Box to select the appropriate box for current operations.
  • One service per operation, and appropriate characteristics for each one of them. For instance, service Read would have characteristics BoxId (write only), MsgId (write only), MsgContent (read only). In this architecture, there can be a service Authentification, used before any other operation, typically for the treasure hunt.
  • One service per operation and per box. It increases a lot the number of services, but reduces the number of characteristics by 1 per service.

The first architecture is conceptually the easiest, but probably the hardest to implement. The second one may create concurrence issues when several devices are connected. For the fourth one, the risk is to have too much services and exchange a lot of messages just to give the list of services. But in the third case, the problem is the same, except that it is moved to the level of sending the list of boxes.

There is still a problem of atomicity. With the second architecture for instance, when the client reads the message 136 and then writes the message 137, it first writes id=136 (the server updates content=foo), then reads content=foo, then writes id=137 and content=bar, but in which order? The server must wait for both characteristics to be updated or the box will be corrupted. Possible solutions are to add an extra boolean characteristics that could act as a mutex, or to require a short delay (how much?) after each update before processing.

Right now, we don’t understand BLE well enough to decide what the best architecture is. We need to start developing an Android app with BLE API to know what is easy or not to implement and how much messages it represents.

Another idea: maybe we could use characteristic descriptors (for instance the MsgId as a descriptor for MsgContent), but we are not sure yet of what changes between attributes and descriptors, concerning notifications, updates and number of messages. Are descriptors really the same as attributes, but only at a different level, with different semantics?


PHEW! That was already a lot of work for a few days. I will now spend the rest of the day working on ADC and reading documentation for Android development. Then I will rest a little, because next week will be very dense: we must finish our STM32 lab, design the PCBs, learn how to use the nRF51822 and how to develop Android apps… and develop our first app!

Day 10: hardware brainstorming

Sorry, this post won’t contain awesome pictures of Pokeballs like PLUME! This afternoon was more hardware searching than 3D printing for Drops N’ Roses. We spent a long time talking with Alexis, Sam and Tarik about our memory needs and their cost. FRAM seemed quite interesting, but it is very expensive compared to our needs. We computed life expectancies based on consumption in standby mode and concluded that we would probably use an extern flash memory of 16 or 64 Mb (ensuring a 13 year autonomy with a button cell… if nobody reads nor writes anything, of course!), for a very low price: around $3!

Another very fun idea came to us this afternoon: considering that some drops will be in the outside or behind a window and considering the pins we wanted to add for connecting any external device… why couldn’t we connect a pocket calculator solar panel? With that, a (rechargeable) button cell could really last several years! Sadly, it won’t work for the underground. But it isn’t a bad idea either, and we will think about it.

Day 9: community managing

After a very busy week-end working on the PSSCs and Monday’s presentation with Matthieu and Adèle, I had not much time for Drops N’ Roses.

But today, I began acting as a community manager for our group! I put some content on our hackster page and changed our team name for something more appropriate (I was amazed to see that Matthieu had not changed this awful ‘blyste_and_lau’ team name yet!). Be careful, the address has changed, so don’t use your old bookmark if you had one, follow the link on the ROSE 2014 group instead.

I also began working on a wiki page for a community of underground explorers. They were very enthusiastic with our project and they can’t wait testing our devices! I hope they won’t be disappointed…

Besides, this afternoon, I finally began progressing on my lab on STM32. Now the JTAG is working, hurray! I expect a lot of fun with my new STM32 best friend now.