M5Core2 library fork that supports multi-touch and provides Arduino-style virtual buttons [update: and gestures and events] [update2: MERGED !]

felmue

hmm, I think that is a tricky one. I tried a few things but have not found a solution.

I believe the touch IC constantly monitors the electrostatic field and automatically adapts to slow changes to get a 'baseline'. And only big changes in a short period of time will be taken into account for touch.

So when the finger already is on the screen when the device boots that is simply taken as baseline and only lifting and touching the screen again is a big enough change to count as touch.

Thanks
Felix

Rop

@vkichline Booting with the screen pressed. I had not thought of that... It will be fixed.

vkichline

I find that if you boot with the screen touched, M5.Touch.ft6336(0x02, 11, data) returns nothing but 11 zeros until you release-retouch the screen.
You get the same stream of zeros if you boot without touching the screen, until it is touched.
Frankly I doubt if there's a solution. The screen knows knows no "untouched state", it has to figure it out each time its booted. But perhaps some arcane stream of I2C commands con compensate.

I think the only "special boot condition" one might detect with the Core2 will be positional; the device is upside down, or standing on one particular edge. The presence/absence of an SD card could be used as well, but then you'd be much more likely to encounter unintentional special boot conditions.

Or perhaps a special condition could be detected just before reboot and written to NVS.

Rop

@vkichline Ah, OK. So it's not only my state-machine... Hmmm... I like the upside down idea, that would be pretty untypical in the real world.

BTW: I made a new version that does pretty things on the screen. Check out the example with the 'visual' branch of my fork. Please test. I think you'll like it. Will document it soon and submit another PR.

Rop

@vkichline Hey, there's also a RTC. So you could check whether a second user reset happens soon after. That's pretty intuitive.

vkichline

@rop said:

Check out the example with the 'visual' branch of my fork. Please test. I think you'll like it.

I do like it! The results the touch demo gets from a single page of code are remarkable!
I do have some feedback though. (If you ask me to test something, you will always get bug reports.)

Could you enable Issues for https://github.com/ropg/M5Core2/tree/visual? Better to keep a discussion issues on that branch with the repo.

vkichline

@rop,
In TouchButton::pressedFor(uint32_t ms),
_time - _lastChange is always equal to zero. pressedFor(n) always returns false.
It looks like in some of these TouchButton routines, _time should be replaced with millis().

Rop

I did some work the past few days. :)

First of all my stuff is now three parts. M5Touch provides the Touch library and nothing else. It will talk to M5Button (if that's present') which provides Buttons, Gestures and Events. PointAndZone provides the Point and Zone primitives that they both depend on. M5Button's Button also does hardware buttons that can have a zone on the screen to draw their buttons in and it can be used independently from M5Touch. Yes, of course that means Button and the whole Events thing are coming to the M5Stack classic.

Zones (and thus Buttons) have a rot1 flag that makes them stay in rotation 1, meaning that they stay in the same place even if you rotate the screen. To see that demoed, compile the example app, hold the device in the desired direction and swipe from what's then the top to the bottom of the screen. Also try pressing the off-screen circles in different rotations. (Yes, when the screen is upside down they have negative y coords... :)

Much has changed internally and I will document everything painstakingly over the next days. It is now much easier to write code for the library itself because objects have an instance pointer so you can reference the screen from some other object without #including the whole M5Core2.h kitchen sink or doing other ugly things. All my changes to existing files such as M5Display compile on both classic and Core2: all the changes are behind #ifdefs, #define in Config.h.

Everything is super-efficient, such that Button's visual capabilities are not in the way if you're not using them, etc etc.

As for existing code:

M5.Touch has been split up: some functionality (addHandler, etc) now lives in M5.Event
TouchPoint -> Point, TouchZone -> Zone, TouchButton -> Button, TouchEvent -> Event
Event names start with E_ not TE_

It's all in the visual branch on my fork

@vkichline The pressedFor bug is also fixed...

felmue

Hi @Rop

I am trying to dynamically create (and delete) a button, but I get a crash when the button is about to be deleted. (When I use a static button it works.)

The code creates a MsgBox and upon pressing button B a new empty screen with a fullscreen button is opened and when I touch it I get the crash.

Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited). Exception was unhandled.

I am sure I am overlooking something so if you have a minute I'd appreciate your input.

Thanks
Felix

#include <M5ez.h>

#define STATIC_BUTTON

void buttonCallback(Event& e);

bool page_done = false;

#ifdef STATIC_BUTTON
Button button1(0, 0, 320, 240, false ,"button1");

void showPage() {
  button1.addHandler(buttonCallback, E_TOUCH + E_BTNONLY);
  page_done = false;
  while(page_done == false) {
    M5.Touch.update();
  }
  button1.delHandlers(buttonCallback);
}

#else

Button *button1 = NULL;

void showPage() {
  button1 = new Button(0, 0, 320, 240, false, "button1");
  button1->addHandler(buttonCallback, E_TOUCH + E_BTNONLY);
  page_done = false;
  while(page_done == false) {
    M5.Touch.update();
  }
  button1->delHandlers(buttonCallback);
  delete(button1);
}
#endif

void buttonCallback(Event& e) {
  page_done = true;
}

void setup() {
  ez.begin();
}

void loop() {
  ez.msgBox("M5ez button test", "Button test !", "showPage");
  showPage();
}

Rop

I've worked extensively on the events system, so that there is now only one event per call to M5.update(). That means you can now also check for events in loops, no need for handler functions if you don't want them. Get the new library and try this:

#include <M5Core2.h>

ButtonColors on = {RED, WHITE, WHITE};
ButtonColors off = {BLACK, WHITE, WHITE};

void setup() {
  M5.begin();
  while (true) {
    screen1();
    screen2();
  }
}

// Never comes here
void loop() {
	M5.update();
}

void screen1() {
  M5.Lcd.clearDisplay(YELLOW);
  Button tl(5, 5, 150, 110, false ,"top-left", off, on, TL_DATUM);
  while (true) {
    M5.update();
    if (tl.isPressed()) return;
  }
}


void screen2() {
  M5.Lcd.clearDisplay(BLUE);
  Button tr(165, 5, 150, 110, false, "top-right", off, on, TR_DATUM);
  while (true) {
    M5.update();
    if (tr.event.type == E_DBLTAP) return;
  }
}

As you can see you don't need to orry about manually deleting the buttons, they go away as the function ends. But you cannot call one function from the other, because that leaves the variables in existence. Then you'd have to 'new' and 'delete' them indeed.

felmue

Hello @Rop

Thank you for the update. Unfortunately no luck for me. When I run your code with the latest M5Core2 from your visual branch I get a crash as soon as the code tries to instantiate the button in screen1() function.

When I declare the buttons globally the code works, but as you mentioned they don't go away.

I wonder if the button object is too large to be declared in a function?

Thanks
Felix

felmue

Hi @Rop

Update2: I've created a pull-request with below fixes for your convenience.

https://github.com/ropg/M5Core2/pull/2

I have found the reason for the crash: drawFn is not NULL, yet not set correctly which causes the crash when calling drawFn(this, bc) on line 212.

void Button::draw(ButtonColors bc) {
  // use locally set draw function if aplicable, global one otherwise
  if (drawFn) {
    drawFn(this, bc);

I've fixed it by modifying two lines (137, 157) in M5Button.h and initialising drawFn with nullptr.

void (*drawFn)(Button* b, ButtonColors bc) = nullptr;

Update: Just found another crash when using new and delete. Fixed by setting _freeFont to nullptr (lines 108 and 125).

const GFXfont* _freeFont = nullptr;

Thanks
Felix

Rop

@felmue Thanks! I fixed it but in the cpp and not in the .h. I prefer to not have too much in the header files.

The version that's online now (I merged 'visual' into master on my repo) has fixes. Also there's now a copy of the original M5Stack library on my Github as well, it has the identical M5Button library file with Buttons and Events. Allows you to compile

#include <M5Stack.h>

void setup() {
  M5.begin();
  M5.BtnA.setLabel("Test");
  M5.BtnB.setLabel("Wow !");
  M5.BtnC.setLabel("Amazing !");
  M5.BtnA.off = M5.BtnB.off = M5.BtnC.off = {BLUE, WHITE, NODRAW};
  M5.BtnA.on = M5.BtnB.on = M5.BtnC.on = {RED, WHITE, NODRAW};
  M5.Events.addHandler(eventDisplay);
  M5.Buttons.draw();
}

void loop() {
  M5.update();
}

void eventDisplay(Event& e) {
  Serial.printf("\n%-12s %-18s", e.typeName(), e.objName());
  if (e.type == E_RELEASE || e.type == E_PRESSED) Serial.printf("%5d ms", e.duration);
}

(I'll post this wider after a bit more testing and documenting.)

felmue

Hi @Rop

playing with your latest code. Cool stuff.

You probably might want to remove or comment the Serial.println() in below function.

bool Button::pressedFor(uint32_t ms) {
	Serial.println(_time);
	return (_state && _time - _lastChange >= ms);
}

Cheers
Felix

Rop

@felmue Removed that line.

I just updated my M5Stack classic version of the library. Buttons have a few new features, such as myButton.cancel() which stops any further high-level event processing for that button, only E_RELEASE will be sent after that. It's used internally for gestures, but you can also use it for something like this:

#include <M5Stack.h>

// Shortcuts: the & means it's a reference to the same object
Button& A = M5.BtnA;
Button& B = M5.BtnB;
Button& C = M5.BtnC;

// New buttons, not tied to hardware pins, so off until set with setState()
Button AB = Button(55, 193, 102, 21, true, "BtnAB");
Button BC = Button(161, 193, 102, 21, true, "BtnBC");

void setup() {
  M5.begin();
  A.off = B.off = C.off = AB.off = BC.off = {BLUE, WHITE, NODRAW};
  A.on  = B.on  = C.on  = AB.on  = BC.on  = {RED,  WHITE, NODRAW};
  C.tapTime = 0;
  B.longPressTime = 700;
  M5.Events.addHandler(eventDisplay, E_ALL - E_TOUCH - E_RELEASE);
  M5.Events.addHandler(buttonDown, E_TOUCH);
  M5.Events.addHandler(buttonUp, E_RELEASE);
  M5.Buttons.draw();
}

void loop() {
  M5.update();
}

void buttonDown(Event& e) {
    if (A && B) {
        A.cancel();
        B.cancel();
        AB.setState(true);
    }
    if (B && C) {
        B.cancel();
        C.cancel();
        BC.setState(true);
    }
}

void buttonUp(Event& e) {
    if (!(A && B)) AB.setState(false);
    if (!(B && C)) BC.setState(false);
}


void eventDisplay(Event& e) {
  Serial.printf("\n%-14s %-18s", e.typeName(), e.objName());
  if (e.type == E_RELEASE || e.type == E_PRESSED) Serial.printf("%5d ms", e.duration);
}

These two M5ez-like combi-buttons now behave like real buttons, with doubleclick and all. The event viewer in this example does not present the - noisy - E_TOUCH and E_RELEASE events but all the others. As you can see one button has no taps, another fires a longpress event after 700 ms. Also if (myButton.isPressed() ... can now be shortened to if (myButton) ...

Things have become cleaner and simpler on the inside, and there is now only one state-machine for touch and regular buttons. Will update the Core2 version later or tomorrow.

Rop

New version is up.

Changes:

Introduced M5.background, the button object that is underneath all others and gets the events if nobody else does. So all events now have a button attached. E_BTNONLY pseudo-event removed.
Various state-machine tweaks.
Added keywords.txt for Arduino syntax highlighting.
Internally cleaner, some functionality moved from M5Touch to M5Buttons.
Added M5.Touch.dump() to see FT6336 registers.
M5.begin() now shows touch: FT6336 ready (fw id 0x10 rel 1, lib 0x300E) as one of the lines.
M5.Touch.stale is the location of the last past touch that is kept in memory of the FT6336. Can be used in light sleeping: only wake the ESP32 up two times a second and still see if someone touched.

Rop

This post is deleted!

Rop

I just pushed a new version that handles two fingers on same button (background is a button) properly.

With that version, try this (it will be an example later) to nicely visualise what the sensor sees.

 #include <M5Core2.h>

const int r = 50;

void setup() {
  M5.begin();
  M5.Events.addHandler(moveEvent, E_MOVE);
  M5.Events.addHandler(touchEvent, E_TOUCH);
  M5.Events.addHandler(releaseEvent, E_RELEASE);
}

void loop() {
  M5.update();
}

void moveEvent(Event &e) {
  M5.Lcd.drawCircle(e.from.x, e.from.y, r, BLACK);
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, e.finger ? BLUE : RED);
}

void touchEvent(Event &e) {
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, e.finger ? BLUE : RED);
}

void releaseEvent(Event &e) {
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, BLACK);
}

Rop

Or, even shorter (and with slightly thicker and better visible circles):

#include <M5Core2.h>

const int r = 50;

void setup() {
  M5.begin();
  M5.Lcd.fillScreen(WHITE);
}

void loop() {
  M5.update();
  Event& e = M5.background.event;
  if (e & (E_MOVE | E_RELEASE)) circle(e & E_MOVE ? e.from : e.to, WHITE);
  if (e & (E_TOUCH | E_MOVE)) circle(e.to, e.finger ? BLUE : RED);
}

void circle(Point p, uint16_t c) {
  M5.Lcd.drawCircle(p.x, p.y, r, c);
  M5.Lcd.drawCircle(p.x, p.y, r + 2, c);
}

Rop

New version is up. You can now make gestures that have only directions. So that whole bit in the first part of the example is now:

// Defines gestures
Gesture swipeRight("swipe right", 160, RIGHT, 30, true);
Gesture swipeDown("swipe down", 120, DOWN, 30, true);
Gesture swipeLeft("swipe left", 160, LEFT, 30, true);
Gesture swipeUp("swipe up", 120, UP, 30, true);

The first number is the minimum distance, the RIGHT, LEFT, etc are just #defines for compass directions, the 30 thereafter is 'plus or minus 30 degrees'. The Gesture definitions have the same rot1 flag (the true at the end above) to denote that the direction is relative to the screen in rotation 1. You can still specify two zones for start and finish before the name of the gesture. If you want to specify just one, say ANYWHERE for the other. This direction stuff works because support for it is now in the underlying Point class. You can also say pointA.directionTo(PointB) or if (PointA.isDirectionTo(PointB, 0, 15)) ... to see if it's due north plus or minus 15 deg. Event has the same functions without the To because it has a start and end point in there already. So if you can want to see if a key was dragged upwards you just put something like that in the E_DRAGGED handler.
Key repeat with myButton.repeatDelay and repeatInterval, gives repeated E_PRESSING events.
The Events object is gone and its (few) functions are now functions of Buttons. Last time I move stuff around like that, I swear.
All my code is now formatted (with clang-format) for easier reading and reworked to be maximally understandable. Will work in some more documentation on how I did things, esp. rotation and other tricky bits. (The .clang-format config file is in the root, see the comment in there for usage. I chose Google's style guide with a few minor tweaks, because that was closest to what the rest of the library was doing already anyway.)
I expect to be submitting a Pull Request sometime in the next few days after I finish with the documentation.
I'd be very happy with any test reports.