Leone Huang's Projects

Image: FlowEditor allows zooming out to get a full view of the script's structure.

FlowEditor is becomes IDE for in-game scripting -- allowing scripters to write, view, and test their scripts in the same application. Additionally, scripts have access to a 'time machine' that allows them to save and restore the execution of the flow. The application includes support for additional media such as images, animations, and audio.

Technical Challenges

The biggest challenge for this project is accepting text input from the user. One may think this is a simple task, however, when adding touch keyboards and alternative input methods for CJK (Chinese Japanese Korean), the browser does not provide adequate APIs to facilitate automatically-formatted and structured text input.

Let's consider the possible methods to accept text input from the user.

Capture key presses directly and emulate a text editor. (addEventListener("keydown", ...))

This is the most naive approach, but could not support touch keyboards (especially with autocorrect) nor IMEs (input method for CJK charactesr). Having used desktop applications with a mobile keyboard, not having autocorrect is fatal to productivity on the small screen. Thus, this option was ruled out.

Use an invisible textarea

We can create an html <textarea> element and keep the user's cursor focused on the textarea.

This is the approach taken by Electron code editors such as Microsoft's VSCode and (formerly) Github's Atom. This approach only adds support for IMEs, but does not support autocorrect for touch keyboards. Additionally, since the cursor is inside an invisible textarea, we have to position this cursor ourselves.

We can solve autocorrect by altering the invisible textarea method -- specifically, not clearing (textarea.value = "") on every input event. Instead, on every input event, we can then compare the old textarea value and the new textarea and apply the changes to the actual data.

However, we run into a different fatal problems using this method on mobile.

1. On Android, a user can move the cursor by dragging left and right on the spacebar. It turns out Chrome does not send any JavaScript events, so we cannot reflect the cursor movement in the invisible textarea visually. We can work around this by polling the position of the cursor every frame, which would be fairly inexpensive.

2. We can't move the cursor to reflect changes when the user selects text, since moving the cursor would remove the selection!

In attempt to fix these issues, I had to consider an alternative.

Use a visible textarea

We can fix the visual issues by overlaying the textarea over the visual text. We need to ensure that the textarea and the visual text have the exact same spacing and size, otherwise the textarea's (now visible) cursor can be placed in the right position on the text.

Unfortunately, this is not possible. The browser has different logic to break lines for regular text and text in textareas. Different logic to break lines means the cursor is sometimes placed incorrect, which is incredibly confusing for the user, thus, this option is also not possible.

Use contenteditable

Contenteditable would fix all problems with visuals. The only issue is the complexity of parsing contenteditable input. There is no (standard) API to extract the text inputted into a contenteditable element.

This means we must attach a MutationObserver to the contenteditable element, parse each change in element, and then dispatch those as events the rest of the system understands (such as "change the text in this area", "remove this line", or "insert a line", or, all of those at the same time, since a user can also paste text!).

contenteditable is the most complex option, however, is the only viable option to ensure the best experience for FlowEditor users.

There are still some workarounds for IME input, however, other applications such as StackEdit and Google Chrome's DevTools that contentediatble must make same compromises.

The second biggest challenge is the compilation and execution of the flow instructions. We must take the boxes and graph the user creates and turn that into executable code. The flow was designed similarly to CPU instructions to minimize the complexity of the implementation of the generalized flow executor. This is important since the flow executor needs to be packaged with any programs that need to execute the flow.

Leone has completely switched to using his own apps for learning Japanese, and has also found use re-purposing tAnki to created flashcards to study in university, since the app allowed compete offline (no account) studying in the browser.