前回紹介したGitHub - hollance/YOLO-CoreML-MPSNNGraph: Tiny YOLO for iOS implemented using CoreML but also using the new MPS graph API. iOSでのYOLO実行環境はSwiftでした。SwiftはUI環境を整備しなくてはいけないので、私にはとてもやっかいです。これをOpenframeworksで実行させたら、簡単に画像を加工できるはず。この記事には、YOLO標準データをCoreML専用に変換するPythonスクリプトも入っているので再学習も可能のはずです。結果、ソースの変更でopenFrameworksでもCoreMLが実行できました。肝心の画像認識部分の実行速度は、Swift版より今回移植したC++版の方がかなり早くなっているようです。画像加工が簡単なので少し表示内容を変更しています。画像認識部分はマルチスレッドにしているので、画像を加工しても画像表示はCoreML(物体認識部分）に影響されません。ただし実行時iPhoneがかなり熱くなるので、連続して実行する場合は１分前後が目安です。iPhone7以降で実行可能。
Thermal Cam 3 is compatible with LEPTON 3.5,it is necessary to replace ArduinoIDE（ESP_LEPTON3.5_1220.ino）.See below!!!.
Now we are ready to upgrade for Thermal Cam and Thermal Cam 3, we will release the installation method. When using Thermal Cam (or Thermal Cam3), it is necessary to replace all Arduino IDE.
Dan O さんに現在スイッチサイエンスで販売しているボードや、Thermal Cam・Thermal Cam3の比較動画を作っていただきました。2つのアプリの違いやバージョンアップ内容がよく分かると思います。
The sample image is Thermal Cam 3, but Thetmal Cam just changes the button to red and there are no operational changes. However, Thermal Cam has only 4 color map selection numbers.
Thermal Cam (LEPTON1用アプリ）の画像。ボタンが赤に変わります
Thermal Cam (for LEPTON1 & 3) がバージョンアップしました。Thermal Cam (for LEPTON 1 & 3) has been upgraded.
This time only with Thermal Cam 3, bug fixing, speed improvement, connectivity improvement in places where WIFI is difficult to connect, addition of image inversion function.
Please see this image for the contents.
内容の詳しい説明は最後の「新機能について」を参照。
For the detailed explanation of the contents of version upgrade, see the last "About new functions".
そもそもLEPTONサーモセンサーってなに？ What is LEPTON thermosensor in the first place?
This sensor alone senses heat and shows thermal image. Thermal Images can be seen in the dark without any lighting. Please see the manufacturer site for details.
The previously announced LEPTON 1 application has a resolution of 80 x 60. The application interpolates the image by 8 times and reproduces the image of about 320x240. The LEPTON 3 application can reproduce 320 x 240 movies by quadruple interpolation of the original 160 x 120 resolution. Is it the same? Please see the difference below. Can you see the difference in detail of Jaggies and background ? . It is a feeling that a veil has peeled off when actually used. Until now LEPTON 1 will lead to the discovery of small parts that you had missed. Thermal Cam 3 is easy to occur error because communication is high rate as the resolution is high, but because the algorithm is changed, the communication is very stable, resulting in slightly higher video speed than Thermal Cam.
LEPTON 1 seems to be good, but the distant view is blurred (not the details). In this interpolation method, jaggies are conspicuous on oblique lines. (Because the previous image is used, the button has changed)
Image of LEPTON 3. You can see that the details of the background are clearly shown. There is no problem with oblique lines. The whole is sharp.
背景の丸い物体はこんな形の照明です→
なぜ WROOM02 (ESP8266) を使うのか？Why use WROOM 02 (ESP 8266)?
その消費電力の少なさと価格の安さに尽きます。消費電力はESP32やPi Zero の約半分。基盤を選べば280mAhの極小リポバッテリーでLEPTON3の場合でも１時間以上赤外線動画を発信し続けます。
It consumes less power and lower price. Power consumption is about half of ESP32 and Pi Zero. If a good pcb is chosen, it will continue to transmit infrared video for over 1 hour even in case of LEPTON 3 with 280 mAh mini lipo battery.
This means that you can use it for various purposes, including toy drone. However, as we introduced in previous articles, there are many problems. Although it tends to run to ESP 32 with higher performance if it is normal, I continued to stick to ESP 8266. Who will believe that you can receive images of LEPTON 3 with ESP 8266 and still be able to receive on iPhone from around 100 m away! ! .
The above picture is a board specialized for zerotech Dobby, which Mr. Wiwao developed by himself. Since there is nothing to improve on ESP 8266 naturally, there is no problem with skill. With this combination of this board and software of this time it was able to reproduce images with almost no noise of LEPTON 3 from iPhone under the condition that both are 100 m away.
Moreover, the place where we measured was a very bad environment with WIFI relay point of about 20 places. (Actual video can be confirmed below) * 1
Unfortunately the maximum video reception distance on a general board is around 50 m. This application does not choose a board, but if you want performance and stability, you need a custom pcb made by Wiwao. If you can successfully distribute movies with ESP 8266, there is nothing to benefit from using other processors or ESP32.
※ The contents are almost the same even with Thermal Cam (LEPTON 1).
LEPTON3+Thermal Cam3をDobbyに取り付けて撮影すると.......。When LEPTON 3 + Thermal Cam 3 is attached to Dobby and shot ........
Noise enters the image at regular intervals, but the GPS in Dobby seems to be doing bad things. Currently, iPhone application side makes corrections, almost no noise comes in. The actual photographed image on the upper right combines Dobby's image data. I think that it can be confirmed that the resolution enough to withstand practical use.* Thermal Cam (LEPTON 1) can not be good images in detail so far.
※Thermal Cam(LEPTON1）では、ここまで詳細に画像を再現できません。
Thermal Cam3はUDP OSC通信をiPhoneデザリングで実現していますが、周りの妨害電波からほとんど影響を受けません。Thermal Cam 3 conducts UDP OSC communication with iPhone tethering, but it is not hardly affected by surrounding jamming waves.
Although the next movie is recording on the station premises where the WiFi relay point is likely to affect packets at around 20 places, it can be confirmed that there is no image noise at all, as the frame rate goes down slightly.
※Thermal Cam(LEPTON1）でもノイズはほとんどありません。
夜なのにこんな画像が録画できます。Even though it is night, you can record such an image.
人も車も風景も、そして電線も.....。このクオリティー。結構いいかもしれません。
People, cars, landscapes, electric wires ...... This quality. It might be pretty good.
There are 20 WIFI relay points on WROOM 02 (ESP 8266) side. There were 7 places on the iPhone side. It seems that a little packet loss (indicator yellow flashing), but the speed of the video was not a problem. For shooting this image, I use his latest board of Wiwao.
LEPTON 3 is swaying in the wind. It was confirmed that there was almost no delay and no image noise. This place is a residential area where the left side is a cliff soon and it is a place of "devil" that can not reproduce images satisfactorily if error correction of packet is not done properly.
※Thermal Cam(LEPTON1）でも内容はほとんど変わりません
※ The contents are almost the same in Thermal Cam (LEPTON 1)
This application requires installation of the following separate parts and source. Please procure parts and install software according to the order. When source compiling WROOM 02, it is necessary to set memory and CPU speed setting to 80 MHz and 160 MHz respectively. (Thermal Cam (for LEPTON 1) had lowered the frequency before, but compiled with increasing frequency at V 2.0) * 5. Please refer to the board settings as follows.
First, get the ESP 8266 development board, LEPTON 1 or LEPTON 3, LEPTON breakout board.
Below is the procedure from where to obtain the required parts to LEPTON image reproduction.
And ... Sorry we made you wait. We decided to sell the board we made. In addition to stabilizing the signal of SPI, the distance of WiFi will dramatically increase within the technical range! ! . We optimized board size to make it general purpose.
In Japan, you can get it with Cornes Technology or Digi - Key. LEPTON 1 is recommended with a shutter.
もし電子機器を多少かじったことがある人であればFLIR ONEを持っていれば、簡単に取り外すことができます。特殊ドライバ（ヘキサドライバーT5H（センター穴付））が必要で、ビスさえ外せば簡単にLEPTON3を取り外せました。と言うことは、簡単に元に戻せます。（2017年10月6日現在Apple ストアで販売されているFLIR ONEに使われているセンサーはLEPTON1（解像度80X60）。LEPTON3が使われている機種はFLIR ONE Proです。）
If you have kicked the electronic device somewhat, you can easily remove it if you have FLIR ONE. A special driver (Hexa driver T5H (with center hole)) is necessary, you can easily remove LEPTON 3 by removing the screws. To say is easy to undo. (The sensor used for FLIR ONE sold at the Apple store as of October 6, 2017 is LEPTON 1 (resolution 80 X 60), and the model in which LEPTON 3 is used is FLIR ONE Pro.)
The LEPTON sensor itself is completely waterproof, and it is quite strong for impact, so it does not bother with adverse conditions, but detachable shutter parts have a breakable disadvantage. Please handle with care. LEPTON regularly closes the shutter and calibrates the image, but if the shutter breaks, afterimages will remain, such as image failure will come out. Currently it seems that we do not sell replacement parts for the shutter, so careful handling is necessary.
3. BREAKOUT BOARDが必要です。 BREAKOUT BOARD is required.
日本ではコーンズテクノロジーかDigi-Keyで手に入ります。
In Japan, you can get it with Cornes Technology or Digi - Key.
4. Arduino IDEをインストールします(Macの場合) Install Arduino IDE (Mac)
Be sure toselect 2.2.0 for the version. (Since the above figure has already selected 2.2.0, it has not appeared on the selection screen.) If you have already selected 2.3.0 please drop the version to 2.2.0. Installing this source in 2.3.0 does not work well.
5. ボード設定は以下の通りとしてください。 The board settings should be as follows.
The serial port will be displayed if you connect the ESP 8266, but choose the one that is not Bluetooth. The caveat is that if you plug or unplug ESP 8266 multiple times on a Mac, it will not recognize USB. If you do not recognize it, you will recognize it when you change the USB connector on the Mac side. If there are only two USB connectors like a MacBook and both do not recognize, restart ....... is. Please note thatFlash Frequency = 80 MHz, CPU Frequency = 160 MHz.
When initializing iPhone as usual, ssid becomes Japanese. Please correct it to half-width alphabetical as it may cause an error.
Choose Settings -> General -> Information -> Name -> Current iPhone Name> to change the name. Changing the name does not affect the behavior of the iPhone itself.
8. 今回製作したESP8266Arduino ソースをダウンロードします。Download the ESP 8266 Arduino source produced this time.
Please download it according to your LEPTON. (Please note the file name) Open Arduino IDE and double click on the file you downloaded now. Can you put it in the folder? As you are asked, choose YES. We commented on the stabilization of OSC and SPI, the elimination of UDP packet loss problem, and the solution method of WSP reset specific to ESP. If you choose a board, LEPTON images can be reproduced in places more than 100 m away.
9. 今回のiPhoneアプリをダウンロードします。Download this iPhone application.
Search by W & T. Look for Themal Cam or Themal Cam 3 according to LEPTON. I am sorry but funds have bottomed up. I think that it does not come out so much, so I think that it is high with regard to Themal Cam 3, but I would like to collect about 1,000 yen. Since the various tests are cleared, the application works very stably.
10. 今回のソースを開いて一部を修正します。Open this source and fix some.
The parts that need to be modified are the following three places.
Near lines 34 and 35
#define SSID_X “ABiPhone” //iPhone
#define PASS_X "11111111" //iPhone
Set the name of iPhone to ABiPhone, set to 11111111 -> Mobile data communication -> Enter the password by checking the Wi - Fi password.
Line 47 (for LEPTON 1) Near line 53 (for LEPTON 3)
#define LEDpin 16
Please enter the pin number of LED matching your board. Although LED is not particularly necessary, it is convenient to light up as you can understand the state of ESP 8266. The signature is as follows.
LED blinking- when initial WiFi connection is in progress. Blinks when WDT reset is entered in the middle or WiFi reconnection is in progress.
LED Flash short blink Blinks - when iPhone side application is interrupted.
No flashing- When sending data normally.
When the LED glows instantaneously during blinking- When the packet is lost and is sending the same packet again. When there are a lot of shocks, the frame rate of the image decreases because the ambient radio condition is bad and the packet is being corrected continuously.
When GPS is turned on, it glows shimmerly periodically. I think that you can see that there is not much effect on the picture although it surely loses the packet. When BlueTooth is turned on, it glows a lot and the frame rate drops.
When compilation and transfer are done, it completes. Although occasionally an error may appear, please transfer again in that case. (USB is not recognized in many cases)
13. いよいよサーモグラフィー開始 The thermography finally starts !!
I am checking the operation with iPhone 7, 7 plus, SE, iPad mini 4, X. I think that it will work on general iPad, but both require A9 or more CPU. For iPhone 8, 8 Plus, and other iPads, the operation is checked and adjusted with a simulator.
It is not easy from installation to first confirmation of operation, but for everyone who is accustomed to Arduino it is a completely problem free process. While keeping security, it moves with 3 steps (LEPTON side switch ON -> Internet share -> iPhone application launch) after the second time. No troublesome WiFi selection, IP, password entry etc are required at all. Also, even if you use tethering, you will not communicate with the outside, so there will be no communication fee for Thermal Cam 3 operation.
Why do not you make a nice Thermal Cam case according to the application with your idea. I made three kinds of leather cases using the board which was made by Wiwao this time.
In this version upgrade, we need to explain from the concept of dynamic range. This time, three functions were added in roughly divided. In addition, it corresponds to iPhone 8, 8 Plus, iPhone X, iPad Pro 10.5 inch anew.
LEPTON outputs a 14-bit value according to the measured temperature, but if it is monochrome data it will be up to 8,192 tonal levels per pixel. Measurement range is -273 ~ + 300 ℃ (resolution: 0.05 ℃). There are 256 gradations that can be expressed in the current normal display. I add color to it. However, due to limitations of the display device, if the measurable range is set to 256 tones, it will be a very rough image. Furthermore, the temperature range to be measured is only a part of it, and the image can not be expressed because there is no shade. Since the workaround is displayed as an image, the measurement range will be converted to 256 tones here.
Since this method has problems and it will divide the constantly changing temperature range into 256 equal parts each time, as the measurement place moves, the color changes dramatically from screen to screen.
If there is not much difference between maximum and minimum temperatures as in the room, you do not mind, for example if the extreme high temperature part is partially present on the screen, since the resolution will decrease as the temperature difference increases It becomes an extreme temperature width, and the gradation of the room almost disappears.
In order to prevent this, in the case of this example, if it is 256 tones in a certain range close to low temperature, it is possible to express the image.
As a method of adjustment, the dynamic range is made to be able to adjust the low temperature side and the high temperature side of the current maximum and minimum temperature by the whole ratio. Outside adjustment range is white or black. The figure below shows when adjusting the low temperature side.
In the current lowest maximum temperature range, 32% of the low temperature side is expressed in 256 tones. In this case Max temperature is not the temperature of the flame but the maximum temperature of the background. When the high temperature side is adjusted, the Min temperature changes.
As a result, gradation can be represented up to the limit of the resolution of LPTON in a certain temperature range. This is the dynamic range slider of Thermal Cam.
If you want to reproduce a temperature image in a situation where the measurement temperature range is small like indoors, is there no way to eliminate background color change as much as possible using other methods? . It can be solved by changing the measuring range of -273 to + 300 ° C to the minimum room temperature (24 ° C to 34 ° C, for example) in the room, and applying 256 gradation color maps during this period. Since the color corresponding to the measurement temperature is fixed, the image will stabilize.
Set the minimum temperature and maximum temperature as described above and turn on the slide switch. When the color range is set to ON and the Thermal Cam is moved, gray points and the set temperature range are displayed at the upper right of the image.
At the same time, blue and red dots appear in the lowest highest temperature display area at the bottom left. The range of two points indicates the absolute color map range. The current situation has made it possible to judge which area is used within that range. Since the measured temperature zone changes every moment, these two points will also move at all times. By doing this, even if extreme temperatures are partially generated, the background color will not change as shown below (the lighter's fire is in the image).
赤と青の点の範囲と現在のカラーマップがうまく動いていることがわかると思います。
You can see that the range of red and blue points and the current color map are moving well.
Please note that this new function will cause the gradation to be significantly lost if the current color map becomes a small part of the standard color map.
Set Side View on setting. Specify the temperature with the temperature range Target temp you want to check.With this setting, a subview is displayed on the right side of the screen and the target measurement temperature is displayed in green.
As in this example, the maximum temperature minimum temperature range, set temperature range can be confirmed immediately. Please note that when the measurement area is lower than the current minimum temperature, green display will not be made.
ESP 8266 has a WIFI radio frequency band of 2.4 GHz. It may interfere with other Bluetooth or WiFi devices. In this case, we made it possible to cut off the wifi of ESP 8266. In this case, shaking the iPhone, the WiFi of ESP 8266 will stop completely for about 20 seconds. Both radio interference disappear during this time. The risk of crashing when attaching to drone etc. will be reduced.
設定で Shake をONにします。シェイクすると復旧するまでの秒読みが始まります。
Set Shake to ON by setting. Shaking will start the countdown until it recovers.
LEPTONの温度計測に関する全般的な注意点は、 Overall note on temperature measurement of LEPTON
LEPTONは温度計測が安定するまで多少の時間が必要です
作成したアプリは対象熱源の放射率を考慮していません
現在販売しているLEPTON1及びLEPTON3は温度測定の保証はしていません
LEPTON needs some time until the temperature measurement stabilizes
The created application does not consider the emissivity of the target heat source
Temperature measurement is not guaranteed for LEPTON 1 and LEPTON 3 which are currently on sale
以上のことから場面や環境で測定温度に誤差があることをご理解下さい。
Please understand that there is an error in the measured temperature in the scene or environment from the above.
* 1:There are few noises depending on the distance and the surrounding conditions, but the frame rate drops.
Your app (iOS) status is Waiting For Reviewのメールが来る
6月27日 5:40：審査中のメールが来る。
6月27日 6:37：リジェクトメールが来る。ガッカリ😞
We found that your app records the user but does not have the proper indicators that the app is recording, as required by section 3.3.8 of the Apple Developer Program License Agreement<https://developer.apple.com/terms/>.
Specifically, your app records video but does not have a clear visible visual indicator that the app is recording. The small red dot at the corner of the screen is not sufficient to be in compliance.
Next Steps
Please revise your app to implement the required means of indicating the recording activity to all parties. Please note that the recording indicator cannot be disabled and your app cannot go blank during recording.
今回提出した最新版アプリ（1.20.2）ですが、録画前と録画中を明確にしたつもりです。 画像中に現れるインジケーターは、赤外線カメラとのWIFI電波状況を示すもので、接続していないと赤く点灯します。 どこが不足しているか、具体的に教えてください。 今回のアプリの説明ファイルを添付します。 It is the latest version application (1.20.2) submitted this time, but I intend to clarify before recording and during recording. The indicator appearing in the image indicates the WIFI radio condition with the infrared camera, and it will be lit in red if not connected. Please tell me exactly what is missing. I will attach a description file of this application
6月29日15:47 ：こんな内容の文書が書き込まれる
We apologize for the delay. Your application is still in review but is requiring additional time. We will provide further status as soon as we are able. Thank you for your continued patience.
Best regards,
これも遠回しだけどなんとなく申請が通りそうな雰囲気
6月30日6:54：突然審査が通ったメールが来る。すごく嬉しい！！
そして....
結局致命的なリジェクトは１回で済みました。最初から録画ボタンがこんな感じだったら、３〜４日で通ることになります。みなさん録画ボタンにはくれぐれもご注意を。あーそうそう。今回のアプリはW&Tで一発検索できます。Maker Faire TOKYO 2017でお会いしましょう。
サンプル画像はThermal Cam3ですが、Thetmal Camはボタンが赤に変わるだけで操作上の変更点はありません。但しThermal Camはカラーマップの選択数が4色だけになります。
